Rework browser strategy

[holgerd77]

Situation

Our browser strategy having a monolithic browser-builds repository has proven as a dead end, with one failed revival attempt and otherwise being stalled for months.

Therefore we should shift here to an in-repository approach - like everyone is doing it - and directly deliver browser-friendly builds within the different repos + do accompanying testing and optimization. There is some groundwork here one can build on - e.g. karma browser tests in some libraries (e.g. for https://github.com/ethereumjs/ethereumjs-tx), but this is fragmented, often outdated (especially README instructions and examples), so there shouldn't be expected too much from that.

Process

Work towards browser compatibility can as some guidance be done in a two-step process and would encompass the following components:

First round (get it to work):

• Open a new issue for browser compatibility for the library with a reference to this meta issue to keep track
• Analyze the current state of the library and add some notes on the issue page
• Discuss/choose an appropriate bundler (webpack, rollup, other)
• Add testing (karma) of the bundling setup
• Add browser build instructions to the README
• Add caveats and known limitation descriptions to the README
• Update the docs/README with at least one working browser example

Second round (optimization):

• Analyze bundle size, optimize bundling process, make code optimizations
• Analyze browser build performance, benchmark, optimize on that
• Think about browser-specific edge cases and add these to the common test suite

Note1: Note that these too rounds are partly interdependent (especially the choice of a bundler regarding to future optimizations), this write-up is just to give this process some structure.

Note2: It might make sense to think these steps along library rewrite, especially a switch to an ES6 modules structure (for rollup) and/or rewriting a library in Typescript.

Note3: It will probably make sense to start introducing this in a smaller library with not too many dependencies, and not directly start with e.g. the virtual machine.

[axic]

The main reason for the single repo was the build size optimisation. If all these separate repos are packaged separately they all will include a lot of the same dependencies blowing up the resulting size.

Probably the real solution would be having a monorepo for ethereumjs.

[GrandSchtroumpf]

I think monorepo is the way to go. And using Typescript would be a great help to avoid common mistakes when dealing with interoperable projects such as ethereum-js one.

As regard for the bundler, I would choose webpack. It is used widely by many frameworks, and works very well with typescript. And webpack is well known for optimization (second round).

For the testing, I know that Jest gets very popular, and many Karma testers are changing for that.

For the doc, why not use a documentation generator?

[holgerd77]

I am personally not sure/convinced, that a monorepo is the way to go. But apart from that a monorepo won't happen any time soon (12+ months), there are too many other things which have higher priority right now and the different modular repositories are too widely used in production, this would be too much of a shift.

What is with updating the code of the various repos to ES6 modules and then build with Rollup, wouldn't this already solve lots of the issues? We had experiments there already on the VM https://github.com/ethereumjs/ethereumjs-vm/pull/271 and results were extremely promising regarding build size.

[GrandSchtroumpf]

Well, I guess it's a good patch. You can make some good improvement in doing so. But if you want to rewrite some libraries (Note2), you should rethink the whole process. Maybe it can be a concurrent process.

Here are some arguments :

1. Monorepo:
   • This is very convenient for interoperable projects like ethereumjs/*.
   • You can share dependencies between projects, keeping them all in the same version easily.
   • This is a very good User Experience for developers who want to know everything ethereumjs has to provide. Which is a very difficult process to understand if you don't have the whole picture.
2. Webpack
   • Performance (Tree Shaking).
   • You can build dependency graphs to show graphically how projects depend on each other.
   • Eases development (Hot Module Replacement).
3. Typescript
   • Auto-descriptive: You can build interfaces to describe the inputs and outputs of the functions/classes. This is very convenient for community build projects, and reduce the time spend in gitter to answer the question "what entries have the second arguments 'options'".
   • No need to produce and maintain typings file for devs who work with Typescript (who are going to ask them anyway).
   • Works well with webpack.
   • Works well in monorepo thanks to "paths" configuration (help to import others modules from the same monorepo).

Now, as you say, this would be a big shift and may ask too much effort. So if you're ok to give this workflow at try, I can build a small example (at least I can give you feedback after)

[holgerd77]

I am largely sceptical about a Monorepo approach, some arguments:

• Many of the ethereumjs libraries are used in an isolated manner for different purposes, why should someone wanting to use the util repo have to care about the virtual machine
• Cloning/dependency installation becomes a monster, the ethereumjs-testing repo - normally just used by selected repos and having 100s of Megabytes - would now always have to be installed
• Issues/tasks become unmanageable, I am just happy that I have labeled all issues through on the various repos, if this would be now thrown into one that would be a nightmare
• In my view it's more developer friendly - at least on the contribution side - to have a single-purpose repo where one has some focused and cleanly separated room to start from

Think I would find more arguments, just low on time now.

We can nevertheless keep this open for discussion for another 1-2 weeks, I also haven't thought on this too much and some of my reservations might be more and others less justified. I nevertheless have the general relatively strong opinion that - even if positive arguments outweight negative to some extend - that's too much of a project to start and currently I wouldn't be very open to go into this direction.

But apart from that: choosing a better build process (2.) and using TypeScript (3.) would also be (at least to some extend) independent from (1.) (wouldn't it?).

[GrandSchtroumpf]

Indeed, there are some pretty good arguments here. I'll check how we can do something that "looks like" a monorepo to have the advantages with fewer disadvantages (maybe git submodules...). Of course, we can use webpack and Typescript without a monorepo =)

[GrandSchtroumpf]

Just to give more inside about a monorepo(ish) format. Here is an example using submodules: https://github.com/Crypto-lyon/git-submodules-template Each repo is standalone, but you have one to unite them all in which you can have some common scripts and config (install, bundler, linter, tsconfig). It makes interoperability between project easier and cleaner.

[holgerd77]

Outdated, will close.