Development strategy and repository structure

[kirlat]

Right now we're facing several development challenges. Some of them are described in #25, #23, #21, and alpheios-project/node-build#24, but those are not limited to that. The process that we employ now works, but would it be able to serve us in the future? With increased code complexity it might be hard and/or time consuming to use the existing process; as a result, code quality and development time may suffer.

Using those opportunities that are currently available to us it might be paramount to change/make corrections to the existing process so that it will not be a burden but a helping tool for the months to come.

After thinking about it for a while and studying best practices currently established I think the solution to our issues may come in a form of a monorepo with independent packages inside it.

Here is some background. Monorepo has some advantages and disadvantages. I will try to list all that are of importance for us as objectively as I can here so we'll decide if it's worth it. I personally think that in our situation the price we have to pay for it is much smaller than the advantages we will get. The information about monoreops is taken from Wikipedia and several other sources. I will list most interesting of them at the end.

Monorepo's advantages:

• Ease of code reuse – Separate libraries can be reused without resorting to dependency management (npm in our case). We could import modules directly; that's much simpler. It can be an answer to #25 problem.
• Simplified dependency management and less storage needed – In a multiple repository environment where multiple projects depend on a third-party dependency, that dependency might be downloaded or built multiple times. In a monorepo the build can be easily optimized, as referenced dependencies all exist in the same codebase. That can help to solve alpheios-project/node-build#24. It will also save some disk space as dependencies will be installed only once. It will make upgrades faster as well because upgrades have to be done only once.
• Atomic commits – When projects that work together are contained in separate repositories, releases need to sync which versions of one project work with the other. And in large enough projects, managing compatible versions between dependencies can become dependency hell. In a monorepo this problem can be negated, since developers may change multiple projects atomically. This can help with #21.
• Large-scale code refactoring – Since developers have access to the entire project, refactors can ensure that every piece of the project continues to function after a refactor. This is related to our discussion of how a small change in, let's say, data-models can cause a ripple effect in inflection-tables.
• Collaboration across teams – In a monorepo that uses source dependencies (dependencies that are compiled from source), teams can improve projects being worked on by other teams. This leads to flexible code ownership. This is probably not so important for us now.
• Consistent versioning with locked versions across repositories/Loss of version information - If this is employed all packages across repo will have the same version. It has both advantages and drawbacks and that's why its title is reflecting both. We might not go this road but if we do, it can solve our troubles with knowing what particular version of data-models is used in a specific webextension build. It is also an answer to the problem whether and how should we increase a version of components if it was rebuilt with an updated version of data-models. Everything will have the same version and it will be very easy to track changes this way.

Monorepo's drawbacks:

• Consistent versioning with locked versions across repositories/Loss of version information – Although not required, some monorepo builds use one version number across all projects in the repository. This leads to a loss of per-project semantic versioning. This also leads to situation when commit just bumps the version number of a package with no useful changes made to the underlying code. See Babel, for example: https://github.com/babel/babel/commit/75767d87cb147709b9bd9b99bf44daa6688874a9#diff-6f766c8b8c6ab57a546fabbab6a0c169
• Lack of per-project security – With split repositories, access to a repository can be granted based upon need. A monorepo allows read access to all software in the project, possibly presenting new security issues. That's probably is lesser problem for us as we have only one private repo so far and are not planning to have many.
• More storage needed – With split repositories, you can fetch only the project you are interested in. With a monorepo, you might need to fetch all projects. Note that this depends on the versioning system. Not a big issue for us, on my opinion, as we all have all repositories checked out anyway. In fact, monorepo can take less space as noted above.
• Scalability problems – If monorepo grows too large, a version control system may choke with it. I think it is not a problem for us because we're not expected to grow the size of Twitter or React (which all uses monorepos).

Monorepos with separate packages This is when a monorepo has modules developed as separate packages hosted within a single repository, usually called packages. It allows to keep code modular. Each module can be published as a separate package to npm and used independently. Some examples of this are Babel and Jest. I think we shall employ this technique by keeping our existing packages separated or even maybe splitting is more (components can be a candidate) as it is easier with the monorepo approach. There are several tools that can help us with managing separate packages within a monorepo. The most established is Lerna and I think we should consider it first but there might be other approaches as well.

Who uses monorepos? Many large open-source JS projects do including, but not limited to: Babel, Lodash, React, Angular, Ember, Meteor, Jest.

@balmas, @irina060981: Please let me know what your thoughts are on this. Thanks!

Some useful references: A Wikipedia page Repo style wars: Mono vs Multi Cross-repository Component Sharing using Mono-repo Multi-packages Architecture Building large scale react applications in a monorepo

[balmas]

I think this makes a lot of sense. I do have a couple of questions:

1. Would we keep the webextension and embed-lib as separate repositories? What are the pros and cons of this?

2. What about service level libraries, like lexis-cs, and the user word and setting apis? I presume these will still remain separate codebases?

3. What is the most sensible way to merge the repositories without losing commit history?

4. I think we will have to be a to build separate packages within the monorepo. Components is already almost too big for jsdelivr to serve as a package.

[kirlat]

1. Would we keep the webextension and embed-lib as separate repositories? What are the pros and cons of this?

We can do it both ways but I think probably not. Maybe we can put webextension to the monorepo but I'm not sure. On my opinion, monorepo should contain packages that are building blocks of the same app(s). Since embed-lib and nemo-ui do not integrate components directly but load them dynamically they probably should be in a separate directory. Webextension integrates components within its build so they can be in a same monorepo; however, we can keep them separate as well: pulling updated components library into webextension is not a big deal. There are no complicated relationships between the two as there are between several libraries within components.

I think we should start with minimal amount of packages in monorepo possible and then see how it goes. I was thinking we can start to move packages into the monorepo one by one, starting from simplest ones, and then move more once we get experience and trust.

[kirlat]

2. What about service level libraries, like lexis-cs, and the user word and setting apis? I presume these will still remain separate codebases?

I think it's better to keep them separate. I would just separate a messaging service from LexisCS to break a dependency of Client Adapters on it.

[kirlat]

3.What is the most sensible way to merge the repositories without losing commit history?

There are techniques for that that work in majority of cases. Below are some article about that: https://medium.com/@filipenevola/how-to-migrate-to-mono-repository-without-losing-any-git-history-7a4d80aa7de2 https://www.shopsys.com/how-to-merge-15-repositories-to-1-monorepo-keep-their-git-history-and-add-project-base-as-well-6e124f3a0ab3/

[kirlat]

4. I think we will have to be a to build separate packages within the monorepo. Components is already almost too big for jsdelivr to serve as a package.

I'm for it. We can tweak our architecture to make things more efficient. The big part of components is inflection tables data. If we'll move it to Lexis this will allow us to reduce components' size by 1MB at least.

[kirlat]

If we'll try the monorepo, what approach shall use use for versioning packages within it (client adapters, components, etc.)? Shall they all have the same version or will we go with different ones? On my opinion, one version is simpler and makes sense because all libraries are part of the same entity, but I might be wrong.

@balmas, @irina060981: what do you think?

[balmas]

1. Would we keep the webextension and embed-lib as separate repositories? What are the pros and cons of this?

We can do it both ways but I think probably not. Maybe we can put webextension to the monorepo but I'm not sure. On my opinion, monorepo should contain packages that are building blocks of the same app(s). Since embed-lib and nemo-ui do not integrate components directly but load them dynamically they probably should be in a separate directory. Webextension integrates components within its build so they can be in a same monorepo; however, we can keep them separate as well: pulling updated components library into webextension is not a big deal. There are no complicated relationships between the two as there are between several libraries within components.

I think we should start with minimal amount of packages in monorepo possible and then see how it goes. I was thinking we can start to move packages into the monorepo one by one, starting from simplest ones, and then move more once we get experience and trust.

This sounds like a good plan to me.

[balmas]

1. What about service level libraries, like lexis-cs, and the user word and setting apis? I presume these will still remain separate codebases?

I think it's better to keep them separate. I would just separate a messaging service from LexisCS to break a dependency of Client Adapters on it.

I think I want to do this anyway. More on that soon, hopefully.

[balmas]

If we'll try the monorepo, what approach shall use use for versioning packages within it (client adapters, components, etc.)? Shall they all have the same version or will we go with different ones? On my opinion, one version is simpler and makes sense because all libraries are part of the same entity, but I might be wrong.

@balmas, @irina060981: what do you think?

I think 1 version is simpler. I would go with that until we have a clear reason not to.