Suggestion for monorepo approach

[windelbouwman]

Hi!

I saw this project, and one thing I noticed the many mys-json, mys-math etc. libraries all have their own repository. I understand that each of these projects should be optional, and also that they should depend upon each other in a non circular way. One way to reach this goal, is to create a seperate git repository for each of such a standard library component. I understand the motivation to do this, and there are benefits, but there are also some clear drawbacks from this git repository fragmentation:

• configuration management: how do you know which version of mys-statistics is compatible with which version of mys-math? This results in a compatibility matrix, which will grow rapidly as packages grow in number.
• When applying a change to more than one component, you need to commit to two repositories. Ideally, the components should be loosely coupled, and the interfaces clearly defined. Reality is often not like this.
• Packaging. If mys will become a success (no doubt about it ;)), then you will make package maintainers happy by having a single release bundled with libraries. For example the yocto recipe for python is a bit of a mess in that the standard library components are split out manually, so that one can choose which parts of the standard library to install.

My proposal would be, to create a libraries subfolder in this repository, and structure is at such:

• libraries
  • json (contents of the mys-json repository are copied here)
  • math
  • sqlite

Benefits with this approach:

• All libraries are versioned with the same version number, incrementing when a new version of mys is released.
• Cross component changes are easier to make
• Components can be tested in combination, and are guaranteed to work together well
• Package creators can loop over the subfolders of the libraries folder, and create a seperate installable package, or they can choose to install all libraries at once in a single package. But the separation into subfolders helps.

I hope this makes sense! In the past I would have splitted the standard library in many git repositories. Having dealt with many git repositories and git submodules, I believe this was a mistake, I hope I can explain my motivations for this.

See also: https://en.wikipedia.org/wiki/Monorepo

[eerimoq]

Hello!

Thanks for a well written issue. =)

I agree that monorepos are often more convenient, both for developers and maintainers. However, a successful language will have thousands of useful packages implemented by various people all around the globe. My initial idea was that all packages should be developed in a unified way, that is, no packages should be part of the "language repo" itself, but installed using a package manager. It may very well turn out that commonly used packages will be part of the "language" installer, but it's not obvious to me that it is better. Backwards compatibility is mostly about developer discipline, not where the code is stored. Also, CI is often easier to setup with separate repositories. Running tests based on which file was changed in a monorepo is not as easy to do (and running all takes a lot of time).

Regarding your third bullet, packaging, there is no packaging in the sense of shared libraries. A Mys application is always statically linked (at least that's the idea and how it works now). There are no dependencies to install, the Mys build system downloads (or copies locally), compiles and links each application separately.

Looking forward to a response =)

[windelbouwman]

Agreed that the standard library should be independent of the language, this is one of the issues in python. It would hence be good to have a sort of crates.io where packages can be uploaded. There will be many packages created by many different people, but I would argue that a core set of them (roughly the core of python's stdlib) are depended upon by almost all other packages, if not directly, than transitively.

I still see benefit in keeping a set of core libraries in this repository. There is no direct need to package them in the installer, but they can be seperately deployed to a crates.io like website. Just want to point out that a single git repository can have multiple different artefacts, such as a mys language installer and libraries deployed to "crates.io".

[eerimoq]

The Python standard library is huge, at least 100 modules. I doubt the majority of the programs uses the majority of them, directly or indirectly.

One reason for keeping commonly used packages in this repository would be to make them part of the installer and allow the user to import them without adding them as dependencies in the package configuration file.

[windelbouwman]

Another reason for keeping the packages together is easier github access control. It is easier to grant access to other people, if the repository is in one place. Another benefit is code navigation, if code is located in a single git repo, it can be easily transferred to gitlab. Als CI might become easier to setup, allthough it would test all packages on each commit, which might be both good and bad at the same time. Bad in the sense that it might not beed needed, since the package did not change. Good in the sense that it will check for a consistent set of code working nicely together.

[eerimoq]

There are pros and cons with both approaches indeed. I'll not change the structure right now, but maybe in the future. Instead, I want to make all examples build and run as expected, which is a huge task. The compiler/transpiler is barely started and will need serious redesign along the way. Memory management will likely change as well, and maybe generate C code instead of C++. Never ending project realistically, at least for a single person =)