Nim source packages

[FedericoCeratto]

Nim is currently using git/mercurial clone to install packages and installation must be done only using Nimble. It could benefit from having a simple concept of source package, like many other languages.

Current problems:

• this method requires running git and mercurial on buildbots on every build and blurs the line between cloning a library to use it or to develop the library itself
• often buildbots have no network access during the build and can only fetch tar.gz archives before starting the build. It's very common in security-aware organizations including Linux distributions. See #173
• .nimble files can pull files from arbitrary URLs before or during a build: it breaks guarantees that a given release can be installed (reproducibly) at any point it time, leading to code rot and security risk
• as a result, Nim has extremely low penetration in OS package archives. In comparison, D, Vala, Elixir have more packaged tools and libraries across more OSes.
• tarballs can be easily mirrored or cached or hosted on file-storage services like S3. Various organizations use internal mirrors for files and it's difficult to justify a custom git+mercurial mirror only for Nim.
• GitHub and other forges are/were blocked in some countries
• GitHub and other forges refuse service to some countries
• source tarballs created by GitHub tags pack the whole repository including a lot of potentially unneeded data: large training/test vectors, images, autogenerated documentation, slides, pdfs, videos of tutorials and demos
• this behavior encourages developer not to commit large test files (that could be beneficial instead)
• Linux distributions and large organization do legal vetting of licenses using language-independent tooling that can handle tarballs/zips but not run "nimble install"
• existing repositories can be deleted with no warning and no easy way to recover the contents
• there isn't an online mirror of the Nim ecosystem. Setting it up would require caching hundreds of git repositories. OTOH files served over HTTP can be cached easily with proxies - this is very common in large organizations. Also they can be mirrored by archive.org and similar services.
• there isn't a standard format or tooling to generate a distribution of the Nim ecosystem on disk. Various organizations allow DVD or shared network drives as a trusted source.
• https://nimble.directory could mirror the whole Nim ecosystem if packages were stored as files
• the current tools do not sign packages nor check for signatures. Any mirror would require users to blindly trust its contents.
• there are various attack vectors (that affect Nim as well) listed in https://arxiv.org/pdf/2005.09535.pdf

Proposed solution:

• define "Nim source package" as a <packagename>-<version>.tar.gz or .zip file
• upstream developers are not required to provide such source packages. Git/hg support is not removed.
• a tag on GitHub constitutes a valid source package that Nimble and other tools can pull (as a file)
• the package name is defined from packages.json and guarantees uniqueness
• the package can be generated, pulled, installed by packaging tools like Nimble
• by default it contains all .nim .nimble nim.cfg COPYING LICENSE README* files from the repository without empty directories.
• additional files can be included using source_include in the .nimble file and globbing e.g. *.png , tests/*.csv, tests/**/*.txt. ** indicates recursion
• files can be excluded with source_exclude, evaluated later.
• packaging tools can scan a directory and install packages from it
• packaging tools can scan packages.json and pull packages from simple HTTP servers
• with small, compressed archives, services like https://nimble.directory can store and serve the whole Nim ecosystem (with limited disk space requirements)
• archives can be signed and verified

Example .nimble file:

source_include = "manual.txt" "docs/*.rst"
source_exclude = "tests/**" "experiments/*.nim"

[Clyybber]

• ...it blurs the line between cloning a library to use it or to develop the library itself

It should be easy to develop a library after cloning it to use it IMO (nimph is doing a great job at making that easy)

• some buildbots have no network access (by policy) but allow accessing local resources

This doesn't make sense.. I'd have to download the source package anyways?

• GitHub and other forges are/were blocked in some countries

• GitHub and other forges refuse[d] service to some countries

Git and mercurial repositories can be easily mirrorred.

• source tarballs created by GitHub tags pack the whole repository including a lot of potentially unneeded data: large training/test vectors, images, autogenerated documentation, slides, pdfs, videos of tutorials and demos

• this behavior encourages developer not to commit large test files that could be beneficial

• it is not obvious which subset of those files are really needed at build/runtime

Nimble/nimph could easily do a sparse clone+checkout..

• existing repositories can be deleted with no warning and no easy way to recover the contents

But everyone who has cloned or forked it can reupload/push it. I think your proposal makes this worse.

• it is difficult to create an online mirror of the Nim ecosystem

To create one single mirror of the ecosystem means the ecosystem would have to be really centralized, not a good idea IMO.

• it is difficult to create mirror/distribution of the Nim ecosystem on disk

Your proposal won't make this easier.

It could benefit from having a simple concept of source package, like many other languages.

I don't see how this proposal makes anything simpler... (aside from not having to install git and mercurial on the build bot :p ), in fact it creates an (IMO) unneccessary split, which will make things more complicated (for the package managers, for the users and for the package developers)

[andreaferretti]

Just commenting on a few items:

some buildbots have no network access (by policy) but allow accessing local resources. It's very common in security-aware organizations. See #173 GitHub and other forges are/were blocked in some countries GitHub and other forges refuse[d] service to some countries

You can just mirror them and host a packages.json file that points to your mirror instead of the Github repository. In fact, hosting packages locally is easier for Nim than, say, for Java or Python which require language specific tools.

source tarballs created by GitHub tags pack the whole repository including a lot of potentially unneeded data: large training/test vectors, images, autogenerated documentation, slides, pdfs, videos of tutorials and demos this behavior encourages developer not to commit large test files (that could be beneficial, instead) it is not obvious which subset of those files are really needed at build/runtime

This is why Nimble supports skipFiles and skipDirs

existing repositories can be deleted with no warning and no easy way to recover the contents

Well, yes, but you can easily mirror them

it is difficult to create an online mirror of the Nim ecosystem. OTOH files served over HTTP can be cached easily with proxies - this is very common in large organizations. Also they can be mirrored by archive.org and similar services. it is difficult to create mirror/distribution of the Nim ecosystem on disk. Various organizations allow DVD or shared network drives as a trusted source.

How is this difficult? It is enough to write a scripts that git clones all the endpoints listed in packages.json and you have a working clone on disk (git does work from disk uris). If you host the directory where you clone everything with whatever git server you like, you have a hosted working clone.

[FedericoCeratto]

@andreaferretti: updated the RFC to clarify the requirements.

The topic has been discussed with @Araq. He plans on using a tarball for the Nim distribution in #173. It will contain the compiler and usual and also packages.

There isn't an exact specification on how exactly the packages will be cloned and stored.

I'll update the RFC if something comes up.