Add better support for in-repo vendoring of deps

[tionis]

Some support for vendoring dependencies directly in the repo (similar to how go mod vendor would do it) would be useful.

70 discusses some issues regarding this approach. Currently using a jpm_tree in the repo works partly, but may leave compile artifacts lying around as well as keeping the .git folder.

Some option to define a local tree in the project.janet with some subcommands to update vendored deps would be great. Some solution to not commit compile artifacts and .git folders (a .gitignore could be added automatically or simply recommended in the docs) would be needed as well. Lastly, some way to update the deps would be great. If this feature should be git-specific, we could simply use git-subtrees to vendor deps in the repo. This would solve the update problem.

[sogaiu]

As mentioned in #70, I've been finding git-subrepo works well.

There is a detailed comparison of using it vs both git submodules and git subtrees here.

In my use so far, I agree with the document's assessment that it's overall better than the other two alternatives. I had only used git subtrees briefly (it was actually after trying git subtrees out on a number of projects that motivated me to look for an alternative), but I'd been using git submodules for some time before.

I think it's worth checking out the comparison document and trying git-subrepo. The document spells out various relevant issues as well as what was done for (I think most, if not all, of) them.

Whether one adopts the tool or not, the enumeration of concerns in the document is nice to be able to consider, revisit, and point people at :)

[tionis]

That's indeed a nice comparison. I'm currently looking to subrepo myself, since reading it in #70 yesterday. Subrepo might solve the problem of having the code in the repo and updating it, but some pattern for the build system to ease the use of such an approach would be nice. So perhaps the subrepo init and updating could be let to the user and the project.janet could specify a folder to use a jpm_tree with automatic dep compilation etc.?

[sogaiu]

the project.janet could specify a folder to use a jpm_tree with automatic dep compilation etc.?

I'm not sure (^^; But I can relate to wanting some kind of support.

I've put together a sample repository that tries to use only existing jpm functionality. May be it can serve as part of discussion / exploration of what might work / make sense.

The project includes a specific version of spork and further, has a declare-binscript form and a declare-source form. jpm install leads to the script being installed along with a directory that contains within it the specific vendored spork. There are some custom tasks along with some use of add-dep -- see project.janet for some details.

Once installed, as long as the script is the only thing using the directory with the spork bits in it, perhaps this can work ok.

However, if some other code tries to use the directory with the vendored spork along with an ordinary install of spork, it seems to me one might run into issues. In particular, I think there may be potential issues with the C-bits from the two "instances" of spork.

I think the vendoring approach might work in some cases, but there are some scenarios that seem more prone to problems (e.g. providing a library that vendors something with native bits that a user might have installed separately).

May be it makes sense to try to spell out what kinds of things could work and what things won't in some detail before trying to build things into jpm.

[sogaiu]

To explore a bit further, I made another branch that uses two different versions of spork in the same project.

The sample code uses two different versions of json/encode, which is implemented in C.

The method used involved a fair bit of renaming which can be seen in the commits. Perhaps there are better approaches.

Note that this was a simple case though -- if multiple versions of one library are used, the library's design might not account for multiple versions being used in the same program.

[tionis]

Thanks, I will look into it and experiment when I find the time

[sogaiu]

Hopefully our explorations will make various concerns about this area clearer to us mere mortals :)