search

madjam002 / yarnpnp2nix

A performance focused and space efficient way of packaging NodeJS applications with Nix
MIT License
43 stars 5 forks source link

don't use FOD for unplugged packages #14

Open adrian-gierakowski opened 1 year ago

adrian-gierakowski commented 1 year ago

as it leads to hash mismatch errors due to inability to predict the hash of a FOD for all platforms and hence bad user experience

FODs are also not rebuilt when their inputs change and so breaking changes can slip in if FOD with given hash is already ix nix store (or cache)

I'm happy to work on this, but would be great to know if there was any particular reason it was done this way or if you foresee any potential blockers.

related

btw. https://github.com/stephank/yarn-plugin-nixify (or rather my fork of it) correctly handles unplugged packages so it is definitely possible to implement

madjam002 commented 1 year ago

Unplugged packages often need internet access to download files and dependencies, although this may be the exception rather than the norm. How does yarn-plugin-nixify get around this?

The nix sandbox will disable internet access if it is not a FOD as it introduces a source of nondeterminism. This doesn't apply on macOS as there is no network sandboxing with Nix.

adrian-gierakowski commented 1 year ago

How does yarn-plugin-nixify get around this?

It fetches all sources as FODs and then packages which are listed as needing a build step are built in ordinary derivations, in which the .yarn/cache is pre-filled with all deps required by the unplugged package (and the source of the package itself).

Unplugged packages often need internet access to download files and dependencies, although this may be the exception rather than the norm

In such cases it's up to the user to provide all required deps in the nix way, and patching the package to not fetch anything in impure way.

There is also this final step performed for each isolated package, which I don't fully understand and which seems a bit hacky.

adrian-gierakowski commented 1 year ago

Btw if you want to have a look at the source of the plug-in, here’s the fork\branch used in the test repo above

I would really like to switch to using your plugin as I need the workspace support and your implementation seems much more comprehensive and less hacky. But the way unplugged packages are handled atm is a blocker for me.

I’d be happy to work on this. I’m guessing if we paired on it, it could go much quicker. Let me know if you’d be open to it and have some time for it next week.

Btw, if we can solve this, I’d be adding support for private GitHub repos and private npm registries, in a similar way to how I’ve done it for the nixify plug-in

madjam002 commented 1 year ago

Thanks for all the info that's very helpful! I don't really have any bandwidth to be looking into this unfortunately, I recommend making any changes in a fork for now.

For me the most that I would consider implementing is just building unplugged packages in the sandbox by default like you suggest without a FOD, then packages which need internet access can optionally have a FOD added or __noChroot. Patching packages one-by-one is a last resort for me and I'd rather just disable the sandbox for my use case as it's more pragmatic.

Regarding private github repos + npm registries, I'd recommend a fork for this as I don't want to maintain code for this, if you have packages in other repos then my approach would be to add them as flake inputs and build from there + utilise the nix cache rather than using NPM registries for artifacts.

adrian-gierakowski commented 1 year ago

I recommend making any changes in a fork for now.

Sure, will do. Would you accept a PR or should I expect to be using my fork for the foreseeable future?

my approach would be to add them as flake inputs and build from there

Interesting. How would you include a package imported via nix in package.json of another package? Via portal protocol?

Btw. I might not actually need private packages\registries since I’m planning to consolidate everything in a monorepo and use workspace protocol for all private deps.

adrian-gierakowski commented 1 year ago

This is now working on this branch. Not testing an MacOS yet, but will definitely do in the next couple of weeks.

If you're interested in any of the changes I'm happy to discuss and adapt what I've done as think it would be beneficial for everyone to have one version of the plugin which could serve both of our use cases, as this way it would be more likely to server others.

Thanks for all the work you put into it so far!

madjam002 commented 1 year ago

@adrian-gierakowski This is awesome and thank-you for the contribution! I'd love to take a look at getting this merged, just super busy over the next month or so with other projects. When I've got some time I'll look at integrating the changes.