Congrats!

[adrian-gierakowski]

Hi @aakropotkin

What a commendable effort! Looks like you put some serious thought into it.

I wonder if you could comment on the advantages of using this library over https://github.com/madjam002/yarnpnp2nix?

Also, any gotchas one should be aware of before deciding to go full in and convert all their projects to using floco?

Thanks!

[aakropotkin]

Thanks for the kind words.

I've actually used yarnpnp2nix quite a bit and really enjoyed it. For context floco was originally designed for use in a large monorepo with ~100 local projects with ~7,000 deps, several of those had node-gyp compilation phases. With that in mind yarnpnp2nix wasn't a great fit for us because ( at least when I used the tool ) it didn't build dependencies in individual derivations, so any change to any dep triggered a rebuild of everything. Having said that I think the UX side of that tool is top notch, and really liked the approach of using a Yarn plugin.

floco does quite a bit differently, and the context it was designed for really explains most of the design decisions I made.

1. Derivations are driven by bash scripts using stdenv + jq. node is present but only to prepare packages that have install scripts, or build scripts ( local projects only ). Shell aliases for yarn and npm exist to reroute attempts to trigger (yarn|npm) run <script> invocations. This had a pretty significant effect on performance and allowed for "dead simple" modification of build steps. It's a bold departure though, and can be a deal breaker for some projects.

2. Every build and install script is executed in an isolated derivation to improve caching. This means if you have zeromq as a dep in 20 projects, you'll compile it once, not 20 times. This is the largest performance improvement over most other node2nix tools out there. The "catch" here is that the small handful of projects that actually need their peerDependency declarations present when install scripts run need some manual overriding to build. Luckily these projects are incredibly rare in my experience, it's very uncommon for install scripts to refer to their peerDependencies until runtime; but your mileage may vary. If you run into a situation where this is a problem pop into the Matrix chat and I can help you write the override. I plan to write a guide covering this soon but haven't yet.

3. The "metadata/build-recipe" regeneration process isn't plugged into Yarn like it is with yarnpnp2nix. This would be a great extension, and I'd actually love to use what yarnpnp2nix has done as a reference for something similar; but for now you have to manually trigger refreshes. Having said that, today I just finished a working implementation of a routine that would allow the "pins" held in yarn.lock to be read directly without regenerating any metadata ( in impure mode, for pure mode you'd need to override the default fetcher, or create a pdefs.nix file with narHash info ). I need a few days to get that tested, work out kinks, and exposed as a convenient interface - but it's a big breakthrough.

4. floco is significantly easier to extend and integrate with other Nix infrastructure. I'm obviously biased, but this framework was designed from the ground up to be flexible and transparent. It's made developing the tool significantly harder, but it I think it's been worth the extra effort. The module system provides opportunities to optimize unique workflows very easily. If you want to use builtins.fetchTree for local builds but nixpkgs.fetchzip in CI to optimize caching, you just flip a switch. You can declare resolution overrides without a proxy in a transparent manner. You can programmatically add and remove deps from your indirect dependencies. If you want to delete every package that starts with the letter "g" across the entire build plan, you can do that in ~4 lines of code. Go nuts with it - it's just a matter of reading docs, sources, and experimenting.

5. Workspace support in floco is relatively undeveloped at the moment. I used it on a huge workspace in our monorepo, but the extensions I used were tailored to our unique usage and weren't really sensible to add the the upstream repo. I intend to use some of the same approaches to provide workspace support in the future, but for now you basically have to run fromPlock -- -pt -- --install-links --workspace=false a bunch of times and make a big imports = [./project-a/floco-cfg.nix ./project-b/floco-cfg.nix ...]; list to mimic a workspace. It's worth waiting a bit if your projects are reliant on workspaces.

6. The ylockToPdefs translator works but the interface is wonky and I haven't exposed it for "real usage". There's a guide where I explain how to use it, but there's some limitations about peerDependencies and dependency cycles that won't work for some projects. The routine I finished today will resolve those issues once I have it fully integrated into the framework though so this is a temporary limitation.

7. While floco is faster, more flexible, and generally "danker" - it's also more complicated. For folks that want to take advantage of the optimizations and features it's a great tool that's worth picking up and dedicating time to learning. I'm cranking out new guides and CLI/UX improvements every day, but for complex projects the ~4 line initialization process in the README probably won't "magically work" in the way that yarnpnp2nix and node2nix often do. Again, the routine I wrapped up today should put us on level footing with them though once it's integrated.

Hope this helps, and again I really appreciate the kind words - I worked my ass off on this thing haha. Feel free to pop onto the Matrix any time if you want help setting up a project or if you have any questions.

[adrian-gierakowski]

Thank for the comprehensive reply and apologies for taking so long to follow up.

I hope to find some time to play with floco in the next couple of weeks, as I have a monorepo on JS/TS services which needs to be packaged with nix. So far we’ve converted one service using a our fork of https://github.com/stephank/yarn-plugin-nixify. But that required also forking yarn berry which I’m not keen on maintaining long term.

I only recently found out about https://github.com/madjam002/yarnpnp2nix and at first read it seemed a bit more well rounded. However I’m not sure if either of these are well suited for packaging workspaces consisting of multiple services. It doesn’t seem like a single yarn.lock, which is usually what people have in monorepos, is going to cut it.

We currently build services in isolation, each having its own lock file, Dockerfile and ci job, but would like to introduce a top level workspace as it can provide better DX when working across multiple service\packages at the same time.

I was looking at ways to maintain both top level and individual lock files per service\package, but I wonder if that would still allow us to deduplicate and\or keep in sync deps used across multiple services\packages.

Your project looks like it could address all of our needs. Hope I’ll be able to come back to you with some useful feedback soon!

[aakropotkin]

floco was made for your exact use case. What's missing is a guide that covers a "complex setup" like you're describing, but your situation is precisely what I use the tool for.

If you hop over onto Matrix I can walk you through setting things up. Alternatively you can wait for a fresh guide. I need to make a public monorepo to use as an example, or pick an existing project. If you have any suggestions let me know. I might build npm using a yarn.lock+ floco just because I think it would be ironic, but I'm open to other recommendations.