search

nmattia / niv

Easy dependency management for Nix projects
https://github.com/nmattia/niv
MIT License
1.52k stars 74 forks source link

Extendable updating/fetching #300

Open shajra opened 3 years ago

shajra commented 3 years ago

Hi.

It occurred to me that there's a whole slew of "enhancement" fetching issues already in the tracker:

A few observations:

As @michaelpj pointed out in the comments, there's some possibility for extension via a Nix expression (#137). But the logic for updating is still part of the Haskell code of Niv. It would be nice to see if we could factor out updating to non-Haskell as well.

I think fetching is more obviously solved with a Nix expression, because fetching in the Nix work is a pure function.

Updating, is a bit more tricky because of side-effecting, specifically reaching out to the internet and getting an answer that might change from moment to moment. But one of the things that Nix can produce is not only a function, but a runCommand-style script. This script can function like IO in the Haskell world. And I think that gives us the full flexibility we need to make a Nix-based extension system that covers both fetching and updating.

This approach to extension would be similar to what Direnv does. Like Direnv, you could continue to have built-in support for the things that the most users will benefit from. But also, by having community support with these extensions, you will also get some inspiration for what's most useful, and how to implement it. One can feed the other.

What do you think?

michaelpj commented 3 years ago

I still think the approach I suggested in https://github.com/nmattia/niv/pull/137 was nice for this kind of customization. You feed sources.nix an overlay when you import it and you can use that to define extra fetchers.

shajra commented 3 years ago

@michaelpj I scanned the PR there. It's good to know it's there. I missed that feature.

From what I saw, this seems to address fetching, but not updating, right? I think in my original issue, I didn't make clear that updating is something that I'm really interested in solving for as well, because it's a large portion of what I find Niv useful for. Otherwise, I'm just crawling a lot of websites.

But if your approach covers updating as well, then I'm pretty happy. Just need to play around with it and see how it works.

michaelpj commented 3 years ago

Yes, you're quite right, it only handled fetching! At the moment I think all the updating is done by the Haskell code, which makes it a bit harder to extend in an ad-hoc way.

michaelpj commented 3 years ago

Also, to be clear, that was just a suggestion, it's not actually implemented in niv.

shajra commented 3 years ago

Yes, you're quite right, it only handled fetching! At the moment I think all the updating is done by the Haskell code, which makes it a bit harder to extend in an ad-hoc way.

Okay, I think we're converging on the same sentiments. One reason to use a shell breakout is because it allows us to do just a little bit more... specifically the stuff that's in Haskell for updating.

But I don't want to completely cut Haskell out of the equation. I think for the common cases, it's perfectly reasonable to not have to call shell at all. Shell would just be an alternate path.

I'll update the description of this issue to make clear my interest in updating in addition to fetching.

shajra commented 3 years ago

@michaelpj since you're involved in the Haskell.nix world, you may appreciate that the interest in this ticket butts up against my interest in better automating something like the management of materializations for a Haskell.nix project. The materialization scripts are similar in spirit to the scripts I proposed in this issue's description. Except... Haskell.nix scripts actually modify the filesystem. I was thinking originally of scripts that didn't modify the filesystem, but output JSON to then be injected back into sources.json (possibly with some transformation). Also, Haskell.nix is kind of two-stage (I think)... one to get the hash of the new materialization, and another to save it to the filesystem.

Ultimately, I want happy-path, push-button updating of Nix projects, with good error reporting when things go wrong. At that point, have I gone way past the original purview of Niv? Am I really going done that path of making a different tool entirely that may subsume some of the functionality of Niv?

shajra commented 3 years ago

I was also reminded that @expipiplus1 also maintains an application that's in this space... https://github.com/expipiplus1/update-nix-fetchgit

I feel that the approach there of syntactically matching Nix expressions might be too fragile for what I have in mind, though. I can appreciate the appeal of hitting the 80% case. I just find myself in the 20% case a lot more with my personal projects (Haskell.nix being one of those situations for me).

colemickens commented 3 years ago

I also would like this, and is part of the reason that I cannot adopt niv for nixpkgs-wayland. nixpkgs-wayland contains a script, update.shthat usesnix-prefetch` to update packages sources and hashes (including cargoSha256 and vendorSha256) in a fully automated, CI-driven fashion.

If niv supported extensible fetching/updating, then I could probably wire up nix-prefetch and then use niv to drive most of the updating process.