Allow ingot dependencies to be resolved via git, ipfs etc

[cburgdorf]

Allowing Fe to resolve ingots from remote locations

Fe currently supports ingot dependencies that already sit somewhere in the local file system. To use such a dependencies one has to specify it in the fe.toml.

Example:

name = "basic_ingot"
version = "1.0"

[dependencies]
basic_ingot_dep = "../basic_ingot_dep"

In this example, our Fe program depends on basic_ingot_dep which sits at ../basic_ingot_dep relative to the root (where the fe.toml sits.

It is currently already possible to define dependencies via the expanded syntax which currently only makes sense if one wants to specify a version e.g.

[dependencies]
basic_ingot_dep = {path = "../basic_ingot_dep", version = "1.0" }

I propose that we keep everything about that but also use the expanded syntax to allow to retrieve ingots through an open and flexible mechanism from remote sources such as Github or IPFS.

Example:

name = "basic_ingot"
version = "1.0"

[dependencies]
basic_ingot_dep = "../basic_ingot_dep"
other = { type = "git", path="../other", source = "https://github.com/something/other.git" }
more = { type = "ipfs", path="../more", source = "ipfs://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi" }

In the example above other is a dependency that would be retrieved from Github and stored locally under ../other and more is a dependency that would be retrieved via IPFS and stored under ../more.

A few short notes:

• It might make sense to eventually keep all dependencies under a common sub directory but for now I consider that out of scope to get things going
• Continuing on the previous point we might be able to infer the place for where to store the dependency locally instead of having the user to specify it via path. But that leads to questions about how to handle conflicts etc and so I'm inclined to just keep it simple and let the user tell us where to put it. We should probably put some constraints on that like not writing to places outside of the projects directory.
• Resolvers could over individual extra settings. For instance resolving a dependency via git could allow to specify a version, tag or branch. But again, I consider that to be out of scope for the first iteration.

[sbillig]

Useful references, maybe: https://github.com/rust-lang/cargo/blob/master/src/cargo/sources/git/mod.rs

https://blog.rust-lang.org/2023/01/10/cve-2022-46176.html

[cburgdorf]

Took a look at what Cargo does. I think they deal with a few more complexities that we don't have to care about and hence their git handling is a wild mix of using native git, git2 and gix with the aim to fully migrate to gix eventually.

I think with the stuff we care about we can just use gix right away. The main reason to use gix over git2 is that it allows us to do a shallow clone to only fetch the latest revision which is all we care about (and makes dependency resolving faster). Git2 has no support for shallow clones yet.

Making a shallow clone of a git remote roughly looks like this with gix:

fn clone(remote: &str, to: &str) -> Result<(), Box<dyn std::error::Error>>{
    let url = gix::url::parse(gix::bstr::BStr::new(remote.as_bytes()))?;
    let mut prep = gix::prepare_clone(url, to)?
                                    .with_shallow(gix::remote::fetch::Shallow::DepthAtRemote(NonZeroU32::new(1).unwrap()));
    let (mut rep, _) = prep.fetch_then_checkout(gix::progress::DoOrDiscard::from(Some(gix::progress::Discard)), &gix::interrupt::IS_INTERRUPTED)?;
    rep.main_worktree(gix::progress::Discard, &gix::interrupt::IS_INTERRUPTED)?;
    Ok(())
}