fix(stages): Cloning Abstractions

[refcell]

Description

Inside the stages crate, there is a stage/cannon.rs stage for the cannon prestate generation, a stage/allocs.rs Allocs stage for devnet allocs, and a cloning stage to clone the optimism monorepo.

This is a naive, sequential pipeline that starts with the cloning of the monorepo and then proceeds with generating the prestate and allocs.

This ticket is to make the monorepo cloning configurable through the stack.toml, such that under a [stages] table, a user can define something like isolate=true to force each stage to use isolate environments. In an optimism-monorepo exclusive stack, this would force each stage to clone the monorepo and execute it's behavior.

For non-isolated stages, the prestate stage would come first and check that the optimism monorepo is cloned, cloning it if not and leaving it available for all subsequent stages.

[refcell]

See: https://github.com/anton-rs/op-up/pull/43#discussion_r1355765632

[merklefruit]

A couple extra thoughts:

• fetching data from the op monorepo is the best way to keep everything up to date without doing manual work here
• still, cloning the entire monorepo is slow, consumes a lot of bandwidth and disk, and most of the directories won't be used at all for the goal of opup anyway

There are a couple solutions:

• the isolate=true stages could specify which directories they need to clone from the monorepo, and we will use --sparse-checkout to fetch just that.
• or we can download the repo instead of git cloning it:
  • zipball: wget https://github.com/[user]/[repo]/archive/[branch].zip
  • tarball: curl -L http://github.com/[username]/[repo]/archive/[branch].tar.gz | tar zxf -
• we could create an op-up branch of the optimism monorepo, which essentially deletes everything we don't need, and downloading that branch will be faster and lighter (but it will have to be updated every once in a while)

[merklefruit]

so I did a test on the tarball method, here's the results:

wifi connection: not good not bad, 300mbps down

• git clone took 23s
• curl tarball took less than 3s

I didn't expect such a huge difference, I think this is an easy win if we need to have the monorepo locally.

Wdyt @refcell?

[refcell]

so I did a test on the tarball method, here's the results:

wifi connection: not good not bad, 300mbps down

• git clone took 23s

• curl tarball took less than 3s

I didn't expect such a huge difference, I think this is an easy win if we need to have the monorepo locally.

Wdyt @refcell?

This is awesome! Would probably make sense to have tarball fetching be the default, and then allow for recursive git cloning via a configurable just in case.

[merklefruit]

Sounds perfect