Modularize ops-bedrock devnet

[jinmel]

Is your feature request related to a problem? Please describe.

The current ops-bedrock devnet is defined in one docker-compose.yml file. This makes spinning up multiple L2s on devnet difficult. We need to have arbitrary number of L2s defined in devnet to test interoperability of opstack chains.

Describe the solution you'd like

Split the docker compose yaml file into base.yml defining containers for L1 layer, and overlays-l2.yml defining containers for L2 layer. Then we can do docker compose -f base.yml -f overlays-l2-A.yml -f overlays-l2-B ... The role and responsibility of the devnet launch script is to manage the meta data files into separate directories so that user can deploy arbitrarily number of L2s to the same devnet.

In order to deduplicate the container definitions in overlays-l2-x.yml, we use extends directive to inherit common configuration for any L2 layer, overriding chain specific metadata and configuration files for each L2 deployments

Describe alternatives you've considered

Defining additional L2 chains statically in docker compose file: It loses extensibility to multiple L2 chains if we are going to test interop message across arbitrary number of chains. If testing interoperability of two chains is enough, this could also be a reasonable choice since it incurs lesser change to current codebase

Additional context

[tynes]

Hey @jinmel, I would be very supportive of this change and would review a PR if you submitted it

[jinmel]

Hey @tynes, I was working on extending the current ops-bedrock docker compose project and felt that trying to fit multiple L2s into a single Docker Compose setup might not be very scalable and incurs some inevitable hacks such as writing a template for docker compose and generating with different container names. It seems like I might be reinventing the wheel, given what Kubernetes and Kustomize already accomplishes. What are your thoughts on keeping the current devnet as it is and creating a new directory with Kubernetes/Kustomize configurations? This way, people could easily extend it with their own components. For instance, we could run arbistack and opstack within the same network.

[tynes]

I would prefer to not create a completely new solution for a devnet as it would create additional maintenance overhead for the team. We used to have a minikube based local devnet but at some point we migrated to docker compose. We want to optimize the devnet for easy maintenance and easy + reliable usage for devs who develop their applications on top of it. I would prefer to implement something more minimal that may introduce some tech debt, for example scoping the implementation directly to only supporting 2 L2s. This would remove the need to implement arbitrary templating. Would this unblock your usecase? Then in the future, the project of further improving the devnet can be revisited

[jinmel]

@tynes Great point and thanks for the clear scope of this PR.

[jinmel]

@tynes I first tried to remove the allocs and make all the l1 and l2 genesis dynamic so that it is extensible to multiple L2s in future, but I just found out that allocs to genesis files is important for unitests here: https://github.com/ethereum-optimism/optimism/tree/develop/packages/contracts-bedrock#test-setup. I attempted to squeeze in two set of allocs into L1 genesis for two chains, but it seems to create more tech debt than actual gain. What would you say if we had a devnet launch script dedicated for cross chain testing only?

[tynes]

We spoke about a plan going forward:

• Create an interop devnet deploy script that uses the L1 contracts deploy twice
• Update the devnet python code in a minimal way to call the interop devnet deploy script, perhaps with an --interop flag
• Update the devnet python code so that it can call docker compose up twice to bring up 2 L2 systems

The goal is to very minimally modify the python devnet code, since it has a lot of tech debt. This will immediately unblock the development of applications that utilize interop.

For the interop deploy script, it should set the deploy script into state similar to the pattern here:

https://github.com/ethereum-optimism/optimism/blob/bd7ceb02ec9f6ef2e3332cbb28201a12977f3b60/packages/contracts-bedrock/scripts/Deployer.sol#L24-L26

This ensures that we don't have the compile overhead that comes with using CREATE directly in the code, resulting in bloated bytecode

For longer term, we can consider using kurtosis. This can serve the basis for a new devnet as well as a multi client test suite

[tedim52]

Here's a pull request that builds on top of @barnabasbusa work on the optimism-package, it allows you to spin up N optimism rollups locally: https://github.com/ethpandaops/optimism-package/pull/20. Hopefully this helps