search

plasma-group / plasma-chain-operator

A one-click deploy Plasma chain
MIT License
33 stars 25 forks source link

Add Operator cloud deployment #30

Open karlfloersch opened 5 years ago

karlfloersch commented 5 years ago

Scaling Plasma

Because security of the chain is ensured by the exit game, it is safe for operators to scale past the computational limits of consumer hardware. Now that we have a reasonable Plasma implementation, I think it's time to see just how far we can push this thing!

Current Architecture

The Plasma Chain operator as it stands is a single node.js application. However, internally it has three separate processes:

  1. Express API / Eth Service -- Parent process
  2. State Manager -- Child 1, makes use of a LevelDB database & an append only log
  3. Block Manager -- Child 2, makes use of a separate LevelDB database & ingests the append only log

This separation of concerns is critical to the long-term scalability of Plasma Operators as it allows for different functions to be easily processed in parallel. Each component when benchmarked alone can already achieve thousands of transactions per second, & with a little love I'm sure could go way faster.

The following is a rough diagram of what you can find in: https://github.com/plasma-group/plasma-chain-operator

untitled diagram 4 -page-1

Hand-wavy AWS toolstack architecture

This is a rough outline of how one might port this design to a cloud provider, in this case AWS. The general structure remains the same; however, message passing is more sophisticated (rabbitmq) and signature checking is trivially parallelizable (aws lambda). The significant downside of this architecture is it's expensive, so if there are better approaches I'd love to hear them.

untitled diagram 4 -page-2 1

The ask

The ask for this issue is to take the first steps to realize the vision of a more scalable operator. That means breaking out the API, tx-log, state, and block managers into their own Docker containers, handle message passing between them with some message queue, and set up some deployment script.

That said, comments on the overall architecture & approach to scaling are more than welcome. I've had very little peer review!! eek!

Hugs! 😊

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 1000.0 DAI (1000.0 USD @ $1.0/DAI) attached to it.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Workers have applied to start work.

These users each claimed they can complete the work by 2 months, 4 weeks from now. Please review their action plans below:

1) mul1sh has applied to start work _(Funders only: approve worker | reject worker)_.

I'll break down the plasma chain operator into different docker containers as required. I'll then use either apache kafka or rabbitmq to handle the queue of messages between the various containers.

Finally i'll come with a simple and clear bash script that ties in all the above components together for easy deployment. Looking at the workload i can confidently say i'll get this done in less than a week. Thanks

Learn more on the Gitcoin Issue Details page.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 3 weeks, 6 days ago. Please review their action plans below:

1) websoftwares has applied to start work _(Funders only: approve worker | reject worker)_.

Hello

I have read your project description and it looks absolutely interesting, i have experience in CI/CD in various programming languages and also a long development career.

I have deployed in lambda various projects also with low costs by choosing the right strategy (lambda@edge/lambda step/lambda functions/etc) in combination with containers on ECS with auto-scaling.

Let me know if you want to have a conversation to talk further in details.

Greetings Boris

Learn more on the Gitcoin Issue Details page.

2) websoftwares has applied to start work _(Funders only: approve worker | reject worker)_.

Hello

I have read your project description and it looks absolutely interesting, i have experience in CI/CD in various programming languages and also a long development career.

I have deployed in lambda various projects also with low costs by choosing the right strategy (lambda@edge/lambda step/lambda functions/etc) in combination with containers on ECS with auto-scaling.

Let me know if you want to have a conversation to talk further in details.

Greetings Boris

Learn more on the Gitcoin Issue Details page.

3) mathewdgardner has applied to start work _(Funders only: approve worker | reject worker)_.

I work with node, rabbitmq and kubernetes with eks and aks daily. Just this week I configured rabbitmq (for node any python services) to run in a container using a persistent volume claim so that we could have durable queues and persistent messages while still sitting inside kube and it's still very fresh in my mind.

At my day job I own the stack so that means I write code, then see it through to production. I look to automate things for tight feedback loops so scripting won't be an issue.

This would be my first bounty so I'm eager to see this come together!

Learn more on the Gitcoin Issue Details page.

4) voxelot has been approved to start work.

Just finished reviewed the repo and it looks pretty straightforward for what needs to be re-worked to convert from IPC to AMQP or SQS. S3 based block ingestion should also use SQS to react to S3 events. If using other AWS native features like Lambda, I think it makes sense to stick with cloud-native managed services like SQS as we'd would definitely cut costs and complexity in comparison to AMQP / Kafka.

The requirements for this task could shift a bit as other new features are added, so we'll need to lock down the scope of this task so we have a good starting point to grow from. Here's some issues I came across that may depend on this task later on:

23 - Would require extra S3 logic to reingest the txlogs

37 - May entail config management for the docker containers (e.g. confd / AWS parameter store)

I have experience with cloudfront/terraform, AWS, Docker, K8S, Fargate and picture this repo morphing into something easy to use like the Serverless CLI but one that manages containers and resources for plasma operators.

Let me know if you'd like to keep the existing IPC interfaces as a fallback for spinning up local environments without any remote/docker/AWS dependencies.

Learn more on the Gitcoin Issue Details page.

spm32 commented 5 years ago

You're good to go @voxelot!

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

Voxelot commented 5 years ago

@karlfloersch could I get a 5-10 day snooze on this bot? Also is there a gitter or slack available for more fluid conversation?

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

gitcoinbot commented 5 years ago

@voxelot Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

ghost commented 5 years ago

@karlfloersch , i noticed that this got stuck, due to changes on the main code.

Question: are you interested to get a 2nd top-level architectural 2nd opinion / processing suggestions here? Could be done with a bounty budget of 1K.