Gateway node stability - Githubissues

livepeer / project-management

Livepeer open source ideas, dapps, and improvement proposals.

9 stars 2 forks source link

Gateway node stability #26

Open dob opened 6 years ago

dob commented 6 years ago

High level BLOCKER.

The new web player plays off a gateway node, but the Livepeer process crashes, or gets to a weird state where it needs to be restarted. There’s a number of underlying issues here, including the memory issue, buffering too many segments, stream restarts, etc. But a takeaway is that a more stable gateway node would make it a lot easier for folks to get started viewing their broadcasts without having to host their own embedded players.

https://github.com/livepeer/go-livepeer/issues/227

dob commented 6 years ago

@jozanza built a nice simple script that handles autorestarting of the node process, which mitigates this issue. I'm still going to leave this open as a blocker though as...

1) We've still seen the node get to a state where the gateway won't serve any new content until the node is restarted (even though it didn't crash). It's just much more rare now.

2) We don't really have proper monitoring on the process...just a simple while loop script which isn't fully reliable.

3) The memory issue still exists causing the process to not necessarily be long lived as it should be.

dob commented 6 years ago

Current update on this is that we're getting closer, but I don't think we close it until the memory issue fix is deployed.

ericxtang commented 6 years ago

I think even if the memory is fixed, we still have the bandwidth issue. Ideally we can implement a load balancing solution on a few different levels:

The CDN should be able to read from a load balancer across a few gateway nodes
We should have a player plugin that can talk to the Livepeer network and discover multiple "exit nodes" to load videos from