Run Experimental SDK in cluster

[lyoshenka]

requirements

• experimental sdk is running in production cluster
• if experimental sdk errors or is unresponsive, this does not affect performance for live users
• sdk team can run a command to deploy new versions of sdk without involving andrey

non-requirements (for now)

• understandable log messages. sdk team will deal with messages it does not understand
• fast deploy/restart time.
• stable sdk. it may error or freeze. it may also be killed at any time by lbrytv or k8s or andrey if they detect that it poses a threat to performance for live users
• too much of andrey's attention. if he has to spend a lot of time on outages or debugging, experimental sdk will be shut down until issues are addressed

punchlist

• [ ] figure out how to deploy nvme nodes, as required by sdk hardware requirements
• [ ] decrease timeout on lbrytv-api requests to experimental sdk so they don't pile up if sdk is unresponsive. this can be very conservative to start.
• [ ] give sdk team the process for how to deploy and monitor experimental sdk
• [ ] sdk team cleans up logging at least enough for andrey to know what's going on as he's managing the cluster

[lyoshenka]

talked to @andybeletsky. new plan:

• sdk team runs and manages full nodes on their own, outside cluster. set up several servers like the one we have, for redundancy
• andrey needs
  • instructions on how to run and configure light clients
  • where wallets are stored (filesystem?)
  • whats the wallet loading procedure? this is hardcoded in lbrytv rn and may be the biggest chunk of work for andrey. would be good to know this so he can estimate his time

[github-actions[bot]]

This issue hasn't seen any action in two months, please consider making progress on it or closing if no longer necassary