[Predictoor bot] Make it easy to deploy/monitor/manage 20 dynamic-model predictoor bots on mainnet

[trentmc]

Summary

Problem: "I just want a very simple, naive, cheap, e2e setup that enables me to put 0.1 OCEAN on the line." --Roberto

Solution: Update READMEs to show how to run 20 bots via PM2.

Background / Motivation

We want to deploy/monitor/manage 20 dynamic-model predictoor bots on mainnet.

The current predictoor bot code can do this, but has negatives (details below). This README describes options, analyzes them, and proposes a solution.

Criteria

• Easy to deploy. Low effort to launch remote process(es).
• Easy to monitor. Easy to track behavior and performance of each feed. Includes via logs & maybe analytics for model-building, prediction accuracy, etc.
• Easy to manage. Easy to see (remote) processes: whether live, cpu usage, memory usage etc. Easy to stop and re-start a single feed. Doesn't fail if one feed fails.
• Unique eth accounts. We need 1 account per feed, to avoid txs from a single account piling up in the queue.
• Scalable compute. Fully leverage >>1 processors/boxes for the compute needs. If we don't do this, we'll likely have get a bottleneck in data gathering, model building, or model inference.

Candidate Approaches, Analysis

Run one bot across 20 feeds. Status quo in README and pdr_backend/predictoor/ code.

• Pros: easy to deploy and monitor.
• Cons: not easy to stop and re-start a single feed. Doesn't currently have unique eth accounts, but could be built in. Compute scales if sklearn modeling is multi-threaded or multi-process; but this is partly satisfactory at best.

Run one bot per feed, via Kubernetes. Status quo of Berkay's running. pdr_backend/predictoor/ code supports running one feed in a bot. This slack thread between Roberto & Berkay has details.

• Pros: K8S cluster mgmt means can deploy, can monitor, can manage (but not easy). Easy to do unique eth accounts and scalable compute.
• Cons: K8S has high friction to onboard and use. Not easy for juniors

Run one bot per feed, via PM2. Status quo of Roberto's running. pdr_backend/predictoor/ code supports running one feed in a bot. READMEs have some PM2 guidance, and we can add more. Details in Appendix.

• Pros: PM2 cluster mgmt means can deploy, can monitor, can manage -- easily in PM2 (!). Easy to do unique eth accounts and scalable compute.
• Cons: need to expand the "multi-bot" aspect. Roberto's notes don't show it

Recommend: the third approach! "Run one bot per feed, via PM2". But we need to ensure we have truly good guidance for >20 bots.

TODOs for this issue

Pre-requisites:

• [x] Do pdr-backend#258 first: "Remove onboarding friction: README goes straight to local bot / remote network.

Main work:

• [x] Add info run one bot with PM2, including PM2 configuration. Uses Roberto's info from Appendix. [commit]
• [ ] Have better guidance on running >1 bot, in line with the goals of this issue. (The Appendix does not give guidance)

Appendix: Roberto's approach for "Run one bot per feed, via PM2"

(This slack message / thread led to this github issue!)

Roberto in slack:

I created a helper config that makes it easier to manage agents.

If you want to be like Berkay and get started with 1 localnet predictoor:

1. Make sure you have pm2 installed
2. Drop this file into your /pdr-backend/ folder
3. Run predictoor agent 1 with: pm2 start localnet-predictoor.config.js
4. Monitor the agent: pm2 logs localnet-predictoor

You can now update your config file to use a different predictoor approach.

If you wanted to run 20 agents (5m + 1h) in production, you do it too starting with remotenet-predictoor.config.js file.

You will start by running testnet-BTC-5m, and can quickly change to production-btc-5m, also adding more agents

The JS files are in this zip file.

[trentmc]

UPDATE: with this commit have merged in code that includes Roberto's suggestions with PM2. Yay!

It also addresses Berkay's goal to have better PM2 config. Yay!

However, we still need better guidance on running >1 PM2 processes. Hence why this issue remains.

I updated the description accordingly.

[trentmc]

Newer, mostly overlapping issue: https://github.com/oceanprotocol/pdr-backend/issues/369

(Hence why I just closed this one)