oceanprotocol / ocean-node

Apache License 2.0
22 stars 8 forks source link

Errors during node start #627

Closed hashfriend closed 1 week ago

hashfriend commented 3 weeks ago

Sorry for random reporting but have some errors shown in logs when starting up, think needs more docs on required env vars in combination with type of deployment. Docs say Docker is prefered and PRIVATE_KEY is only required env var, but in env var docs more env vars are said to be set if using Docker. So which ones?

For reference, running on Ubuntu 22.04 with Docker, all in default state with latest apt update and some basic server hardening. This also includes running ufw as firewall.

Starting all up with:

docker run -e PRIVATE_KEY=0xxxxx ocean-node:mybuild

First error I ghet is because of missing RPCS env var, preventing indexer from running:

error: CONFIG: ❌ Missing or Invalid RPCS env variable format, Running node without the Indexer component...

So suppose RPCS is required? So which networks are required then?

As for those warnings:

warn: CONFIG:  Missing "INTERFACES" env variable. Will use defaults...
warn: CONFIG:  ⚠ ️ Invalid or missing "AUTHORIZED_DECRYPTERS" env variable...
warn: CONFIG:  ⚠ ️ Invalid or missing "ALLOWED_VALIDATORS" env variable...
warn: CONFIG:  ⚠ ️ Invalid or missing "P2P_BOOTSTRAP_NODES" env variable...
warn: CONFIG:  ⚠ ️ Invalid or missing "P2P_ANNOUNCE_ADDRESSES" env variable...
warn: CONFIG:  ⚠ ️ Invalid or missing "P2P_FILTER_ANNOUNCED_ADDRESSES" env variable...
warn: CONFIG:  Missing "FEE_AMOUNT" env variable. Will use defaults...
warn: CONFIG:  Missing "FEE_TOKENS" env variable. Will use defaults...
warn: CONFIG:  ⚠ ️ Invalid or missing "OPERATOR_SERVICE_URL" env variable...
warn: CONFIG:  ⚠ ️ Invalid or missing "ALLOWED_ADMINS" env variable...
warn: CONFIG:  Missing "MAX_REQ_PER_SECOND" env variable. Will use defaults...
warn: CONFIG:  ⚠ ️ Invalid or missing "RATE_DENY_LIST" env variable...
warn: CONFIG:  ⚠ ️ Invalid or missing "UNSAFE_URLS" env variable...

Using defaults when env vars are not set probably should not trigger a warning (only log level?) but mayvbe that's just me. As far as I see all seems fine despite them not set?

I then tried to check if node is available following networking docs but have no success. I tried to run on server:

curl http://localhost:8000/getP2pPeer?peerId=16Uiu2HAkwWe6BFQXZWg6zE9X7ExynvXEe9BRTR5Wn3udNs7JpUDx

and get

curl: (7) Failed to connect to localhost port 8000 after 0 ms: Connection refused

And from outside:

curl http://myip:8000/getP2pPeer?peerId=16Uiu2HAkwWe6BFQXZWg6zE9X7ExynvXEe9BRTR5Wn3udNs7JpUDx

But can't get it to connect to anything. Is this expected and is my node still accessbile by other peers?

Firewall is setup like so:

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere                  
123/udp                    ALLOW       Anywhere                  
8000                       ALLOW       Anywhere                  
22/tcp (v6)                ALLOW       Anywhere (v6)             
123/udp (v6)               ALLOW       Anywhere (v6)             
8000 (v6)                  ALLOW       Anywhere (v6)

Any more ports I should open?

In the end I see stuff like this so it looks like all is running fine?

2024-08-17T09:17:33.674Z info: P2P:     P2P:    Incoming connection from peer 16Uiu2HAmLhRDqfufZiQnxvQs2XHhd6hwkLSPfjAQg1gH8wgRixiP
2024-08-17T09:17:33.675Z info: P2P:     P2P:    Using /dns4/node1.oceanprotocol.com/tcp/9000/p2p/16Uiu2HAmLhRDqfufZiQnxvQs2XHhd6hwkLSPfjAQg1gH8wgRixiP
2024-08-17T09:17:33.813Z info: P2P:     P2P:    Performing P2P task: {"command":"status","node":"16Uiu2HAkvsFbRdqezR3YNe61PocYB34Uv4AvWxu9VZPAhAj2dTMd"}
2024-08-17T09:17:33.816Z info: CORE:    Checking received command data for Command "status": {
    "command": "status",
    "node": "16Uiu2HAkvsFbRdqezR3YNe61PocYB34Uv4AvWxu9VZPAhAj2dTMd"
}
2024-08-17T09:17:33.818Z info: CORE:    CORE:   Command status started execution...
2024-08-17T09:18:33.729Z info: P2P:     P2P:    Incoming connection from peer 16Uiu2HAmLhRDqfufZiQnxvQs2XHhd6hwkLSPfjAQg1gH8wgRixiP
2024-08-17T09:18:33.730Z info: P2P:     P2P:    Using /dns4/node1.oceanprotocol.com/tcp/9000/p2p/16Uiu2HAmLhRDqfufZiQnxvQs2XHhd6hwkLSPfjAQg1gH8wgRixiP
2024-08-17T09:18:33.838Z info: P2P:     P2P:    Performing P2P task: {"command":"status","node":"16Uiu2HAkvsFbRdqezR3YNe61PocYB34Uv4AvWxu9VZPAhAj2dTMd"}
2024-08-17T09:20:33.838Z error: P2P:    Caught "Cannot read properties of undefined (reading 'search')" on republishStoredDDOS()

Finally as for system requirements, seeing that readme now says "latest iOS". As there is no way to have any process of any app running on iOS indefinetly in the background, this seems like a wrong suggestion. I don't see how any mobile OS could be used as an always online network node unless with a custom rooted ROM. Am I missing something?

hashfriend commented 3 weeks ago

I see that docs are getting scattered in folders and subfolders and important infio buried within threads in Discord. Why not update the actual readme and docs folder? I do not have time to hang out in all your channels for info which should be part of a single page doc.

Is /docs the most up to date documentation to get the node running, or is it /deployment? Or is it somewhere in the Discord threads? Or on docs.oceanprotocol.com which surprisingly has no mention of how to run a node whatsoever? Main readme does not mention anything for /deployment and for proper DX following existing project structure this folder should not exist. /script and /docs are already there, now you mix it up in a new /deployment folder. What else you expect developers to do apart from deployment that it justifies its own folder adding to the confusion?

Teams with a way lower market cap do it like this and do not require their community to hunt for information in multiple folders & files. One single source of information. I am sure your UX people can help you with that https://docs.streamr.network/guides/how-to-run-streamr-node/

As for deployment script, new questions and findings:

Regardless, when following the rest of /docs after following the script in /deployment, I can't get info out of my node either:

$ curl http://localhost:8000/getP2pPeer?peerId=16Uiu2HAkwWe6BFQXZWg6zE9X7ExynvXEe9BRTR5Wn3udNs7JpUDx
null

Also not showing up on nodes.oceanprotocol.com which is a resource not mentioned at all in any of the multiple places you call docs. I guess I need to read 100 messages in multiple Discord threads to get info to run a node. Highly frustrating, especially considering I was trying to run a node before you even announced it but because of missing and confusing docs my node was not reachable as it seems and despite reporting in here nobody answered

update: got it working after investing another hour by ignoring the main readme, nuking everything and starting with the new install script. Figured nodes.oceanprotocol.com is not searchable by IP address. My node is now showing up but out of the box indexer is still not running. In the indexer section on my dashboard I see

Indexing queue is empty.
Node is not running Indexer. No need to get queue at this point!

Which probably is caused by:

error: CONFIG:  ❌ INDEXER_NETWORKS is an empty array, Running node without the Indexer component...

Yet by default this env var is not set in what is advertised as the recommended way to run a node. Also INDEXER_NETWORKS is not documented anywhere. Is running indexer required for incentives eligibility?

Additionally, none of the test commands listed in the networking docs are working, they all return null

As for nodes.oceanprotocol.com, my node's data has not moved for multiple days, and is still showing older version despite multiple version updates.

hashfriend commented 1 week ago

giving up here as team has signaled that noisy Discord threads are the preferred way of bug reporting where team members actually answer.