search

oceanprotocol / pdr-backend

Instructions & code to run predictoors, traders, more.
Apache License 2.0
26 stars 22 forks source link

Barge issues: pdr-trueval, connecting subgraph #194

Closed trentmc closed 11 months ago

trentmc commented 11 months ago

See attached log. temp.txt

trentmc commented 11 months ago

One more error:

ocean-ocean-contracts-1  | [Error: EBUSY: resource busy or locked, rmdir '/ocean-contracts/artifacts'] {
ocean-ocean-contracts-1  |   errno: -16,
ocean-ocean-contracts-1  |   code: 'EBUSY',
ocean-ocean-contracts-1  |   syscall: 'rmdir',
ocean-ocean-contracts-1  |   path: '/ocean-contracts/artifacts'
ocean-ocean-contracts-1  | }
ocean-ocean-contracts-1  | Warning: Visibility for constructor is ignored. If you want the contract to be non-deployable, making it "abstract" is sufficient.
ocean-ocean-contracts-1  |   --> contracts/utils/OceanToken.sol:33:5:
ocean-ocean-contracts-1  |    |
ocean-ocean-contracts-1  | 33 |     constructor(
ocean-ocean-contracts-1  |    |     ^ (Relevant source part starts here and spans across multiple lines).
alexcos20 commented 11 months ago

One more error:

ocean-ocean-contracts-1  | [Error: EBUSY: resource busy or locked, rmdir '/ocean-contracts/artifacts'] {
ocean-ocean-contracts-1  |   errno: -16,
ocean-ocean-contracts-1  |   code: 'EBUSY',
ocean-ocean-contracts-1  |   syscall: 'rmdir',
ocean-ocean-contracts-1  |   path: '/ocean-contracts/artifacts'
ocean-ocean-contracts-1  | }
ocean-ocean-contracts-1  | Warning: Visibility for constructor is ignored. If you want the contract to be non-deployable, making it "abstract" is sufficient.
ocean-ocean-contracts-1  |   --> contracts/utils/OceanToken.sol:33:5:
ocean-ocean-contracts-1  |    |
ocean-ocean-contracts-1  | 33 |     constructor(
ocean-ocean-contracts-1  |    |     ^ (Relevant source part starts here and spans across multiple lines).

that is just a warning, you can ignore it

trentmc commented 11 months ago

address.json.txt

alexcos20 commented 11 months ago

See attached log. temp.txt

Looks like pdr-backend is trying to connect to "172.15.0.15:8000". But on MacOs, container do not have ip addresses, so we need to find another way to connect to subgraph (maybe something like host.docker.internal:8000 ?). I don't use a Mac, so I cannot test it

alexcos20 commented 11 months ago

address.json.txt

all good, all contracts are deployed on "development" network

trizin commented 11 months ago

I don't see any log messages from the subgraph, could it be down? Could you check the subgraph logs?

trentmc commented 11 months ago

I rebooted everything, cleaned everything, it's working now

My guess it was a subgraph issue. But now it's all good.

Thanks everyone for your help.

trentmc commented 11 months ago

So I had it going, good.

Then I stopped it, and re-started it. Now it's failing again :(. Looks like a subgraph error. Here's the log from the subgraph docker process:

...
- Generate types for GraphQL schema
✔ Generate types for GraphQL schema

Types generated successfully

> ocean-subgraph@3.0.9 create:local-barge
> graph create oceanprotocol/ocean-subgraph --node http://172.15.0.15:8020

- Creating subgraph in Graph node: http://172.15.0.15:8020/
✖ HTTP error creating the subgraph: EHOSTUNREACH
/usr/src/app/node_modules/@oclif/core/lib/errors/index.js:21
    throw new exit_2.ExitError(code);
    ^

ExitError: EEXIT: 1
    at Object.exit (/usr/src/app/node_modules/@oclif/core/lib/errors/index.js:21:11)
    at CreateCommand.exit (/usr/src/app/node_modules/@oclif/core/lib/command.js:131:23)
    at /usr/src/app/node_modules/@graphprotocol/graph-cli/dist/commands/create.js:46:22
    at ClientHttp.Client._parseResponse (/usr/src/app/node_modules/jayson/lib/client/index.js:190:12)
    at /usr/src/app/node_modules/jayson/lib/client/index.js:149:10
    at ClientRequest.<anonymous> (/usr/src/app/node_modules/jayson/lib/client/http.js:100:7)
    at ClientRequest.emit (node:events:513:28)
    at Socket.socketErrorListener (node:_http_client:494:9)
    at Socket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:157:8) {
  oclif: { exit: 1 },
  code: 'EEXIT'
}
trizin commented 11 months ago

Looks like the subgraph is unavailable, could you check if it's running and inspect the logs if it's running?

trentmc commented 11 months ago

Yes, exactly, the subgraph is unavailable.

That error listed above was from the docker subgraph process.

Here's the "liveness" of the process:

trentmc@tlm-macbook: ~/code/barge $ docker ps -a|grep ubgr
94d6837431e7   oceanprotocol/subgraph:predictoor          "/usr/src/app/docker…"   7 hours ago   Up 7 minutes                                                                                         ocean-subgraph-1

And, from trentmc@tlm-macbook: ~ $ docker logs 94d6837431e7:

trizin commented 11 months ago

Thanks for providing the logs. Looks interesting, I haven't seen something like this before. Could be related to what @alexcos20 mentioned above https://github.com/oceanprotocol/pdr-backend/issues/194#issuecomment-1746954677 although it's weird that it works from time to time and then fails

trentmc commented 11 months ago

[Berkay] Could be related to what @alexcos20 mentioned

[what @alexcos20 mentioned] But on MacOs, container do not have ip addresses, so we need to find another way to connect to subgraph (maybe something like host.docker.internal:8000 ?)

Yet this was working two weeks ago. And earlier today, if briefly.

Perhaps it's the following. There are two possible subgraph URLs:

export SUBGRAPH_URL="http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph"
export SUBGRAPH_URL="http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph"

IIRC one was working (top one?) and the other wasn't.

trizin commented 11 months ago

Yep first one should work. However this error occurs in the subgraph container while deploying the subgraph. Perhaps try a clean start by running the cleanup script and removing ~/.ocean folder?

trentmc commented 11 months ago

Perhaps try a clean start by running the cleanup script and removing ~/.ocean folder?

Yep, WIP.

I hit another snag: earlier today "to be prudent" I upgraded my Docker from 4.22 to 4.24. Then Docker kept randomly hanging. After googling around, it seems many people had that issue, and the fix was to go back to 4.22.

I'll keep you posted.

trentmc commented 11 months ago

Docker seems to be behaving better now.

However, the oceanprotocol/subgraph:predictoor container has this as its full log. Something's up I think?

trentmc@tlm-macbook: ~ $ docker logs -f 204871861e18
deploy subgraph is true
Waiting for contracts to be deployed

> ocean-subgraph@3.0.9 quickstart:barge
> node ./scripts/generatenetworkssubgraphs.js development && npm run codegen && npm run create:local-barge && npm run deploy:local-barge

Using custom ADDRESS_FILE instead of ocean-contracts npm dep
Skipping mumbai
Skipping polygon
...
Skipping sepolia
Skipping oasis_saphire

> ocean-subgraph@3.0.9 codegen
> graph codegen --output-dir src/@types

    Error: ENOENT: no such file or directory, open 'subgraph.yaml'
    Code: ENOENT
trentmc commented 11 months ago

Good news! I think in the previous run, while I'd run the cleanup script, I had missed deleting ~/.ocean. Because in my next run, I did do that and I think it's working.

Some logging from subgraph container is below. Fingers crossed...

...
Skipping oasis_saphire
Creating subgraph.yaml for development
     Adding veOCEAN

> ocean-subgraph@3.0.9 codegen
> graph codegen --output-dir src/@types

- Apply migrations
...
- Generate types for GraphQL schema
✔ Generate types for GraphQL schema

Types generated successfully

> ocean-subgraph@3.0.9 create:local-barge
> graph create oceanprotocol/ocean-subgraph --node http://172.15.0.15:8020

- Creating subgraph in Graph node: http://172.15.0.15:8020/
Created subgraph: oceanprotocol/ocean-subgraph

> ocean-subgraph@3.0.9 deploy:local-barge
> graph deploy oceanprotocol/ocean-subgraph subgraph.yaml -l $npm_package_version --ipfs http://172.15.0.16:5001 --node http://172.15.0.15:8020
...
  Compile data source: veFeeDistributor => build/veFeeDistributor/veFeeDistributor.wasm
- Compile subgraph
...
- Upload subgraph to IPFS
                .. QmVHM3xfMTNQUrC3psri54SeFMpQZSEWMH1X7tdM6CtMXX
- Upload subgraph to IPFS
✔ Upload subgraph to IPFS

Build completed: QmQfEowKwF6kJUXRGrjx8usvrpV8hw3nyW57QTAv5HHxF4

- Deploying to Graph node http://172.15.0.15:8020/
Deployed to http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph/graphql

Subgraph endpoints:
Queries (HTTP):     http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph

Summary of lessons so far:

trizin commented 11 months ago

Great!

trentmc commented 11 months ago

Alas, I am still having trouble getting predictoor to see the subgraph.

In console, I do the following:

cd code/pdr-backend/

# Setup virtualenv
cd pdr-backend
source venv/bin/activate

# Set envvars
export ADDRESS_FILE="${HOME}/.ocean/ocean-contracts/artifacts/address.json"
export RPC_URL=http://127.0.0.1:8545
export SUBGRAPH_URL="http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph"
export PRIVATE_KEY="0xc594c6e5def4bab63ac29eed19a134c130388f74f019bc74b8f4389df2837a58"

# run random predictoor bot (agent)
python pdr_backend/predictoor/main.py 1

And the result I get from the last line is:

HTTPConnectionPool(host='172.15.0.15', port=8000): Max retries exceeded with url: /subgraphs/name/oceanprotocol/ocean-subgraph (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x1081a6690>, 'Connection to 172.15.0.15 timed out. (connect timeout=1.5)'))
No feeds found. Exiting

Also, the subgraph url of interest does not load in my browser. http://172.15.0.15:8000/subgraphs/name/oceanprotocol/ocean-subgraph

Help? @trizin

trizin commented 11 months ago

Try to use the localhost:9000 for subgraph url

trentmc commented 11 months ago

Try to use the localhost:9000 for subgraph url

Aha! And that works! My predictoor log:

(venv) trentmc@tlm-macbook: export SUBGRAPH_URL="http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph"

(venv) trentmc@tlm-macbook: ~/code/pdr-backend $ python pdr_backend/predictoor/main.py 1

--------------------------------------------------------------------------------
Config:
PredictoorConfig1={private_key=0xc594c6e5def4bab63ac29eed19a134c130388f74f019bc74b8f4389df2837a58, rpc_url=http://127.0.0.1:8545, s_until_epoch_end=60, stake_amount=1, subgraph_url=http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph, _abc_impl=<_abc._abc_data object at 0x104c9ba40>, owner_addresses=[], pair_filters=[], source_filter=[], timeframe_filter=[], web3_config=<pdr_backend.util.web3_config.Web3Config object at 0x101bf3f50> /PredictoorConfig1}

................................................................................
Feeds (detailed):
...

Starting main loop...

--------------------------------------------------------------------------------
Take_step() begin.
  block_number=793, prev=0
  Got new block. Timestamp={block['timestamp']}
    Process [Feed 0x0356a ETH/USDT|binance|5m] at epoch=5655343
      282 s left in epoch (predict if <= 60 s left)
      Done feed: too early to predict
    Process [Feed 0x4bde0 BTC/TUSD|binance|5m] at epoch=5655343
...

And that url (http://localhost:9000/subgraphs/name/oceanprotocol/ocean-subgraph) also works in my browser, bringing up the expected text boxes for entering queries.

trentmc commented 11 months ago

This issue is solved, hooray:)

trentmc commented 11 months ago

@trizin so that I can update my own mental model:

What's the relation between these urls? Why is it a remote IP address for the docker container, and a local (IP address?) for the agent & browser? Thanks!

alexcos20 commented 11 months ago

because docker on macOS does not support brige, which is the driver that we are using to create and assign ip addrs to all containers (https://docker-docs.uclv.cu/docker-for-mac/networking/#there-is-no-docker0-bridge-on-macos)

alexcos20 commented 11 months ago

and https://docker-docs.uclv.cu/docker-for-mac/networking/#per-container-ip-addressing-is-not-possible

trentmc commented 11 months ago

Thanks Alex!

trizin commented 11 months ago

According to my knowledge:

https://github.com/oceanprotocol/barge/blob/main/compose-files/thegraph.yml#L6 This exposes the port 8000 in the container to the host network, thus it's accessible.

"172.15.0.15" is the internal ip of the container, other containers in the network can access it. As far as I know the docker network is isolated from host network unless you expose a port so you cannot access it.