Ensure JSON RPC relay is usable

leninmehedy commented 6 months ago

solo allows us to deploy relays using command solo relay deploy -i node0,node1,node2. We need to ensure that the deployed relay is actually usable.

I have attempted run an existing smoke test that runs query through the relay, but so far it didn't work.

Here is how ran the test:

Deploy network and relays using solo(https://www.notion.so/swirldslabs/Technical-Workshop-Solo-8818724b161a4d7a970f48b840952d71)
Expose Hedera Explorer and ensure transactions and accounts are visible (https://www.notion.so/swirldslabs/Technical-Workshop-Solo-8818724b161a4d7a970f48b840952d71?pvs=4#79bf814ca85f4ae3b3bd13d127e8899b)
Expose a network-node-svc:50211 port
Expose relay port: 7546
Clone https://github.com/hashgraph/hedera-local-node 5a. Run npm install 5b. Run npm run generate-accounts 3 that would generate 3 keys with title Accounts list (Alias ECDSA keys) 5c. Update test/smoke/hardhat.config.js with two account private keys from the Alias ECDSA keys generated in step 5b
Go into test/smoke and run npm install and then run npx hardhat test

Ideally the smoke test should pass. However I am getting error like below:

➜  smoke git:(main) ✗ npx hardhat test
WARNING: You are currently using Node.js v21.7.1, which is not supported by Hardhat. This can lead to unexpected behavior. See https://hardhat.org/nodejs-versions

  Smoke Tests
    Relay
      1) Should perform a crypto transfer transaction
    Websocket server
      2) "before all" hook for "Should receive events for subscribed contract"

  0 passing (3s)
  2 failing

  1) Smoke Tests
       Relay
         Should perform a crypto transfer transaction:
     ProviderError: [Request ID: ab106718-64b1-4796-874a-02de4fd5df3b] Error invoking RPC: queryA and queryAaaa ENODATA fullstack-deployment-rest
      at HttpProvider.request (/Users/leninmehedy/code/src/github.com/hashgraph/hedera-local-node/test/smoke/node_modules/hardhat/src/internal/core/providers/http.ts:90:21)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at HardhatEthersProvider.getBalance (/Users/leninmehedy/code/src/github.com/hashgraph/hedera-local-node/test/smoke/node_modules/@nomicfoundation/hardhat-ethers/src/internal/hardhat-ethers-provider.ts:171:21)
      at Context.<anonymous> (/Users/leninmehedy/code/src/github.com/hashgraph/hedera-local-node/test/smoke/test/Smoke.js:17:39)

  2) Smoke Tests
       Websocket server
         "before all" hook for "Should receive events for subscribed contract":
     ProviderError: [Request ID: 48b12f92-c683-4d70-a790-5c2c38cff33e] Unknown error invoking RPC
      at HttpProvider.request (/Users/leninmehedy/code/src/github.com/hashgraph/hedera-local-node/test/smoke/node_modules/hardhat/src/internal/core/providers/http.ts:90:21)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at HardhatEthersProvider.getBlockNumber (/Users/leninmehedy/code/src/github.com/hashgraph/hedera-local-node/test/smoke/node_modules/@nomicfoundation/hardhat-ethers/src/internal/hardhat-ethers-provider.ts:134:25)
      at HardhatEthersSigner.sendTransaction (/Users/leninmehedy/code/src/github.com/hashgraph/hedera-local-node/test/smoke/node_modules/@nomicfoundation/hardhat-ethers/src/signers.ts:122:25)
      at ContractFactory.deploy (/Users/leninmehedy/code/src/github.com/hashgraph/hedera-local-node/test/smoke/node_modules/ethers/src.ts/contract/factory.ts:111:24)
      at Context.<anonymous> (/Users/leninmehedy/code/src/github.com/hashgraph/hedera-local-node/test/smoke/test/Smoke.js:39:16)

I have asked for help from mirror-node and hedera-local-node team and awaiting/trying the suggestions.

leninmehedy commented 6 months ago

Just got a clue that relay uses web3 service of mirror-node that I disabled recently without understanding its use. So need to enable web3 and try again.

JeffreyDallas commented 6 months ago

I started hedera local node , then use npx hardhat test to send test transactions, and got different errors


  Smoke Tests
    Relay
      1) Should perform a crypto transfer transaction
    Websocket server
      2) "before all" hook for "Should receive events for subscribed contract"

  0 passing (991ms)
  2 failing

  1) Smoke Tests
       Relay
         Should perform a crypto transfer transaction:
     ProviderError: [Request ID: 1f37dcc5-d0e6-448c-9ce0-e88fc83b9986] Requested resource not found. address '0x3Ed813A5ba1e165F815E96ce64a91AF9AB39F2CA'.
      at HttpProvider.request (/Users/jeffrey/hedera-local-node/test/smoke/node_modules/hardhat/src/internal/core/providers/http.ts:90:21)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at HardhatEthersSigner.sendTransaction (/Users/jeffrey/hedera-local-node/test/smoke/node_modules/@nomicfoundation/hardhat-ethers/src/signers.ts:125:18)
      at Context.<anonymous> (/Users/jeffrey/hedera-local-node/test/smoke/test/Smoke.js:20:11)

  2) Smoke Tests
       Websocket server
         "before all" hook for "Should receive events for subscribed contract":
     ProviderError: [Request ID: e037b4c6-5190-410b-bf34-e7083346ca07] Requested resource not found. address '0x3Ed813A5ba1e165F815E96ce64a91AF9AB39F2CA'.
      at HttpProvider.request (/Users/jeffrey/hedera-local-node/test/smoke/node_modules/hardhat/src/internal/core/providers/http.ts:90:21)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
      at HardhatEthersSigner.sendTransaction (/Users/jeffrey/hedera-local-node/test/smoke/node_modules/@nomicfoundation/hardhat-ethers/src/signers.ts:125:18)
      at ContractFactory.deploy (/Users/jeffrey/hedera-local-node/test/smoke/node_modules/ethers/src.ts/contract/factory.ts:111:24)
      at Context.<anonymous> (/Users/jeffrey/hedera-local-node/test/smoke/test/Smoke.js:39:16)

leninmehedy commented 6 months ago

@JeffreyDallas I believe the relay is not able to connect to the the mirror-node rest service within the kubernetes cluster. Not sure if it is a problem with relay itself, or I misconfigured the service endpoint for the rest service.

Relay "works" if I expose the mirror-node rest service to localhost and configure relay to hit using URL like localhost:7551. Here we're configuring it in solo: https://github.com/hashgraph/solo/blob/main/src/commands/relay.mjs#L46

Also we configure to hit the node directly instead of haproxy-svc, this is also something we need to discuss with Nathan: https://github.com/hashgraph/solo/blob/main/src/commands/relay.mjs#L73

If you are investigating this, please consult with @nathanklick for guidance as that is what I was planning to do before picking it up.

JeffreyDallas commented 6 months ago

I created a ticket in hedera-local-node repo after disucssing with smart contract team https://github.com/hashgraph/hedera-local-node/issues/610

JeffreyDallas commented 5 months ago

First part of smoke test passed Per Alfredo Gutierrez and Xin, second part of smoke test failed because there is no fee file and exchange rate file uploaded to mirror node yet.

mirror_web3.log

We need a command line tool to upload fee file and exchange rate file to mirror node but that is out of scope of this ticket

npx hardhat test

  Smoke Tests
    Relay
      ✔ Should perform a crypto transfer transaction (4035ms)
    Websocket server
      1) Should receive events for subscribed contract

  1 passing (16s)
  1 failing

  1) Smoke Tests
       Websocket server
         Should receive events for subscribed contract:
     Uncaught 
  AggregateError: 
      at internalConnectMultiple (node:net:1102:18)
      at afterConnectMultiple (node:net:1615:5)

JeffreyDallas commented 5 months ago

I have seen repo hedera-json-rpc-relay fixed its chart release workflow, so I tested with latest v0.48.0

but smoke test failed again with

│     err: {                                                                                                                                                                                                                           │
│       "type": "MirrorNodeClientError",                                                                                                                                                                                               │
│       "message": "Not found",                                                                                                                                                                                                        │
│       "stack":                                                                                                                                                                                                                       │
│           Error: Not found                                                                                                                                                                                                           │
│               at MirrorNodeClient.handleError (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:224:29)                                                                                                            │
│               at MirrorNodeClient.<anonymous> (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:206:22)                                                                                                            │
│               at Generator.throw (<anonymous>)                                                                                                                                                                                       │
│               at rejected (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:25:65)                                                                                                                                 │
│               at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                                                                                                                                          │
│       "statusCode": 404                                                                                                                                                                                                              │
│     }

If I removed --set config.MIRROR_NODE_AGENT_CACHEABLE_DNS=false then the smoke test failed differently

│     err: {                                                                                                                                                                                                                           │
    │       "type": "MirrorNodeClientError",                                                                                                                                                                                               │
    │       "message": "queryA and queryAaaa ENODATA fullstack-deployment-rest",                                                                                                                                                           │
    │       "stack":                                                                                                                                                                                                                       │
    │           Error: queryA and queryAaaa ENODATA fullstack-deployment-rest                                                                                                                                                              │
    │               at MirrorNodeClient.handleError (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:224:29)                                                                                                            │
    │               at MirrorNodeClient.<anonymous> (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:206:22)                                                                                                            │
    │               at Generator.throw (<anonymous>)                                                                                                                                                                                       │
    │               at rejected (/home/node/app/packages/relay/dist/lib/clients/mirrorNodeClient.js:25:65)                                                                                                                                 │
    │               at process.processTicksAndRejections (node:internal/process/task_queues:95:5)                                                                                                                                          │
    │       "statusCode": 567                                                                                                                                                                                                              │
    │     }

will continue to ask for assistant from smart contract team to see how to resolve this.

Ivo-Yankov commented 5 months ago

The issue is that the relay is unable to access the mirror node rest api with the provided url. I managed to get it to work with the following configuration in relay.mjs:

    ...
    valuesArg += ` --set config.MIRROR_NODE_URL=http://10.244.0.21:5551`
    valuesArg += ` --set config.MIRROR_NODE_URL_WEB3=http://10.244.0.22:8545`
    ....

Some important details:

the protocol needs to be http and not https
the IPs in the example are from the mirror node rest and web3 pods, but they are different every time. There is probably a better way to do it, but from my testing it didn't work with constants.FULLSTACK_DEPLOYMENT_CHART
it's important to specify the ports
MIRROR_NODE_URL_WEB3 is required for some methods in the relay - eth_call and eth_estimateGas

Unfortunatelly there seems to be a similar issue with the network nodes - whenever the relay submits a transaction it fails with a timeout error, I'll let you know if I find a solution.

jeromy-cannon commented 4 months ago

@JeffreyDallas , is this resolved?

I saw this thread if it isn't: https://swirldslabs.slack.com/archives/C06FWU24B7Y/p1715286403180439

JeffreyDallas commented 4 months ago

Last time I tried, submit transaction failed, will try it with new releases of rpc relay

leninmehedy commented 4 months ago

@JeffreyDallas @Ivo-Yankov axios needs fully qualified DNS names of the service see here

Can you please try and see if it works with updating the variables here:https://github.com/hashgraph/solo/blob/main/src/commands/relay.mjs#L48

valuesArg += ` --set config.MIRROR_NODE_URL=http://${constants.FULLSTACK_DEPLOYMENT_CHART}-rest.${namespace}.svc.cluster.local:5551`
valuesArg += ` --set config.MIRROR_NODE_URL_WEB3=http://${constants.FULLSTACK_DEPLOYMENT_CHART}-web3.${namespace}.svc.cluster.local:8545`

JeffreyDallas commented 3 months ago

I don't think DNS is the issue, from the log file mirror_rest.log or mirror_web3.log

we can see that API call from rpc relay can reach mirror node already

solo_mirror_logs.zip

jeromy-cannon commented 2 months ago

@JeffreyDallas ,

in: https://github.com/hashgraph/solo/pull/481/files

@nathanklick and I are looking at this and have some thoughts:

create accounts using Solo instead of LocalNode. If something is missing, we can add it.
generate the traffic using MirrorNode Monitor, I think we have a GHIssue, and the FST Charts logic I think was merged recently.
add the hardhat test case to our repo with its own subproject examples and its own package.json to remove the dependency on the other repo (we can add other examples in this subproject in the future)
make dummy hardhat app under examples
2nd case, building nodejs app that uses hedera network, I want to write unit tests, those unit tests should leverage a solo provisioned network
- how to write a simple app
- how to write unit tests that stand up and teardown solo
[ ] @jeromy-cannon needs to split some of this into a separate story to be done first

hashgraph / solo

Ensure JSON RPC relay is usable #244