Usage as Websocket Client fails

[kaeptnb]

Hello,

when setup dshackle as a Polygon Failover RPC, seems to sync fine:

| INFO | Multistream | State of MATIC: height=22251457, status=[OK/3], lag=[1, 1, 0], weak=[]

also a curl works flawless:

curl -m3 -s -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_blockNumber","id":1}' http://10.1.1.61:8545/poly {"jsonrpc":"2.0","id":1,"result":"0x15387bd"}

but when trying to access it from a chainlink node, dshackle is throwing:

| WARN | WsFactory$EthereumWs | Failed to proceed with a RPC message: FAIL_ZERO_SUBSCRIBER

and the chainlink node complains about a non functioning rpc provider:

[WARN] Unexpected error while backfilling heads headtracker/head_tracker.go:250 err=fetchAndSaveHead failed: primary websocket (ws://10.1.1.61:8545/poly) call failed: websocket: close 1005 (no status) errVerbose=websocket: close 1005 (no status)

is that a dshackle misconfiguration from my side?

Greetings & thx for this great proxy!

[splix]

Hi @kaeptnb, thank you for reporting it.

Is it possible to catch the traffic sample for such request from Chainlink? Like with Wireshark or similar.

I'd like to see what Dshackle receive and how it responds

[kaeptnb]

Hello! Thx for looking into this! See attached file...

[tcpdump-dshackle-chainlinknode-polygon.pcap.gz] (https://github.com/emeraldpay/dshackle/files/7741672/tcpdump-dshackle-chainlinknode-polygon.pcap.gz)

Greetings

[splix]

I didn't figure out what is the reason because the log looks like a normal interaction. But I've made several other improvements and fixes to Websocket since the last release, so maybe it's fixed already. Can you try with dev build? You can get it at https://artifacts.emerald.cash/builds/dshackle/2021-12-23/20211224-0751-8526c2a/dshackle-0.12.0-SNAPSHOT.zip Thank you

[kaeptnb]

tried the dev version, but with that the cl-node complains at startup about:

error starting app: The method eth_chainId does not exist/is not available failed to verify chain ID for node primary-0-137

and indeed: "eth_chainId" seems not available via dshackle anymore:

curl -m3 -s -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_chainId","id":1}' http://dshackle.localhost:8545/poly {"jsonrpc":"2.0","id":1,"error":{"code":-32601,"message":"The method eth_chainId does not exist/is not available"}}

whereas ofc: curl -m3 -s -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_chainId","id":1}' http://non-dshackle-rpc:8545 {"jsonrpc":"2.0","id":1,"result":"0x89"}`

Merry Christmas!

[splix]

Oh, that's a silly bug. Fixed now with https://artifacts.emerald.cash/builds/dshackle/2021-12-24/20211225-0710-f2b471c/dshackle-0.12.0-SNAPSHOT.zip

[kaeptnb]

Now CL Node complains:

[ERROR] LogBroadcaster: Unable to backfill a batch of logs log/eth_subscriber.go:187 err=primary websocket (ws://dshackle.localhost:8545/poly) call failed: The method eth_getLogs does not exist/is not available errVerbose=The method eth_getLogs does not exist/is not available

chainlink[471944]: primary websocket (ws://dshackle.localhost:8545/poly) call failed

[splix]

That one is more tricky, needs some design changes. Will take a bit more time. Thank you for reporting it

[splix]

So, the issue is that Dshackle doesn't currently provide eth_getLogs method, but before the recent changes it didn't return error on that method. So I think that was actual reason that caused Chainlink disconnect.

In general, I would need to do some design changes to support eth_getLogs correctly, i.e., with caching and even load balancing.

But for now I recommend to just enable it as is. I mean it's just not in the default list of methods available methods on upstreams but you can always enable it in config. Just add to each of your upstream in your dshackle.yaml, like:

    .....
    - id: my-upstream-1
      chain: matic
      methods:
        enabled:
          - name: eth_getLogs
      connection:
        ethereum:
            ......
    .....

It should resolve that issue.

[kaeptnb]

that startsup the cl node! But immediatly after startup, the cl node complains:

Jan 07 14:19:00 vd-cln1 chainlink[677661]: 2022-01-07T14:19:00.238+0100 [INFO] LogBroadcaster: Fetched a total of 0 logs for backfill log/eth_subscriber.go:145 evmChainID=137 Jan 07 14:19:00 vd-cln1 chainlink[677661]: 2022-01-07T14:19:00.238+0100 [INFO] LogBroadcaster: Finished async backfill of 0 logs log/eth_subscriber.go:159 evmChainID=137 Jan 07 14:19:12 vd-cln1 chainlink[677661]: 2022-01-07T14:19:12.696+0100 [WARN] BlockHistoryEstimator: error fetching blocks gas/block_history_estimator.go:227 err=BlockHistoryEstimator#fetchBlocks erro> Jan 07 14:19:12 vd-cln1 chainlink[677661]: remote eth node timed out Jan 07 14:19:12 vd-cln1 chainlink[677661]: github.com/smartcontractkit/chainlink/core/services/eth.wrap Jan 07 14:19:12 vd-cln1 chainlink[677661]: /home/cln6/git-co/chainlink/core/services/eth/node.go:371 Jan 07 14:19:12 vd-cln1 chainlink[677661]: github.com/smartcontractkit/chainlink/core/services/eth.node.wrapWS Jan 07 14:19:12 vd-cln1 chainlink[677661]: /home/cln6/git-co/chainlink/core/services/eth/node.go:359 Jan 07 14:19:12 vd-cln1 chainlink[677661]: github.com/smartcontractkit/chainlink/core/services/eth.node.BatchCallContext Jan 07 14:19:12 vd-cln1 chainlink[677661]: /home/cln6/git-co/chainlink/core/services/eth/node.go:135

and dshackle :

Jan 07 14:22:09 vd-cln1 dshackle[677563]: 2022-07-01 14:22:09.292 | WARN | ProduceLogs | Cannot find receipt for tx e08b080ae0e1b4797376f2e6dd91608100631e1ee07b30df725a71686e7fc40f Jan 07 14:22:09 vd-cln1 dshackle[677563]: 2022-07-01 14:22:09.391 | WARN | CompoundReader | Failed to read from io.emeraldpay.dshackle.reader.RekeyingReader@9de713f Jan 07 14:22:09 vd-cln1 dshackle[677563]: io.emeraldpay.dshackle.upstream.rpcclient.JsonRpcException: HTTP Code: 403 Jan 07 14:22:09 vd-cln1 dshackle[677563]: at io.emeraldpay.dshackle.upstream.rpcclient.JsonRpcError.asException(JsonRpcError.kt:34) ~[dshackle-0.12.0-SNAPSHOT.jar:?]

Jan 07 14:24:09 vd-cln1 dshackle[677563]: 2022-07-01 14:24:09.767 | WARN | ProduceLogs | Cannot find receipt for tx 76bee3f174d8a3de0843ea15d3e66179e63cc13eb5725c5598f893b3a5c26544 Jan 07 14:24:12 vd-cln1 dshackle[677563]: 2022-07-01 14:24:12.389 | INFO | Multistream | State of MATIC: height=23442097, status=[OK/2,LAGGING/1], lag=[0, 2, 0], weak=[mainnetmaticnetwork] Jan 07 14:24:18 vd-cln1 dshackle[677563]: 2022-07-01 14:24:18.217 | WARN | ResponseParser | Failed to parse JSON from upstream: Unrecognized token 'Oops': was expecting (JSON String, Number, Array, Obje> Jan 07 14:24:18 vd-cln1 dshackle[677563]: at [Source: (byte[])"Oops...nothing here!"; line: 1, column: 6]

[splix]

It seems that the upstream is not working and returning Oops...nothing here! as a response to any request

[kaeptnb]

i left out all failover/other rpc endpoints for test now, leaving only one 100% known to work RPC (local matic node) in the config;

dshackle logs: 2022-07-01 18:54:37.947 | INFO | Multistream | State of MATIC: height=23449125, status=[OK/1], lag=[0], weak=[] 2022-07-01 18:54:52.947 | INFO | Multistream | State of MATIC: height=23449132, status=[SYNCING/1], lag=[0], weak=[localmaticnode]

chainlink logs: 2022-01-07T18:54:59.314+0100 [WARN] BlockHistoryEstimator: error fetching blocks gas/block_history_estimator.go:227 err=BlockHistoryEstimator#fetchBlocks error fetching blocks with BatchCallContext: primary websocket (ws://10.1.1.61:8545/poly) call failed: remote eth node timed out: context deadline exceeded errVerbose=context deadline exceeded remote eth node timed out

It seems Chainlink's "BlockHistoryEstimator" doesnt work with dshackle; From the Chainlink docs:

"FixedPrice uses the configured values for gas price BlockHistory dynamically adjusts default gas price based on heuristics from mined blocks."

And indeed - with "FixedPrice" the node starts up and - seems (hard to test 100% right now)- to work , but with "BlockHistory"-Mode , which would be needed in production - it doesnt work yet.

Greetings!

[Tobias35]

we need this for polygon and chainlink

[ebadiere]

We need this fix as well.