search

weavedb / rdk

WeaveDB Rollup Development Kit
3 stars 3 forks source link

rate limit causing the node to get stuck #26

Closed drumfeet closed 6 months ago

drumfeet commented 7 months ago

The issue can be observed when using the weavedb sdk. However, with the rollup node, it is causing the rollup mechanism to get stuck. It will stop committing transactions to Warp and rollup mechanism will also fail. You will need to restart the node on your local environment to recover from the issue.

NetworkCommunicationError: Wrong response code: 403. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: lGr3GC2f1ZPsyLXzFAe-iSh0JTFKV5aRB0q2Fdu4oaVBDOrJW13FZA==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>
    at getJsonResponse (/Users/marss/Documents/GitHub/wdb-rollup-demo/sdk-node/node_modules/warp-contracts/lib/cjs/utils/utils.js:109:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async HandlerBasedContract.bundleInteraction (/Users/marss/Documents/GitHub/wdb-rollup-demo/sdk-node/node_modules/warp-contracts/lib/cjs/contract/HandlerBasedContract.js:277:29)
    at async SDKNODE.send (/Users/marss/Documents/GitHub/wdb-rollup-demo/sdk-node/node_modules/weavedb-sdk/index.js:886:12)
    at async SDKNODE.execNext (/Users/marss/Documents/GitHub/wdb-rollup-demo/sdk-node/node_modules/weavedb-sdk/index.js:838:25) {
  error: 'Wrong response code: 403. <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">\n' +
    '<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">\n' +
    '<TITLE>ERROR: The request could not be satisfied</TITLE>\n' +
    '</HEAD><BODY>\n' +
    '<H1>403 ERROR</H1>\n' +
    '<H2>The request could not be satisfied.</H2>\n' +
    '<HR noshade size="1px">\n' +
    'Request blocked.\n' +
    "We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.\n" +
    '<BR clear="all">\n' +
    'If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.\n' +
    '<BR clear="all">\n' +
    '<HR noshade size="1px">\n' +
    '<PRE>\n' +
    'Generated by cloudfront (CloudFront)\n' +
    'Request ID: lGr3GC2f1ZPsyLXzFAe-iSh0JTFKV5aRB0q2Fdu4oaVBDOrJW13FZA==\n' +
    '</PRE>\n' +
    '<ADDRESS>\n' +
    '</ADDRESS>\n' +
    '</BODY></HTML>'
}
drumfeet commented 7 months ago

Below is another error message that is also caused by rate limit imposed by Warp.

NetworkCommunicationError: Wrong response code: 429. <!doctype html>429429 Too Many Requests

On weavedb-sdk

Access to fetch at 'https://gw.warp.cc/gateway/v3/interactions-sort-key?contractId=Awwzwvw7qfc58cKS8cG3NsPdDet957-Bf-S1RcHry0w&from=000001402714%2C1712935612850%2C397028989582cf736f7503ae0051484191477047a3011860526b98e0996c1d7e&to=000001405721%2C1713321869894%2C88cef6f8423859869fa8a59e9b936067a3208f23b1e4cd487038f43043514abb&fromSdk=true&confirmationStatus=not_corrupted' from origin 'http://localhost:3002' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

GET https://gw.warp.cc/gateway/v3/interactions-sort-key?contractId=Awwzwvw7qfc58cKS8cG3NsPdDet957-Bf-S1RcHry0w&from=000001402714%2C1712935612850%2C397028989582cf736f7503ae0051484191477047a3011860526b98e0996c1d7e&to=000001405721%2C1713321869894%2C88cef6f8423859869fa8a59e9b936067a3208f23b1e4cd487038f43043514abb&fromSdk=true&confirmationStatus=not_corrupted net::ERR_FAILED 429 (Too Many Requests)

This can be easily replicated by reading contract state with a few hundred transactions. For example: https://sonar.warp.cc/#/app/contract/tbg8t02nuUl_KahdVcOd6lxDeFDgtEQnVIyyqR8i8Nw?network=mainnet

I also submitted a test case for this issue https://github.com/weavedb/rdk/pull/30

Basically, there are 2 rate limit error codes.

  1. error code 403 - occurs when submitting transactions to Warp
  2. error code 429 - occurs during state evaluation(fetching cache) or submitting transactions to Warp
drumfeet commented 7 months ago

Upon further investigation, maybe the rate limit imposed by Warp is not the main cause that is getting the node to get stuck. It seems that if I only use a single process to write queries to the db on a rollup node every 15 seconds, it stop committing to Warp when it gets rate limited, but the node will just start committing to Warp again after about 2-3 hours from my observation of the db contract in Warp explorer.

drumfeet commented 6 months ago

@ocrybit I confirm that I do not get the error 429 anymore when applying the sequencerUrl and apiKey. Error 403 does not seem to occur as well. However, I am getting another error when executing the test case for error 403. https://github.com/weavedb/rdk/blob/drumfeet-tests/node/node-server/test/rate-limit-403.js

error: 'Wrong response code: 500. [737b5563-5d31-48e4-b354-2813bb0e5508]: timeout'
ocrybit commented 6 months ago

500 is an unknown error on the remote server and not something we can fix on our side. At least the rate limit errors don't happen anymore. We can have logic to retry or circumvent it when 500 occurs.