search

godwokenrises / godwoken-kicker

one line command to start godwoken-polyjuice chain.
19 stars 24 forks source link

docker_web3-indexer_1 Exit 1 (Error: error returned from database: sorry, too many clients already) #248

Open gpBlockchain opened 2 years ago

gpBlockchain commented 2 years ago

env

git log --format="%H" -n 1

guopenglin@192 godwoken-kicker % git log --format="%H" -n 1
14bfd45f0a1337ad2023ccedaf781781eefd1806

./kicker ps

guopenglin@192 godwoken-kicker % ./kicker ps
:> docker-compose -f docker/docker-compose.yml ps

                 Name                                Command                  State                           Ports                     
----------------------------------------------------------------------------------------------------------------------------------------
docker_check-ckb-started-successfully_1   /entrypoint.sh http://ckb: ...   Exit 0                                                       
docker_ckb-indexer_1                      /bin/ckb-indexer -c http:/ ...   Up             0.0.0.0:8116->8116/tcp                        
docker_ckb-miner_1                        /bin/docker-entrypoint.sh  ...   Up             8114/tcp, 8115/tcp                            
docker_ckb_1                              /bin/docker-entrypoint.sh  ...   Up             0.0.0.0:8114->8114/tcp, 0.0.0.0:8115->8115/tcp
docker_godwoken_1                         /var/lib/layer2/entrypoint.sh    Up (healthy)   0.0.0.0:8119->8119/tcp, 0.0.0.0:8120->8120/tcp
docker_postgres_1                         docker-entrypoint.sh postgres    Up             0.0.0.0:5432->5432/tcp                        
docker_redis_1                            docker-entrypoint.sh redis ...   Up             0.0.0.0:6379->6379/tcp                        
docker_web3-indexer_1                     /var/lib/web3-indexer/entr ...   Exit 1                                                       
docker_web3_1                             /var/lib/web3/entrypoint.sh      Up (healthy)   3000/tcp, 0.0.0.0:8024->8024/tcp

docker_web3-indexer_1 logs

[2022-05-12T03:42:24Z INFO  gw_web3_indexer::runner] Sync block 3787
[2022-05-12T03:42:57Z INFO  gw_web3_indexer::runner] Sync block 3788
[2022-05-12T03:43:33Z INFO  gw_web3_indexer::runner] Sync block 3789
[2022-05-12T03:44:01Z INFO  gw_web3_indexer::runner] Sync block 3790
[2022-05-12T03:44:25Z INFO  gw_web3_indexer::runner] Sync block 3791
[2022-05-12T03:45:14Z INFO  gw_web3_indexer::runner] Sync block 3792
[2022-05-12T03:45:48Z INFO  gw_web3_indexer::runner] Sync block 3793
[2022-05-12T03:46:27Z INFO  gw_web3_indexer::runner] Sync block 3794
[2022-05-12T03:47:04Z INFO  gw_web3_indexer::runner] Sync block 3795
[2022-05-12T03:47:47Z INFO  gw_web3_indexer::runner] Sync block 3796
[2022-05-12T03:48:34Z INFO  gw_web3_indexer::runner] Sync block 3797
[2022-05-12T03:49:19Z INFO  gw_web3_indexer::runner] Sync block 3798
[2022-05-12T03:49:46Z INFO  gw_web3_indexer::runner] Sync block 3799
[2022-05-12T03:50:38Z INFO  gw_web3_indexer::runner] Sync block 3800
[2022-05-12T03:51:36Z INFO  gw_web3_indexer::runner] Sync block 3801
[2022-05-12T03:52:24Z INFO  gw_web3_indexer::runner] Sync block 3802
[2022-05-12T03:52:55Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x2c3cecc164f7ff0ab8c45fd3071d8c75733e8c9f0ef9f0dafb7679ad75b7ab1e
[2022-05-12T03:52:55Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:52:55Z INFO  gw_web3_indexer::error_receipt_indexer] delete error tx receipt 1
[2022-05-12T03:53:04Z INFO  gw_web3_indexer::runner] Sync block 3803
[2022-05-12T03:53:12Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x0ebf723b13989b2fa38f52b72116a666ec30a3c9789b7dd5870fe644a3e4f1ba
[2022-05-12T03:53:12Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:53:12Z INFO  gw_web3_indexer::error_receipt_indexer] delete error tx receipt 0
Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already
[2022-05-12T03:53:56Z INFO  gw_web3_indexer::ws_client] Successfully connected
[2022-05-12T03:53:56Z INFO  gw_web3_indexer::ws_client] Message send ok
[2022-05-12T03:53:56Z INFO  gw_web3_indexer::ws_client] ErrorTxReceipt type mismatch: invalid type: string "0x1", expected struct ErrorTxReceipt
[2022-05-12T03:53:58Z INFO  gw_web3_indexer::runner] Sync block 3804
[2022-05-12T03:54:28Z INFO  gw_web3_indexer::runner] Sync block 3805
[2022-05-12T03:55:23Z INFO  gw_web3_indexer::runner] Sync block 3806
[2022-05-12T03:56:09Z INFO  gw_web3_indexer::runner] Sync block 3807
[2022-05-12T03:57:09Z INFO  gw_web3_indexer::runner] Sync block 3808
[2022-05-12T03:57:53Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x323a475fd0e32d148d644970ca5839cca01bbe156ca33f5a48f1cdab0959672b
[2022-05-12T03:57:53Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:58:07Z INFO  gw_web3_indexer::runner] Sync block 3809
[2022-05-12T03:58:11Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x6cafbb0cb7ca6f6dfa547a4a0e9661fc4c850878d85c1692614f0971c14e750b
[2022-05-12T03:58:11Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:58:11Z WARN  sqlx::query] DELETE FROM error_transactions WHERE …; rows: 0, elapsed: 17.937s

    DELETE FROM
      error_transactions
    WHERE
      block_number <= $1

[2022-05-12T03:58:11Z INFO  gw_web3_indexer::error_receipt_indexer] delete error tx receipt 2
[2022-05-12T03:58:22Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x768f2a6956dfa83bf6075ae378d4151a3d56a84ebf32e446c8cb99ee52136d47
[2022-05-12T03:58:22Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T03:59:05Z INFO  gw_web3_indexer::runner] Sync block 3810
[2022-05-12T03:59:56Z INFO  gw_web3_indexer::runner] Sync block 3811
[2022-05-12T04:00:45Z INFO  gw_web3_indexer::runner] Sync block 3812
[2022-05-12T04:01:21Z INFO  gw_web3_indexer::ws_client] receive error tx receipt: 0x0x234dbdc5b4f0e564f1a17f0f97102289ffcc8c1bad2401c693f8d34449685e1f
[2022-05-12T04:01:21Z WARN  gw_web3_indexer::error_receipt_indexer] unsupported polyjuice reason []
[2022-05-12T04:01:21Z ERROR gw_web3_indexer::error_receipt_indexer] clear expired block error receipt error communicating with the server: connection aborted
[2022-05-12T04:01:28Z INFO  gw_web3_indexer::runner] Sync block 3813
Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already
[2022-05-12T04:02:30Z INFO  gw_web3_indexer::ws_client] Successfully connected
[2022-05-12T04:02:30Z INFO  gw_web3_indexer::ws_client] Message send ok
[2022-05-12T04:02:30Z INFO  gw_web3_indexer::ws_client] ErrorTxReceipt type mismatch: invalid type: string "0x2", expected struct ErrorTxReceipt
Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already
[2022-05-12T04:02:34Z INFO  gw_web3_indexer::ws_client] Successfully connected
[2022-05-12T04:02:34Z INFO  gw_web3_indexer::ws_client] Message send ok
[2022-05-12T04:02:34Z INFO  gw_web3_indexer::ws_client] ErrorTxReceipt type mismatch: invalid type: string "0x3", expected struct ErrorTxReceipt
Error: error returned from database: sorry, too many clients already

Caused by:
    sorry, too many clients already
keroro520 commented 2 years ago

Thanks for your report. @gpBlockchain

It seems this report involves several issues related to web3-indexer:

Could you please confirm these issues? @classicalliu cc @Flouse

Flouse commented 2 years ago

Is that a clean env? Would you clean your workspace and try again?

classicalliu commented 2 years ago

unsupported polyjuice reason

unsupported polyjuice reason and ErrorTxReceipt type mismatch could be ignored.

sorry, too many clients already How to reproduce this error ?

gpBlockchain commented 2 years ago

unsupported polyjuice reason

unsupported polyjuice reason and ErrorTxReceipt type mismatch could be ignored.

sorry, too many clients already How to reproduce this error ?

stress test ？i grep log " too many clients already"

guopenglin@192 godwoken-kicker % cat kick.log | grep " too many clients already" | wc -l
   57786

too many client aleady Up and down related logs

web3_1                            | [1]     at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
web3_1                            | [1] 2022-05-12T05:04:57.117Z [info]: POST / 200 1872ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.129Z [info]: POST / 200 3580ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.129Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.217Z [info]: POST / 200 3314ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.217Z [info]: POST / 200 2693ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.222Z [info]: POST / 200 3159ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.222Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.236Z [info]: POST / 200 2568ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.240Z [info]: POST / 200 1873ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.244Z [info]: POST / 200 2573ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.250Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.309Z [info]: POST / 200 3086ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.313Z [info]: POST / 200 2357ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.313Z [info]: POST / 200 2347ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.315Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.321Z [info]: POST / 200 2985ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.321Z [info]: POST / 200 2339ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.322Z [info]: POST / 200 2136ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.322Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
web3_1                            | [1] 2022-05-12T05:04:57.324Z [info]: POST / 200 1965ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.325Z [info]: POST / 200 1051ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.326Z [info]: POST / 200 2338ms @eth_getBlockByNumber
web3_1                            | [1] 2022-05-12T05:04:57.327Z [error]: JSONRPC Server Error: [bound getBlockByNumber] error: sorry, too many clients already error: sorry, too many clients already
web3_1                            | [1]     at Parser.parseErrorMessage (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:287:98)
web3_1                            | [1]     at Parser.handlePacket (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:126:29)
web3_1                            | [1]     at Parser.parse (/godwoken-web3/node_modules/pg-protocol/dist/parser.js:39:38)
--
gpBlockchain commented 2 years ago

cat kick.log| grep 2022-05-12T04:02:3 > time.log time.log

docker_web3-index should try to catch the too many clients exception and try again？