search

threefoldfoundation / tfchain

Official implementation of the ThreeFold blockchain
https://explorer.threefoldtoken.com
Apache License 2.0
13 stars 7 forks source link

Explorer crashes when requesting an address with too many transactions #379

Open robvanmieghem opened 4 years ago

robvanmieghem commented 4 years ago

Even crasges the host os

LeeSmet commented 4 years ago

Investigation shows the host os did not go down, but OOM killed server processes before killing the tfchain daemon. An active ssh connection also hung for about 10 mins before eventually recovering, though logs do not show why this happened

LeeSmet commented 4 years ago

Currently the server handles the load fine (with curl). cpu about 30-40 %. Memory is also more than fine. The response for the call causing the daeomon to be OOM'd returns about 252MB worth of json. Another call currently returning about 520MB worth of json also seems to run fine

robvanmieghem commented 4 years ago

pfew, 520 MB of json for an http call, if it were only the website, we could easily add paging but I'm afraid of the Mobicage wallet.

GlenDC commented 4 years ago

Such huge amounts is only a problem for block creator wallets. No block creator wallet is used by a Mobicage wallet user. And we know for sure it won't be used ever.

Paginating for the explorer could also be done, but I prefer to not spent more effort on the old explorer it until we know for sure we want to keep that alive. For now I am of the opinion we want to migrate to a new explorer as soon as possible. But I guess we can agree on those things by the end of this week, beginning of next week, after we played all well enough and iterated on the new explorer MVP.

LeeSmet commented 4 years ago

The caddy logs show that around the time the explorer crashed a lot of hash lookups came in simultaniously:

- - [28/Oct/2019:11:49:57 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                          [1084/1954]
- - [28/Oct/2019:11:49:57 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:57 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:58 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:49:59 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316                                                                                     
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:50:00 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:51:17 +0100] "GET /explorer/hashes/ac36f96b1318c51b774c448649ccb85a2dea785bc1f1e03b4daa972c5ae7506c HTTP/2.0" 200 67316
- - [28/Oct/2019:11:51:18 +0100] "GET /explorer/hashes/0195de96da59de0bd59c416e96d17df1a5bbc80acb6b02a1db0cde0bcdffca55a4f7f369e955ef HTTP/2.0" 502 16
...

Source IP removed

robvanmieghem commented 4 years ago

Regardless of the new explorer, paging does not seem like a bad idea