search

simpleledger / SLPDB

SLP Token Graph Indexer
https://simpleledger.cash/project/slpdb/
MIT License
35 stars 33 forks source link

SLPDB Crash Info. NFT test. Block 664399 #75

Closed christroutner closed 3 years ago

christroutner commented 3 years ago

This issue captures the results of a crash.

Output of SLPDB

...
SLPJS Validating: 7e97fb427ee482b3a6b3b15c073bd29efc7a6aae5f6a0487d0b8f107eff5f8d4
[INFO] JSON RPC: getRawTransaction 9ce4697f1e42c0def618510e25c0aa394798be7fea524b5a7a781104a5c277f7
SLPJS Validating: 2f0f67fd0ffd9249c5991032a8a105994c1eca3688eb1c180a407e52c7fae58c
[INFO] JSON RPC: getRawTransaction 40d69e71fb0c9361a266d5c49f2c3147683169ed20fb6e95b66461841c3360cd
[INFO] JSON RPC: getRawTransaction ea0e3ff39dfaa9c74a3a91a2f63f8770e3976afc914a760f9b0c058a59903951
SLPJS Validating: ea0e3ff39dfaa9c74a3a91a2f63f8770e3976afc914a760f9b0c058a59903951
[INFO] JSON RPC: getRawTransaction d4b005b92d4129615111bf30855b9cd454c8595ffd45a4a903dae130b0335313
[INFO] JSON RPC: getRawTransaction 3c9ff9ddbbde36e584c4b9f26b646f85107255c74656a7e9df20832a76719a04
SLPJS Validating: 3c9ff9ddbbde36e584c4b9f26b646f85107255c74656a7e9df20832a76719a04
[INFO] JSON RPC: getRawTransaction df0ecb2f48dddb99df7a9bd3143b5ecd866e374f19717b39aa3284dc835c810a
[INFO] JSON RPC: getRawTransaction 7ea4464cb41bb7fb2428e297a5474552d2df8d4f2316e91218fcf524281fa174
SLPJS Validating: 7ea4464cb41bb7fb2428e297a5474552d2df8d4f2316e91218fcf524281fa174
[INFO] JSON RPC: getRawTransaction 6e60b7ccd8d14db8e657d5549cd55d361e8747cde31efabaa8aee992e9d15257
[INFO] JSON RPC: getRawTransaction 2cb3fe5599ba8a30fcf45f09f363e624e34a2108d8039bf822faf25292bddf8e
SLPJS Validating: 2cb3fe5599ba8a30fcf45f09f363e624e34a2108d8039bf822faf25292bddf8e
[INFO] JSON RPC: getRawTransaction d144e185dd30d4ba1a6b7c421d32a758d02f18098dae14cd2c32e742238bf524
[INFO] JSON RPC: getRawTransaction 38c3c74ea9d73967207d79a51ebfd238c355b5d511d0c43b173892a570daf5ba

<--- Last few GCs --->

[39:0x2fe7440]  3548613 ms: Mark-sweep 7843.4 (8220.7) -> 7830.6 (8216.2) MB, 6287.9 / 0.0 ms  (average mu = 0.110, current mu = 0.091) allocation failure scavenge might not succeed
[39:0x2fe7440]  3549278 ms: Scavenge 7844.0 (8216.2) -> 7834.2 (8218.2) MB, 75.1 / 0.0 ms  (average mu = 0.110, current mu = 0.091) allocation failure 

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x13cf099]
Security context: 0x24ed1a3408d1 <JSObject>
    1: emit [0xf50b5ee4d91] [events.js:~263] [pc=0x16153e5f6277](this=0x140790d13d09 <Socket map = 0x333f18290ce9>,0x0831575aa0d1 <String[#6]: resume>)
    2: resume_(aka resume_) [0x1fd21979d441] [_stream_readable.js:985] [bytecode=0x97424dea119 offset=57](this=0x0c31327404b1 <undefined>,0x140790d13d09 <Socket map = 0x333f18290ce9>,0x140790d13e29 <ReadableState ...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa093f0 node::Abort() [node]
 2: 0xa097fc node::OnFatalError(char const*, char const*) [node]
 3: 0xb8431e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb84699 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd31055  [node]
 6: 0xd316e6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd3df65 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd3ee15 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd418cc v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd0837b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x1049fbe v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x13cf099  [node]
Aborted (core dumped)
npm ERR! code ELIFECYCLE
npm ERR! errno 134
npm ERR! slpdb@1.0.0-rc11 start: `tsc && node --max_old_space_size=8192 index run`
npm ERR! Exit status 134
npm ERR! 
npm ERR! Failed at the slpdb@1.0.0-rc11 start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/safeuser/.npm/_logs/2020-12-04T21_40_46_289Z-debug.log
christroutner commented 3 years ago

The crash above was run with npm start. It crashed a second time with the same message. I am now running it with this start command:

tsc && node --max_old_space_size=8192 index run

jcramer commented 3 years ago

This version of SLPDB hasn't been optimized to work with every SLP Token ID for the whole blockchain. This version also doesn't work very well with NFT children.

If you are interested in an enterprise version of SLPDB please send me a message on Telegram (user @jscramer).

nyusternie commented 3 years ago

I'm late to this issue, but I just noticed my node (https://slpdb.devops.cash) has stopped at block # 664399.

@christroutner I'm already running --max_old_space_size=8192, so I'm not sure what else I should try other than restart and wait to see if it sorts itself out. how did you eventually resolve this?

@jcramer Here is my Statuses output before I reset the node:

{
    "_id": "5fd06bab7a96dd65da22777c",
    "version": "1.0.0-beta-rc12",
    "versionHash": "2f23af5122f11c24fb86026bea2b198a024c8f9b",
    "deplVersionHash": null,
    "startCmd": "node ProcessContainerFork.js run",
    "context": "SLPDB",
    "lastStatusUpdate": {
        "utc": "Wed, 09 Dec 2020 06:16:11 GMT",
        "unix": 1607494571
    },
    "lastIncomingTxnZmq": null,
    "lastIncomingBlockZmq": null,
    "lastOutgoingTxnZmq": null,
    "lastOutgoingBlockZmq": null,
    "state": "STARTUP_BLOCK_SYNC",
    "stateHistory": [
        {
            "utc": "Tue, 08 Dec 2020 22:28:54 GMT",
            "state": "PRE_STARTUP"
        },
        {
            "utc": "Tue, 08 Dec 2020 22:28:54 GMT",
            "state": "STARTUP_BLOCK_SYNC"
        },
        {
            "utc": "Wed, 09 Dec 2020 00:23:44 GMT",
            "state": "PRE_STARTUP"
        },
        {
            "utc": "Wed, 09 Dec 2020 00:23:44 GMT",
            "state": "STARTUP_BLOCK_SYNC"
        },
        {
            "utc": "Wed, 09 Dec 2020 02:08:20 GMT",
            "state": "PRE_STARTUP"
        },
        {
            "utc": "Wed, 09 Dec 2020 02:08:20 GMT",
            "state": "STARTUP_BLOCK_SYNC"
        },
        {
            "utc": "Wed, 09 Dec 2020 04:11:24 GMT",
            "state": "PRE_STARTUP"
        },
        {
            "utc": "Wed, 09 Dec 2020 04:11:24 GMT",
            "state": "STARTUP_BLOCK_SYNC"
        },
        {
            "utc": "Wed, 09 Dec 2020 05:49:11 GMT",
            "state": "PRE_STARTUP"
        },
        {
            "utc": "Wed, 09 Dec 2020 05:49:11 GMT",
            "state": "STARTUP_BLOCK_SYNC"
        }
    ],
    "network": "mainnet",
    "bchBlockHeight": 664399,
    "bchBlockHash": "00000000000000000413c48242a18a65b0179c59b309f6f7c9405cdb2e7ed107",
    "slpProcessedBlockHeight": null,
    "mempoolInfoBch": {
        "loaded": true,
        "size": 84,
        "bytes": 42955,
        "usage": 134032,
        "maxmempool": 300000000,
        "mempoolminfee": 0.00001,
        "minrelaytxfee": 0.00001
    },
    "mempoolSizeSlp": -1,
    "tokensCount": 57190,
    "pastStackTraces": [
        "[Thu, 03 Dec 2020 08:09:26 GMT] Error: _dbQuery failed with query {\"v\":3,\"q\":{\"db\":\"c\",\"find\":{\"blk.h\":\"000000000000000001a21edc5cb0d9cf63ee4b8eb309ba45add8c6ae9fdf4e29\"},\"limit\":10000000},\"r\":{\"f\":\"[ .[] | { txid: .tx.h, timestamp: (if .blk? then (.blk.t | strftime(\\\"%Y-%m-%d %H:%M:%S\\\")) else null end), slp: .slp } ]\"}} has response {\"errors\":[\"Error: spawn ENOMEM\"]}\n    at Function.<anonymous> (/usr/src/slpdb/query.ts:43:27)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/slpdb/query.js:5:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)",
        "[Wed, 02 Dec 2020 18:41:33 GMT] Error: _dbQuery failed with query {\"v\":3,\"q\":{\"db\":\"c\",\"find\":{\"blk.h\":\"0000000000000000019fed3bc703f938fbbc464acb48e53fc2600595250ec504\"},\"limit\":10000000},\"r\":{\"f\":\"[ .[] | { txid: .tx.h, timestamp: (if .blk? then (.blk.t | strftime(\\\"%Y-%m-%d %H:%M:%S\\\")) else null end), slp: .slp } ]\"}} has response {\"errors\":[\"Error: spawn ENOMEM\"]}\n    at Function.<anonymous> (/usr/src/slpdb/query.ts:43:27)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/slpdb/query.js:5:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)",
        "[Wed, 25 Nov 2020 22:36:15 GMT] Error: _dbQuery failed with query {\"v\":3,\"q\":{\"db\":\"c\",\"find\":{\"blk.h\":\"00000000000000000132f48224f26a4b46afa322d80d517f54acbb90bcbd4fec\"},\"limit\":10000000},\"r\":{\"f\":\"[ .[] | { txid: .tx.h, timestamp: (if .blk? then (.blk.t | strftime(\\\"%Y-%m-%d %H:%M:%S\\\")) else null end), slp: .slp } ]\"}} has response {\"errors\":[\"Error: spawn ENOMEM\"]}\n    at Function.<anonymous> (/usr/src/slpdb/query.ts:43:27)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/slpdb/query.js:5:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)",
        "[Fri, 02 Oct 2020 05:12:15 GMT] Error: _dbQuery failed with query {\"v\":3,\"q\":{\"db\":\"c\",\"find\":{\"blk.h\":\"000000000000000000f596d1f95cde5d0dbcdbe34254f86e74a8b01258bcb967\"},\"limit\":10000000},\"r\":{\"f\":\"[ .[] | { txid: .tx.h, timestamp: (if .blk? then (.blk.t | strftime(\\\"%Y-%m-%d %H:%M:%S\\\")) else null end), slp: .slp } ]\"}} has response {\"errors\":[\"Error: spawn ENOMEM\"]}\n    at Function.<anonymous> (/usr/src/slpdb/query.ts:43:27)\n    at Generator.next (<anonymous>)\n    at fulfilled (/usr/src/slpdb/query.js:5:58)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)"
    ],
    "doubleSpends": [],
    "reorgs": [],
    "mongoDbStats": {
        "db": "slpdb",
        "collections": 5,
        "views": 0,
        "objects": 2360306,
        "avgObjSize": 3188.8848674705737,
        "dataSize": 7178.062520980835,
        "storageSize": 3048.87890625,
        "indexes": 71,
        "indexSize": 1388.00390625,
        "totalSize": 4436.8828125,
        "scaleFactor": 1048576,
        "fsUsedSize": 102635.66796875,
        "fsTotalSize": 253854.44140625,
        "ok": 1
    },
    "publicUrl": "slpdb.devops.cash",
    "telemetryHash": null,
    "system": {
        "loadAvg1": 1.154296875,
        "loadAvg5": 1.05078125,
        "loadAvg15": 1.42236328125,
        "platform": "linux",
        "cpuCount": 4,
        "freeMem": 139.3046875,
        "totalMem": 7980.99609375,
        "uptime": 7099532,
        "processUptime": 1622.466977569
    }
}

will update if anything changes...

christroutner commented 3 years ago

@nyusternie I was able to get my SLPDB past this block by increasing the memory to 32GB and using --max_old_space_size=28000

nyusternie commented 3 years ago

@christroutner thanks man!

i can't upgrade past 16GB on my vps, so I'll try that and see if it works, otherwise I'll have to migrate and re-build from scratch