search

kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB
Other
484 stars 39 forks source link

Failed to decompress data #116

Closed gogoout closed 2 years ago

gogoout commented 2 years ago

We saw this error in the log.

Error: Failed to decompress data
    at LMDBStore.getBinaryFast (/app/node_modules/lmdb/dist/index.cjs:1277:26)
    at LMDBStore.get (/app/node_modules/lmdb/dist/index.cjs:1321:22)
    at /app/node_modules/lmdb │
│ /dist/index.cjs:1613:23
    at /app/node_modules/lmdb/dist/index.cjs:1647:6","level":"error","message":"Error: Failed to decompress data
    at LMDBStore.getBinaryFast (/app/node_modules/lmdb/dist/index.cjs:1277:26)
    at LMDBStore.get (/app/node_modules/lmdb/di │
│ st/index.cjs:1321:22)
    at /app/node_modules/lmdb/dist/index.cjs:1613:23
    at /app/node_modules/lmdb/dist/index.cjs:1647:6

The db is opened with on v2.1.0-beta3

commitDelay: 1,
useVersions: false,
encoding: 'msgpack',
sharedStructuresKey: Symbol.for('structures'),
compression: true,
cache: true,
overlappingSync: true,
separateFlushed: true,

Currently my guessing is when we write the messages, server shutdown (which we most likely have called db.close() within the short amount of graceful shutdown period). This seems very hard to reproduce. We'll update this if we can stably reproduce this.

kriszyp commented 2 years ago

which we most likely have called db.close()

I wonder if this might actually be an important clue. We have unexpected shutdowns quite frequently in our app, but never call db.close(), whether it is graceful or forced shutdown. And db.close could actually be hazardous: it immediately frees all the lmdb internal data structures, but it is quite possible for an existing transaction to still be executing (in a worker thread), in which case the transaction would be operating on potentially corrupted memory. I don't know if this is actually the cause of this issue, but perhaps I should make close() actually wait for all transactions to finish before it really closes the environment/db (which would make it async).

gogoout commented 2 years ago

I have suspected that's what happening as well. TBH I think this is a very unexpected behaviour from the user side. Normally using the grace shut period to closing existing connection is a common pattern.

Llorx commented 2 years ago

I also noticed this. What I do is to create a transaction before working with the database, save the promise to an array, keep track of the promise to clean it from the array, and do an await Promise.all just before the close. It just works.

kriszyp commented 2 years ago

@Llorx you have actually been able to reproduce a decompress data error by calling close() during an awaiting transaction?

Anyway, in v2.1.0, close() now should wait for any outstanding transactions to finish before actually freeing the underlying database structures. Hopefully this addresses this issue, but certainly let me know if you encounter it again.

Llorx commented 2 years ago

@kriszyp what happened to me was, inside a transaction callback, calling close() the node process died with a 0xC0000005 exit code (invalid memread or write). If I remember correctly, it was when the transaction was going to commit. I don't wanted to call close() while there were pending transactions, so I added the promise thingy. Never called close() when there was a pending transaction apart of the transaction where close() was called, but I guess is more or less the same thing.

gogoout commented 2 years ago

fyi, after switching to v2.1.0, start and close the db immediately without doing anything would throw this error

ReferenceError: lastPromise is not defined
    at LMDBStore._waitForTxns (/node_modules/lmdb/write.js:769:56)
    at LMDBStore.close (/node_modules/lmdb/read.js:440:26)
gogoout commented 2 years ago

Also I'm still seeing this error even with await db.close() and I have try catch around close function so it won't affect other db closing process. Another thing I want to add is, this seems only happen when the server restarted, which in our situation, this means, it's getting the value before server shutdown it's processing. So I think your guessing should be correct, this should be something to do with shutdown somehow created some broken data. 

Error: Failed to decompress data
    at Dbi.getByBinary (<anonymous>)
    at LMDBStore.getBinaryFast (/app/node_modules/lmdb/dist/index.cjs:1288:26)
    at LMDBStore.get (/app/node_modules/lmdb/dist/index.cjs:1332:22)
kriszyp commented 2 years ago

Also I'm still seeing this error even with await db.close() and I have try catch around close

The try/catch is for the lastPromise error, not the Failed to decompress error, right?

getting the value before server shutdown it's processing.

Does that mean that puts that are happening during the close()/shutdown seem to result in the corrupted entry? Or that gets during the close()/shutdown trigger this error? And this only happens if an open occurs right after the close() (or during)?

gogoout commented 2 years ago

The try/catch is for the lastPromise error, not the Failed to decompress error, right?

Yes

Does that mean that puts that are happening during the close()/shutdown seem to result in the corrupted entry? Or that gets during the close()/shutdown trigger this error? And this only happens if an open occurs right after the close() (or during)?

That should be puts during the close() period caused the corrupted entry because the error only appeared after when you open the db and get it, never before/during the close. The open will be after close, in fact, in our server, db close only when node process exit, and db will open as soon as node process started again.

kriszyp commented 2 years ago

I have added some more tests (https://github.com/DoctorEvidence/lmdb-js/commit/5b6f6f457c0a0eb49afaf8a7dfa47c096d279304#diff-af95cb6f5f65fe70216ddce70325e785438d5aaff956a0666ebf46ef6402f526R934) to try to reproduce this (but to no avail). I fixed the lastPromise error, and added some more debugging information to the decompression error, that might help give us more information on the cause, which are available in v2.1.1.

gogoout commented 2 years ago

Here are some logs with 2.1.2

Failed to decompress data 4 1102 254 0 4 78 83 148
Error: Failed to decompress data
    at Dbi.getByBinary (<anonymous>)
    at LMDBStore.getBinaryFast (/app/node_modules/lmdb/dist/index.cjs:1291:26)
    at LMDBStore.get (/app/node_modules/lmdb/dist/index.cjs:1335:22)
Failed to decompress data 4 3305 254 0 12 233 227 156
Failed to decompress data 4 2514 254 0 9 210 83 153
Failed to decompress data 4 1688 254 0 6 152 83 150
kriszyp commented 2 years ago

I think if I am reading this correctly, these are valid/correct starts to a LZ4+MessagePack encoding of arrays (of length 12, 9, and 6). Does it sound correct that these entries are arrays (at the top level)? I think you said that, once corrupted, you can reliably get the decompress error for an entry (by id). Perhaps next debugging step would be to turn off compression, and get one of the corrupted entries with getBinary(id) to retrieve the raw compressed binary data, and maybe dump that (to hex or whatever) to take a look at?

gogoout commented 2 years ago

emm, you maybe right, we do have top level array in there. To achieve the next step, I need to be able to try catch the error first. Right now, it seems it wasn't able to catch the error properly?

kriszyp commented 2 years ago

Why can't the error be caught, it should be a standard error? I did add more logging in the last commit to dump the entire compressed block though.

gogoout commented 2 years ago

emm, I thought you may have some insights on it, I'll see if I can get some more info around this. Meanwhile I'm trying to see if I can create a minimal repo to reproduce this.

kriszyp commented 2 years ago

Btw, in v2.1.3, the whole compressed block should be logged if there is a decompression error now.

kriszyp commented 2 years ago

I think there is a slight chance that the fix for #124 (in v2.1.4) will fix this, but I am not holding my breath.

gogoout commented 2 years ago

Emm, for my configuration we are not using version. So shouldn't be caused by this? I'll try update to 2.1.4 but we currently still don't have a chance to do a fresh start test. So it will be hard to tell if 2.1.4 fixed it

kriszyp commented 2 years ago

Ah, ok, never mind, #124 was definitely trigged by using versions.

gogoout commented 2 years ago

Some more logs:

Failed to decompress data 4 2302 bytes:"
"254 0 8 254 243 50 212 114 96 220 0 21 170 97 114 116 105 99 108 101 95 105 100 168 98 114 97 110 100 95 105 100 170 99 114 101 97 116 101 100 95 97 116 173 102 105 116 109 101 110 116 95 110 111 116 101 115 162 105 100 169 105 109 112 111 114 116 95 105 100 172 10 0 241 1 116 97 98 108 101 175 105 110 116 101 103 114 105 116 121 95 200 1 244 37 168 105 115 95 100 114 97 102 116 174 105 115 95 112 101 114 102 111 114 109 97 110 99 101 172 105 115 95 117 110 105 118 101 114 115 97 108 169 108 101 110 103 116 104 95 109 109 163 109 112 110 172 188 4 248 29 95 109 112 110 172 112 97 99 107 105 110 103 95 117 110 105 116 172 112 97 114 116 95 116 121 112 101 95 105 100 185 113 117 97 110 116 105 116 121 95 112 101 114 95 39 0 243 3 173 115 121 110 99 95 97 116 116 101 109 112 116 115 170 117 112 100 197 0 243 25 167 117 115 101 114 95 105 100 163 95 99 95 206 4 252 123 80 169 71 66 82 78 49 57 51 49 55 203 67 22 184 151 189 212 22 0 160 217 36 48 1 0 31 45 5 0 0 247 30 48 49 51 102 102 102 48 54 168 56 51 54 53 53 53 48 52 167 97 117 116 111 100 111 99 205 1 24 194 194 194 205 1 34 170 49 54 50 51 50 54 54 54 56 48 11 0 255 7 1 169 71 67 80 84 54 53 55 52 50 1 4 203 67 23 70 232 255 2 10 176 108 0 12 2 2 0 132 49 65 147 212 114 97 146 168 234 1 33 166 101 247 0 129 212 114 98 146 182 103 97 112 234 0 250 0 114 105 98 117 116 101 95 111 112 116 105 111 110 115 175 23 0 119 115 145 66 212 114 99 150 206 1 26 177 33 0 196 95 105 100 162 105 100 164 110 97 109 101 168 100 13 7 55 1 255 1 203 67 22 238 181 139 58 157 0 166 71 65 84 82 56 56 182 0 14 197 49 52 98 50 49 165 70 49 48 88 49 192 61 0 135 145 66 212 114 100 220 0 16 135 0 81 169 100 97 116 97 208 1 113 171 100 101 115 99 114 105 191 0 15 93 2 23 82 170 109 97 120 95 60 2 97 171 109 101 116 97 95 94 2 33 116 169 12 0 131 116 121 112 101 170 109 105 110 33 0 1 222 0 241 10 175 110 111 114 109 97 108 105 122 101 100 95 110 97 109 101 167 112 97 100 98 95 105 100 164 45 0 6 41 2 179 203 67 22 238 170 226 13 46 0 192 160 244 0 68 163 57 57 49 235 1 129 192 192 192 192 192 173 84 104 76 7 247 34 83 105 122 101 32 49 171 84 72 82 69 65 68 83 73 90 69 49 192 3 203 67 23 42 0 104 199 129 164 212 114 101 220 0 17 173 98 111 111 108 101 97 110 95 118 97 108 117 101 0 1 82 170 100 97 116 101 22 0 14 146 1 27 184 164 1 2 220 1 65 95 105 100 172 25 0 1 11 3 49 105 100 171 13 0 50 117 111 109 196 1 15 150 3 29 17 177 25 20 99 105 99 95 109 97 120 137 0 5 18 0 19 105 177 0 20 173 18 0 1 191 0 18 172 129 9 2 45 0 7 37 2 51 192 192 192 43 1 15 31 2 19 15 38 0 12 3 103 3 31 192 39 0 12 133 48 56 51 97 56 102 100 192 155 1 17 195 157 1 85 97 98 145 66 99 122 2 2 152 0 31 57 75 0 13 110 48 49 52 98 50 98 183 2 29 100 14 2 85 57 163 57 57 50 115 0 13 14 2 23 50 14 2 21 50 14 2 53 131 92 101 36 1 15 140 0 20 15 178 0 12 15 36 1 26 30 101 36 1 20 144 220 0 52 216 9 180 234 2 105 51 53 163 50 48 51 220 0 33 166 76 158 5 245 3 166 76 69 78 71 84 72 192 4 203 67 22 239 65 204 212 28 0 208 0 47 51 53 168 1 13 3 171 0 15 209 0 12 2 2 0 31 49 38 0 13 1 244 1 25 102 208 0 129 205 1 34 192 192 145 66 67 254 1 182 192 203 67 22 178 71 85 211 203 0 192 225 5 99 164 52 53 51 51 181 173 5 246 15 46 109 97 110 117 102 97 99 116 117 114 101 114 115 194 194 195 194 194 192 169 69 85 82 79 82 69 80 65 82 10 0 1 76 0 127 23 7 248 215 96 224 144 150 0 13 1 2 0 243 11 49 151 66 69 192 192 166 71 67 67 65 54 57 192 192 192 169 71 67 80 67 50 54 50 51 49 13 0 2 3 6 255 4 168 71 67 80 79 56 49 50 49 192 167 71 67 83 67 56 50 49 192 56 0 0 95 55 56 54 53 51 56 0 1 47 54 50 56 0 14 31 52 56 0 2 31 51 56 0 14 31 53 56 0 1 47 50 55 56 0 14 31 54 56 0 1 47 54 52 56 0 14 31 55 56 0 2 31 53 56 0 5 159 168 71 67 80 67 56 48 49 48 55 0 1 38 48 52 55 0 182 145 66 70 192 192 192 205 131 25 204 101 122 7 1 30 2 33 204 211 7 0 128 192 192 205 1 220 58 144 144 144 33 204 211 7 0 128 192 192 205 1 220 58 144 144 58 144 144 243 11 49 145 66 69 192 192 166 71 67 67 65 55 53 192 192 192 169 71 67 80 67 49 57 48 51 49 13 0 2 243 6 246 15 168 71 67 80 79 56 49 48 52 192 167 71 67 83 67 56 54 56 192 145 66 70 192 192 192 205 131 89 204 86 27 7 1 207 0 49 205 1 42 8 0 128 192 192 205 3 238 23 144 144 0 11 79 53 48 54 52 56 0 2 31 50 56 0 10 68 50 49 50 50 181 0 10 168 0 31 56 56 0 5 159 168 71 67 80 67 52 57 49 54 111 0 1 38 48 52 55 0 246 1 145 66 70 192 192 206 0 2 168 17 206 0 2 152 157 29 244 6 1 179 1 17 48 6 0 128 192 192 205 2 187 37 144 144 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Error: Failed to decompress data
    at Dbi.getByBinary (<anonymous>)
    at LMDBStore.getBinaryFast (/app/node_modules/lmdb/dist/index.cjs:1295:26)
    at LMDBStore.get (/app/node_modules/lmdb/dist/index.cjs:1339:22)
Failed to decompress data 4 1697 bytes:
254 0 6 161 83 150 71 217 36 48 1 0 31 45 5 0 0 143 48 49 51 53 98 102 52 97 38 0 13 242 55 55 53 53 57 102 100 217 36 52 51 102 49 97 98 99 51 45 50 48 57 98 45 52 52 98 53 45 97 99 48 50 45 100 100 50 97 53 51 97 54 101 101 55 49 167 54 51 53 53 49 52 52 167 111 101 95 110 117 109 115 1 65 145 66 67 145 68 69 192 192 2 0 242 20 169 71 66 82 78 50 48 52 48 50 192 192 195 194 192 194 192 192 176 86 111 108 107 115 119 97 103 101 110 32 71 114 111 117 112 41 0 26 70 44 0 1 2 0 15 169 0 19 1 43 0 231 192 194 192 170 55 48 49 56 51 55 52 54 49 65 11 0 2 30 0 65 192 192 192 0 6 0 47 192 192 37 1 52 255 46 48 48 56 52 54 57 217 36 97 48 53 56 52 53 97 52 45 51 55 51 53 45 52 55 100 48 45 56 57 102 48 45 50 54 56 51 55 57 48 48 97 53 49 102 168 50 56 54 55 55 54 52 54 166 120 108 105 115 116 50 48 37 1 1 84 49 57 51 49 54 37 1 114 166 84 82 73 67 76 79 179 0 4 27 1 3 34 0 1 2 0 15 27 1 13 2 159 0 1 43 0 163 192 194 192 166 49 49 51 57 54 57 7 0 2 22 0 15 19 1 63 243 29 97 102 56 54 97 48 217 36 98 48 102 101 100 52 98 99 45 55 55 56 101 45 52 98 49 56 45 97 55 48 100 45 50 57 53 57 51 50 99 101 52 53 48 50 56 2 20 53 56 2 31 17 19 1 1 15 56 2 69 2 169 0 1 3 1 214 192 194 192 169 55 48 49 57 53 57 56 48 49 10 0 2 28 0 15 35 1 63 244 29 49 57 49 97 98 51 217 36 99 56 48 49 49 102 98 57 45 52 48 99 49 45 52 56 57 52 45 97 54 50 54 45 100 102 56 55 57 49 99 99 50 50 52 98 54 2 19 52 54 2 27 27 35 1 133 168 71 66 82 78 52 53 52 90 3 225 173 70 101 98 105 45 66 105 108 115 116 101 105 110 191 0 41 192 70 40 0 1 2 0 15 59 2 13 2 164 0 1 43 0 1 59 2 83 48 48 52 57 50 7 0 2 22 0 15 24 1 63 244 29 54 53 54 49 101 54 217 36 101 51 56 101 53 57 100 102 45 99 49 55 56 45 52 98 98 55 45 97 55 49 102 45 56 55 56 49 54 53 101 49 98 54 51 97 24 1 31 53 24 1 10 52 57 53 57 77 3 81 164 83 119 97 103 182 0 6 15 1 1 31 0 1 2 0 15 15 1 13 2 155 0 1 43 0 98 192 194 192 168 51 48 17 1 127 170 51 48 32 49 48 32 21 1 73 245 28 49 51 54 52 97 54 217 36 101 100 98 56 97 55 99 102 45 55 54 102 51 45 52 54 97 99 45 98 57 99 57 45 54 102 53 56 50 55 101 50 100 97 56 45 2 19 55 21 1 27 51 45 2 1 136 5 84 49 57 52 48 56 22 1 162 169 86 65 78 32 87 69 90 69 76 220 0 6 102 4 1 37 0 1 2 0 15 28 1 13 2 162 0 1 43 0 180 192 194 192 167 53 56 55 52 50 54 49 8 0 2 24 0 176 192 192 192 0 192 192 192 192 192 192 192 0 0 0 0 0 1 0 0 26 255 255 255 255 0 2 0 0 0 0 0 0 16 56 213 26 4 127 0 0 14 0 0 0 0 0 0 0 160 84 52 4 19 127 0 0 8 0 0 0 0 0 0 0 48 86 48 67 1 127 0 0 48 86 48 67 1 127 0 0 164 6 0 0 1 127 0 0 160 84 52 4 19 127 0 0 160 84 52 4 19 127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 22 0 0 0 8 6 0 0 19 0 0 0 3 0 0 0 160 84 52 4 19 127 0 0 232 68 48 67 1 127 0 0 168 68 48 67 1 127 0 0 8 0 0 0 0 0 0 0 232 68 48 67 1 127 0 0 232 68 48 67 1 127 0 0 232 70 48 67 1 127 0 0 192 68 48 67 1 127 0 0 152 69 48 67 1 127 0 0 232 68 48 67 1 127 0 0 232 70 48 67 1 127 0 0 192 68 48 67 1 127 0 0 160 84 52 4 19 127 0 0 176 90 48 67 1 127 0 0 232 70 48 67 1 127 0 0 8 0 0 0 0 0 0 0 40 71 48 67 1 127 0 0 40 71 48 67 1 127 0 0 40 73 48 67 1 127 0 0 0 71 48 67 1 127 0 0 208 90 48 67 1 127 0 0 176 90 48 67 1 127 0 0 176 92 48 67 1 127 0 0 8 71 48 67 1 127 0 0 3 0 0 0 0 0 0 0 160 84 52 4 19 127 0 0 88 9 252 15 255 126 0 0 247 13 252 15 255 126 0 0 88 17 252 15 255 126 0 0 160 84 52 4 19 127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 160 84 52 4 19 127 0 0 0 0 0 0 7 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 168 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 255 0 0 0 0 0 0 0 0 0 0 0 Error: Failed to decompress data
    at Dbi.getByBinary (<anonymous>)
    at LMDBStore.getBinaryFast (/app/node_modules/lmdb/dist/index.cjs:1295:26)
    at LMDBStore.get (/app/node_modules/lmdb/dist/index.cjs:1339:22)
Failed to decompress data 4 1631 bytes:
254 0 6 95 227 150 73 203 67 23 58 171 127 25 232 64 217 36 48 1 0 31 45 5 0 0 143 48 49 50 53 53 48 48 51 38 0 13 243 38 49 51 50 52 97 48 217 36 48 99 57 57 57 99 100 56 45 98 53 101 55 45 54 48 48 101 45 57 48 55 54 45 101 97 51 57 50 52 51 50 52 55 56 56 13 65 145 66 67 145 68 69 192 1 0 242 11 169 71 66 82 78 49 53 48 56 48 192 192 195 194 192 194 192 192 167 77 97 88 103 101 97 114 33 0 26 70 35 0 1 2 0 15 144 0 13 2 182 0 1 43 0 243 2 192 194 192 166 49 57 48 50 57 51 167 49 57 45 48 50 57 23 0 65 192 192 192 0 6 0 33 192 192 14 1 79 175 200 47 170 88 0 13 127 48 102 54 98 56 53 99 38 0 12 10 14 1 222 100 49 101 45 101 99 56 100 45 54 48 50 50 14 1 27 1 14 1 132 167 71 66 82 78 50 49 51 12 1 114 166 68 101 108 112 104 105 170 0 24 70 32 0 1 2 0 15 139 0 12 3 177 0 1 43 0 163 192 194 192 166 76 83 49 54 48 49 7 0 2 22 0 7 8 1 15 13 2 14 111 49 101 52 54 98 55 116 0 12 5 255 0 255 38 53 52 55 56 53 53 56 49 45 56 101 52 51 45 52 99 100 53 45 97 49 49 55 45 54 99 57 54 102 52 99 57 50 100 100 57 168 50 49 57 48 57 55 56 53 166 120 108 105 115 116 50 3 15 1 1 37 49 51 27 2 113 166 66 114 101 109 98 111 183 0 5 15 1 1 32 0 1 2 0 15 155 0 14 1 193 0 1 43 0 242 4 192 194 192 166 83 56 51 53 48 52 168 83 32 56 51 32 53 48 52 24 0 15 17 1 25 111 50 53 53 48 51 50 118 0 14 3 30 3 243 29 53 55 48 102 53 97 50 100 45 56 101 99 52 45 52 99 51 50 45 98 54 55 102 45 51 49 48 97 51 52 55 57 49 56 48 100 167 55 48 55 51 48 55 57 16 1 15 45 3 3 69 57 52 53 55 33 2 97 87 79 75 73 78 71 184 0 6 44 3 2 34 0 1 2 0 15 158 0 13 2 196 0 1 43 0 193 192 194 192 167 90 52 48 57 55 48 49 168 8 0 31 46 38 2 34 95 51 50 52 97 49 119 0 13 4 37 3 255 30 53 97 49 50 99 53 100 102 45 54 101 99 101 45 52 101 54 50 45 97 54 99 53 45 52 97 102 56 50 48 56 54 53 49 49 102 168 55 54 48 52 48 56 49 54 22 1 4 148 168 71 66 82 78 57 56 49 53 21 1 162 169 82 111 97 100 104 111 117 115 101 216 0 25 70 36 0 1 2 0 15 160 0 18 17 49 43 0 66 192 194 192 166 22 1 31 167 21 1 37 1 137 3 47 99 102 117 0 18 243 32 48 217 36 102 49 49 101 57 51 53 98 45 53 52 49 57 45 52 57 51 100 45 98 55 48 52 45 100 97 50 97 99 101 52 101 97 49 97 50 167 49 52 50 55 53 57 48 20 1 27 11 58 3 1 20 1 68 54 54 54 56 20 1 66 163 84 82 87 207 0 4 14 1 2 30 0 1 2 0 15 153 0 12 3 191 0 1 43 0 163 192 194 192 166 71 83 56 50 50 52 7 0 2 22 0 176 192 192 192 0 192 192 192 192 192 192 192 252 127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 240 206 214 111 15 127 0 0 0 0 0 0 0 0 0 0 16 192 167 0 18 127 0 0 184 80 184 71 15 127 0 0 0 0 0 0 0 255 0 0 176 6 119 0 18 127 0 0 208 219 62 110 15 127 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 207 76 1 0 0 0 0 0 8 76 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 222 10 136 238 36 86 0 0 64 39 209 240 36 86 0 0 240 250 115 0 18 127 0 0 144 246 115 0 18 127 0 0 0 0 0 0 0 0 0 0 176 234 223 237 36 86 0 0 64 39 209 240 36 86 0 0 8 221 62 110 15 127 0 0 160 119 144 22 252 127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 207 214 111 15 127 0 0 0 0 0 0 0 0 0 0 16 192 167 0 18 127 0 0 8 81 184 71 15 127 0 0 0 0 0 0 0 255 0 0 176 6 119 0 18 127 0 0 144 220 62 110 15 127 0 0 7 0 0 0 15 127 0 0 128 192 210 240 36 86 0 0 240 101 135 0 18 127 0 0 16 4 140 0 18 127 0 0 128 192 210 240 36 86 0 0 8 50 197 104 15 127 0 0 32 50 197 104 15 127 0 0 64 39 209 240 36 86 0 0 240 250 115 0 18 127 0 0 144 246 115 0 18 127 0 0 0 0 0 0 0 0 0 0 176 234 223 237 36 86 0 0 64 39 209 240 36 86 0 0 200 221 62 110 15 127 0 0 160 119 144 22 252 127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 48 207 214 111 15 127 0 0 0 0 0 0 0 0 0 0 16 192 167 0 18 127 0 0 88 81 184 71 15 127 0 0 0 0 0 0 0 255 0 0 176 6 119 0 18 127 0 0 80 221 62 110 15 127 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 187 53 1 0 0 0 0 0 8 53 1 0 0 0 0 0 0 53 1 0 0 0 0 0 12 0 0 0 0 255 0 0 0 0 0 0 0 0 0 0 64 39 209 240 36 86 0 0 240 250 115 0 18 127 0 0 144 246 115 0 18 127 0 0 0 0 0 0 0 0 0 0 176 234 223 237 36 86 0 0 64 39 209 240 36 86 0 0 136 222 62 110 15 127 0 0 160 119 144 22 252 127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 80 207 214 111 15 127 0 0 0 0 0 0 0 0 0 0 16 192 167 0 18 127 0 0 168 81 184 71 15 127 0 0 0 0 0 0 0 255 0 0 176 6 119 0 18 127 0 0 16 222 62 110 15 127 0 0 7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 252 40 1 0 0 0 0 0 8 37 1 0 0 0 0 0 0 36 1 0 0 0 0 0 12 0 0 0 0 255 0 0 0 0 0 0 0 0 0 0 64 39 209 240 36 86 0 0 240 250 115 0 18 127 0 0 144 246 115 0 18 127 0 0 0 0 0 0 0 0 0 0 176 234 223 237 36 86 0 0 64 39 209 240 36 86 0 0 168 231 62 110 15 127 0 0 160 119 144 22 252 127 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 112 207 214 111 15 127 0 0 0 0 0 0 0 0 0 0 16 192 167 0 18 127 0 0 248 81 184 71 15 127 0 0 0 0 0 0 0 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 Error: Failed to decompress data
    at Dbi.getByBinary (<anonymous>)
    at LMDBStore.getBinaryFast (/app/node_modules/lmdb/dist/index.cjs:1295:26)
    at LMDBStore.get (/app/node_modules/lmdb/dist/index.cjs:1339:22)
kriszyp commented 2 years ago

Thank you for the logs. From what I can tell, the compressed data is actually valid, but somehow it is has been saved with the wrong size (for example the first one's uncompressed size is 2302, but the compressed size should be 1157, but is still being saved with a compressed size of 2302).

A few follow-up questions: Is there anything in any of the logs about compression errors? The specific strings to look for would be "failed to compress" or "compression not completed". And do you know if the process ends by an explicit process.exit() (or child_process.kill()), or by waiting for the event queue to finish? And are there potentially any attempts to do put calls after the close during the shutdown process?

gogoout commented 2 years ago
kriszyp commented 2 years ago

I made a couple of changes in v2.1.6 that might help, although I still have not figured out what could be going wrong.

gogoout commented 2 years ago

Much appreciated, we may have a chance to properly test this from beginning again maybe in 2 weeks. Will keep posting.

gogoout commented 2 years ago

Some updates here, seems since some version (we jumped from 2.1.7 to 2.2.1). The error in the log seems disappeared. Closing the issue for now as seems it's resolved now. Thanks very much for the hard working!

kriszyp commented 2 years ago

That's great to hear, thank you for confirming this!