Crashes on Windows 10 with intensive tasks

[KaffinPX]

const { ViteAPI, accountBlock } = require('@vite/vitejs')
const { HTTP_RPC } = require('@vite/vitejs-http')
const lmdb = require('lmdb')

const config = require('./config.json')
const cycle = require('./src/vite/cycle')

const db = lmdb.open('./ledger', {
  cache: true,
  strictAsyncOrder: true,
  separateFlushed: true,
  commitDelay: 1000,
  noSync: true,
  noMemInit: true,
})
const vite = new ViteAPI(new HTTP_RPC('https://node-vite.thomiz.dev'))

const syncIndexer = async () => {
  const lastHeight = db.get('lastHeight') ?? BigInt(config.initialHeight)
  const targetHeight = BigInt(await vite.request('ledger_getSnapshotChainHeight'))

  if (lastHeight + 1000n < targetHeight) {
    console.log(`JOB: Sync from height ${lastHeight} to height ${targetHeight}`)

    for (let index = lastHeight; index + 1000n < targetHeight; index += 1000n) {
      const chunks = await vite.request('ledger_getChunksV2', index.toString(), (index + 999n).toString())

      for (const chunk of chunks) {
        const snapshotBlock = chunk.snapshotBlock
        const currentCycle = cycle.getCycle(snapshotBlock.timestamp)
        let cycleData = db.openDB('cycles').get(currentCycle)

        if (typeof cycleData === 'undefined') {
          cycleData = {
            walletCount: db.openDB('accounts').getStats().entryCount,
            interactedWallets: [],
            newWallets: 0
          }
        }

        for (const account in snapshotBlock.snapshotData) {
          const snapshotContent = snapshotBlock.snapshotData[account]
          let accountData = db.openDB('accounts').get(account)

          if (typeof accountData === 'undefined') { accountData = {} }
          if (typeof accountData[currentCycle] === 'undefined') { accountData[currentCycle] = 0 }

          if (snapshotContent.height === 1 || typeof db.openDB('accounts').get(account) === 'undefined') {
            cycleData.walletCount += 1
            cycleData.newWallets += 1
          }

          accountData[currentCycle] += 1
          if (!cycleData.interactedWallets.includes(account)) { 
            cycleData.interactedWallets.push(account)
          }

          await db.openDB('accounts').put(account, accountData)
        }

        await db.openDB('cycles').put(currentCycle, cycleData)
      }

      console.log(`JOB: Got blocks between ${index} - ${index + 1000n}(Target ${targetHeight})`)
      await db.put('lastHeight', index + 1000n)
    }

    console.log('JOB: Target reached, waiting next job...')
  }

  setTimeout(() => { syncIndexer() }, 10000)
}

syncIndexer()

I played with LMDB parameters but didnt worked...

[kriszyp]

I haven't been able to reproduce this with your code as of yet (been running it for a while now). What version of Node are you using? One thing the stack trace suggests is that it might be related to the turbo API calls. You might try deleting @lmdb/lmdb-win32-x64/node.abi*.node (and leaving it run with just the NAPI binary), and see if that makes any difference.

[KaffinPX]

Im using node 16, will try thanks :).

[KaffinPX]

Yep it works, thanks a lot.

[kriszyp]

OK, that's helpful to know. How long does it typically take to reproduce this error? And I kind of made some guesses on some of your code, setting the initialHeight to 10. But I don't have ./src/vite/cycle (instead I just used the timestamp as the currentCycle). Maybe using that is important to reproducing this?

[KaffinPX]

Yep you need a reliable cycle, it tooks about 10 seconds to crash.

cycle.js:

const cycleEpoch = new Date("2019-05-21T12:00:00+08:00").getTime()

module.exports = {
  getCurrentCycle: () => {
    return getCycle(Date.now())
  },
  getCycle: (timestamp) => {
    const elapsedTime = timestamp - cycleEpoch
    const days = Math.floor(elapsedTime / 24 / 60 / 60 / 1000)

    return days
  },
  getDateFromCycle: (cycle) => {
    const elapsedTime = cycle * 24 * 60 * 60 * 1000

    return new Date(cycleEpoch + elapsedTime)
  }
}

[kriszyp]

I have been running it for several hours now, without any errors:

> node .\test-vite.js
JOB: Sync from height 218100 to height 105822815
JOB: Got blocks between 218100 - 219100(Target 105822815)
JOB: Got blocks between 219100 - 220100(Target 105822815)
JOB: Got blocks between 220100 - 221100(Target 105822815)
JOB: Got blocks between 221100 - 222100(Target 105822815)
JOB: Got blocks between 222100 - 223100(Target 105822815)
JOB: Got blocks between 223100 - 224100(Target 105822815)
JOB: Got blocks between 224100 - 225100(Target 105822815)
...

[KaffinPX]

Weird, i think its because network is unused at these heights, gonna give a specific height to you soon

[KaffinPX]

Try height 105400000

[kriszyp]

Still no crashes...

JOB: Sync from height 105400000 to height 106266192
JOB: Got blocks between 105400000 - 105401000(Target 106266192)
JOB: Got blocks between 105401000 - 105402000(Target 106266192)
JOB: Got blocks between 105402000 - 105403000(Target 106266192)
JOB: Got blocks between 105403000 - 105404000(Target 106266192)
JOB: Got blocks between 105404000 - 105405000(Target 106266192)
JOB: Got blocks between 105405000 - 105406000(Target 106266192)
JOB: Got blocks between 105406000 - 105407000(Target 106266192)
JOB: Got blocks between 105407000 - 105408000(Target 106266192)
JOB: Got blocks between 105408000 - 105409000(Target 106266192)
JOB: Got blocks between 105409000 - 105410000(Target 106266192)
JOB: Got blocks between 105410000 - 105411000(Target 106266192)
JOB: Got blocks between 105411000 - 105412000(Target 106266192)
JOB: Got blocks between 105412000 - 105413000(Target 106266192)
JOB: Got blocks between 105413000 - 105414000(Target 106266192)
JOB: Got blocks between 105414000 - 105415000(Target 106266192)
JOB: Got blocks between 105415000 - 105416000(Target 106266192)
JOB: Got blocks between 105416000 - 105417000(Target 106266192)
JOB: Got blocks between 105417000 - 105418000(Target 106266192)
JOB: Got blocks between 105418000 - 105419000(Target 106266192)
JOB: Got blocks between 105419000 - 105420000(Target 106266192)
JOB: Got blocks between 105420000 - 105421000(Target 106266192)
JOB: Got blocks between 105421000 - 105422000(Target 106266192)
JOB: Got blocks between 105422000 - 105423000(Target 106266192)
JOB: Got blocks between 105423000 - 105424000(Target 106266192)
JOB: Got blocks between 105424000 - 105425000(Target 106266192)
JOB: Got blocks between 105425000 - 105426000(Target 106266192)
JOB: Got blocks between 105426000 - 105427000(Target 106266192)
JOB: Got blocks between 105427000 - 105428000(Target 106266192)
JOB: Got blocks between 105428000 - 105429000(Target 106266192)
JOB: Got blocks between 105429000 - 105430000(Target 106266192)
JOB: Got blocks between 105430000 - 105431000(Target 106266192)
....

[kriszyp]

Just in case this makes a difference, what specific version of Node v16 are you using?

[KaffinPX]

[kriszyp]

Oh that's the problem. I can immediately reproduce your issue on that version. Earlier minor versions of Node definitely had some incompatibilities in fast API interfaces. And therefore the fast API is actually disabled for versions earlier than 16.7. I guess that should have been a little broader, so I will disable fast API for version 16.8 and earlier. Of course I'd also recommend upgrading your Node :), but I certainly understand that's not always possible.

[KaffinPX]

Gonna update my node soon, thanks for suggestion and thanks for help ;)

[kriszyp]

This fix should be in v2.6.9, btw.

[KaffinPX]

Ty