Non-existing or expired scroll identifier when exporting collection / index from Kourou

tdislay commented 1 year ago

Expected Behavior

kourou index:export tenant-[group]-[name] Should be able to dump collections with more than 2000 documents. Same thing with kourou collection:export tenant-[group]-[name] [my-collection].

Current Behavior

I get the following error while dumping a collection with 13398 documents:

[X] Error stack: 
KuzzleError: Non-existing or expired scroll identifier.
       at WebSocketProtocol.query (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/node_modules/kuzzle-sdk/src/protocols/abstract/Base.js:109:23)
       at Kuzzle._timeoutRequest (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/node_modules/kuzzle-sdk/src/Kuzzle.js:742:34)
       at Kuzzle.query (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/node_modules/kuzzle-sdk/src/Kuzzle.js:575:21)
       at DocumentSearchResult.next (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/node_modules/kuzzle-sdk/src/core/searchResult/SearchResultBase.js:41:33)
       at JSONLDumper.dump (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/lib/support/dump-collection.js:105:43)
       at processTicksAndRejections (node:internal/process/task_queues:96:5)
       at async CollectionExport.runSafe (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/lib/commands/collection/export.js:30:9)
       at async CollectionExport.run (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/lib/common.js:98:17)
       at async CollectionExport._run (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/node_modules/@oclif/command/lib/command.js:43:20)
       at async Config.runCommand (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/node_modules/@oclif/config/lib/config.js:173:24)
       at async Main.run (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/node_modules/@oclif/command/lib/main.js:27:9)
       at async Main._run (/home/skymer/.nvm/versions/node/v16.16.0/lib/node_modules/kourou/node_modules/@oclif/command/lib/command.js:43:20)
Error status: 404
Error id: services.storage.unknown_scroll_id (https://docs.kuzzle.io/core/2/api/errors/error-codes/services)
 [X] undefined

(I've tried with node 14 & updating kourou & using batch-size, but it has a limit)

Steps to Reproduce

Import a collection with more than 2000 documents (see documents.jsonl.txt and mappings.json.txt) into your kuzzle: kourou collection:import [../../]
Export the collection kourou collection:export tenant-[group]-[name] [collection]

Context (Environment)

Trying to export a collection in order to backup data NodeJS 14 / 16 Kuzzle: 2.19.5 Kourou : kourou/0.24.2 linux-x64 node-v14.20.1

Aschen commented 1 year ago

Seems related to https://github.com/kuzzleio/kourou/issues/159, can you try extending the scroll duration with scrollTTL parameter?

tdislay commented 1 year ago

My connection is too slow, that's the problem

Aschen commented 1 year ago

Then you should increase the scroll TTL so it won't timeout

kuzzleio / kourou