search

kalisio / krawler

A minimalist (geospatial) ETL
https://kalisio.github.io/krawler/
MIT License
54 stars 13 forks source link

When a job timeout, krawler hangs with the following error: Cannot read property 'type' of undefined #135

Closed robinbourianes-kalisio closed 3 years ago

robinbourianes-kalisio commented 3 years ago

Running hubeau krawler job with DEBUG=*, the last log entry before the crash is the following:

krawler:jobs 1970 tasks ran from start in 1800106.770962 ms +0ms @feathersjs/errors Timeout(408): Job timeout reached +0ms @feathersjs/errors {} +1ms krawler:jobs Some tasks failed Timeout: Job timeout reached at new Timeout (/hdd/robin/Taff/kalisio/krawler/node_modules/@feathersjs/errors/lib/index.js:135:17) at Object. (/hdd/robin/Taff/kalisio/krawler/lib/jobs/jobs.async.js:81:68) at Generator.next () at step (/hdd/robin/Taff/kalisio/krawler/lib/jobs/jobs.async.js:112:191) at /hdd/robin/Taff/kalisio/krawler/lib/jobs/jobs.async.js:112:361 at runMicrotasks () at processTicksAndRejections (internal/process/task_queues.js:97:5) { type: 'FeathersError', code: 408, className: 'timeout', data: undefined, errors: {} } +1ms

Without DEBUG=* the only line we get is Cannot read property 'type' of undefined which is printed from here: https://github.com/kalisio/krawler/blob/f86541ae469f65832840bf7abb42a1d32912d596/src/cli.js#L148

robinbourianes-kalisio commented 3 years ago

The type property the code tries to read is from here Somehow throwing the timeout error from here mess up stuff in feathers. Feathers god help requested ! @claustres

claustres commented 3 years ago

If the error hooks are removed from the job the error goes away, it is probably related to the fact some of our internal hooks do not correctly manage the error hook case.