search

kalisio / krawler

A minimalist (geospatial) ETL
https://kalisio.github.io/krawler/
MIT License
54 stars 13 forks source link

Concurrent tasks skipped when a task fails #33

Closed claustres closed 6 years ago

claustres commented 6 years ago

Steps to reproduce

Launch a job with eg 4 concurrent tasks, make one of the task fail

Expected behavior

The others tasks are correctly ran and reported to have been ran

Actual behavior

The tasks of the same concurrency bucket that should run after the failure are skipped, this is because Promise.all() here rejects with the reason of the first promise that rejects.

System configuration

Tell us about the applicable parts of your setup.

Module versions : 0.7

NodeJS version: 8

Operating System: Window

claustres commented 6 years ago

This is actually correctly managed for fault-tolerant jobs (see https://github.com/kalisio/krawler/blob/master/src/jobs/jobs.async.js#L21). Otherwise it is normal to stop as soon as an error is raised.