Data Update is not happening with scheduled jobs

datopian / assembler

The DataHub data assembly line

MIT License

10 stars 2 forks source link

Data Update is not happening with scheduled jobs #56

Closed zelima closed 7 years ago

zelima commented 7 years ago

One of our core datasets finance-fix that should be updated daily, is stuck on 2nd of October - the scheduled job is running but data is never updated.

Acceptance criteria

[x] we have the lates data after scheduled run

Tasks

[x] ~~reproduce locally~~
[x] investigate reason (see analysis)
[x] fix

Analysis

This is happening cause only "main" pipeline is running on scheduled time and derived ones remain same

zelima commented 7 years ago

FIXED. Now derived pipelines are run on schedule and main pipelines is run automatically as it is marked as dirty. See finance VIX https://datahub.io/core/finance-vix