TromsFylkestrafikk / ragnarok

Harvest public transport data for statistics usage
0 stars 0 forks source link

Rescan sink chunks and add 'modified' flag #51

Closed tfk-kaare closed 8 months ago

tfk-kaare commented 8 months ago

It should be possible to rescan sources to compare stored stage 1 data with new stage 1 data, and add a 'modified' flag on these chunks. This should then be used as a configurable scheduled task to re-scan source data from time to time to give an indication of updated content.

Remaining tasks

tfk-kaare commented 8 months ago

Comparing upstream chunk with locally stored chunk involves downloading the actual chunk. If we're already downloading it, we simply save it. I.e. no need for a 'comparison' of upstream vs local. Instead, we add a version on the downloaded chunk, and use this as value for the actual imported data. Whenever chunks are re-downloaded, the version will differ if it is modified upstream, giving the user an orange badge over the fetch status chip.

Remaining task, and maybe follow-up issue is to be able to filter this state. I will not add 'updated' as an additional value of fetch_status. It feels like a hack. Instead, the chunk model now has an accessor and query scope to handle these situations.