search

open-contracting / kingfisher-process

Stores and pre-processes OCDS data in a SQL database
https://kingfisher-process.readthedocs.io
BSD 3-Clause "New" or "Revised" License
2 stars 8 forks source link

Add configuration option for incremental compilation #320

Closed jpmckinney closed 2 years ago

jpmckinney commented 4 years ago

When new data is added to a collection, instead of re-compiling all releases, the previously compiled releases can simply be updated with the new releases (see https://ocds-merge.readthedocs.io/en/latest/update-merged-releases.html).

This is relevant to https://github.com/open-contracting/kingfisher-collect/issues/484 after the initial crawl.

jpmckinney commented 3 years ago

@jakubkrafka This will be an important feature for an upcoming project. It's out-of-scope of our current contract, but we can make a contract extension. For now, I'd just like your quick thoughts on feasibility.

jpmckinney commented 2 years ago

Since #276 is closed for lack of demand, this issue is also closed.

Kingfisher Collect has a separate solution for incremental crawls (where it re-compiles everything).

So far, compilation has not been a bottleneck (as mentioned in #330), so we also don't need this optimization.