search

open-contracting / data-registry

BSD 3-Clause "New" or "Revised" License
4 stars 0 forks source link

Consider replacing kingfisher-process with ocds-merge-rs in pipeline #292

Open jpmckinney opened 1 year ago

jpmckinney commented 1 year ago

I was working on a Rust rewrite of OCDS Merge (need to add a CLI and compare performance to the Python version). Once that's done, I can also explore replacing Kingfisher Process (that would mean using OCDS Kit to upgrade the few collections that are OCDS 1.0 – if that's too slow, I can write a Rust version).

Having fewer items in the queues should increase the stability of RabbitMQ. (If RabbitMQ continues to fall over, maybe we switch to Redis, where enabling its durability features won't be as much of a performance concern, as we'll just have 1 message per job task per publication).

Similar issue for data support server: https://github.com/open-contracting/deploy/issues/437

jpmckinney commented 7 months ago

Noting that since upgrading to yapw 0.1.0 in open-contracting/kingfisher-process@3b7dfc52fd5d1dd0d1c8a3d7e2e3e440d4a0e152 (i.e. since using async consumers, as appropriate), we haven't had RabbitMQ issues.

The main reason for a Rust version, then, is that for large collections, we end up using a lot of CPU (triggering alerts), with RabbitMQ queueing over 1M messages in the release compiler. A Rust version means we use less CPU and finish faster.