Database locks for download and sync tasks.

[saulius]

This is an extraction from previous pull request. This makes backend ready for cluster like environment so that only single node downloads the files, creates entry in tariff_updates, applies the update and reindexes.

[jabley]

I'm a little unclear how this works in a cluster-safe fashion.

• We have a cron job that downloads the updates.
• We have another cron job that applies the updates.

Node 1 could download the updates in a cluster-safe fashion, and Node 2 could attempt to try the updates in a cluster-safe fashion, but Node 2 won't have the update files.

If my understanding is correct, then we need to think again about this. I still don't think storing the zip files in the database is a good solution.

[saulius]

Yes, we agreed that storing files in the DB is out of question now.

Would gluing download & application processes together solve this? Then CHIEF transformation could be left on its own, but still under lock, so that other nodes wouldn't execute duplicate transformation operations.

The ideal way would be store update files in 3rd party independent storage like local S3 alternative. But I don't think there is anything like that in the infrastructure, or is there?

[matthewford]

@jabley we don't have an object store at the moment right? @sauliusg I say lets proceed to merge the two rake tasks

[jabley]

Yeah, the object store isn't available yet.

Merging the download and process seems like the pragmatic thing to do.

[jabley]

Just confirming that we have other things to do before this; what we have currently should work and we don't need to rush in cluster-safe functionality.

[saulius]

Merged download & apply into tariff:sync:apply. Transform step will happen at 6am.