search

alphagov / trade-tariff-backend

Enabling the population and distribution via API of UK Customs tariffs and duties
MIT License
7 stars 6 forks source link

Extract invalid national measure cleaner to a task. #29

Closed saulius closed 11 years ago

saulius commented 11 years ago

Extracted lengthy migration to a task.

jabley commented 11 years ago

My understanding is that Sequel.migration blocks get run under a transaction (if the underlying data store supports that).

I don't think we have that with this task?

saulius commented 11 years ago

Yes, you are right. Wrapped into transaction. Hopefully we won't any kind of transaction buffer size limit with this.

jabley commented 11 years ago

If we have the transaction scope as a batch of 1000 rather than the whole job, that will presumably avoid the possibility of hitting any transaction size limit (or killing performance by building a massive in-memory transaction)?

saulius commented 11 years ago

Yes, that's safe to assume. What about state then, the job may finish in half-done state in case of an error. We can rerun it of course. Is that okay? If so I will move transaction into batch scope.

jabley commented 11 years ago

That sounds good. The operation is idempotent, which is always goodness.

It should run and affect lots of measures initially, then should do essentially nothing when run subsequently (unless we are importing new measures in this state, which is probably a bug).

saulius commented 11 years ago

Okay, I moved the transaction block.