search

alphagov / trade-tariff-backend

Enabling the population and distribution via API of UK Customs tariffs and duties
MIT License
7 stars 6 forks source link

Chief transformer refactor #37

Closed saulius closed 11 years ago

saulius commented 11 years ago

Okay this fix was long due. It is related to the email I sent a while ago. Two major things here:

  1. Certain TARIC updates may invalidate national measures apparently. GoodsNomenclature associated to Measure should span its validity dates (ME8 conformance test). So say we create Measure for GoodsNomenclature that has open ended validity end date. Later Taric updates that GoodsNomenclature to have certain validity date. Some of the national Measures can become invalid. On top of that, CHIEF may try to update those invalid measures later on (just what happened actually). So to prevent it we check for national measure validity after GoodsNomeclature gets updated and do a soft delete (set invalidated by to Taric transaction id) for the measure. Later on, we ignore validations for such measures in case CHIEF will try to update them, so we just allow that. I found that there were 56 national measures invalidated up to 29th of Nov.
  2. Consolidate update application and transformation, so that transformation is invoked after CHIEF updates get applied. This would eventually become an issue for snapshot recreation (in case of need) process. We used to apply all Taric updates, then all CHIEF updates and then do transformation in one go for all CHIEF records. Imagine a scenario where Taric introduces Commodity on 1st of July, CHIEF adds national measures on say the 2nd of July and Taric removes Commodity on 1st of August. If we would apply all Taric's updates then we wouldn't even create CHIEF measure as if that Commodity was never there.
saulius commented 11 years ago

I added a snapshot (snapshot_for_29-11-2012.sql.bz2) that has up to date data including invalidated measures.

jabley commented 11 years ago

Where possible I would like to add associated tests in smokey or similar. Currently some of our tests are ping-like (is the service there?) rather than checking the content of the page.

saulius commented 11 years ago

I'm not sure tests on content are a good thing. Both CHIEF and Taric change that data from time to time so tests may fail even if nothing wrong happened. Do you still want me to list some examples?