search

alphagov / trade-tariff-backend

Enabling the population and distribution via API of UK Customs tariffs and duties
MIT License
7 stars 6 forks source link

Snapshot updates #40

Closed jabley closed 11 years ago

jabley commented 11 years ago

Initial draft of archiving updates.

This is brute force stupid. It will download all items from a given date, not checking the database or filesystem to see if it already has the file locally.

It should only need to be run a single time, to grab all the items from a particular date. After that, we should just get the daily updates stored in the same location.

https://www.pivotaltracker.com/story/show/40611041

I've punted on tests - assuming that the existing tests cover most of the functionality already, and this is intended to be a one-off task to slurp the files for now. We know the download and store code works from having a working system in production.

saulius commented 11 years ago

This looks good. Btw, there is no point in going beyond https://github.com/alphagov/trade-tariff-backend/blob/master/lib/tariff_synchronizer.rb#L59 these two days for Taric and CHIEF respectively, as we have the initial seeds from those dates.

Btw added https://github.com/alphagov/trade-tariff-backend/wiki/Importing-CHIEF-initial-seed-from-CSV just for reference in case it will be needed.

jabley commented 11 years ago

Thanks for the documentation. How do we get the CSV files? Do HMRC provide an Oracle .dmp file and we have to then import that into an Oracle instance and generate the CSV ourselves?

saulius commented 11 years ago

@jabley you generated those CSVs for us :-)

I still got Oracle binary files (not .dmp, no extension), I can send them to you, because I don't know how the magic happens.