uktrade / public-data-api

The source for the Department for International Trade's Public Data API
https://data.api.trade.gov.uk/
MIT License
5 stars 1 forks source link

Supplementary unit for a commodity #186

Open ashickton opened 1 year ago

ashickton commented 1 year ago

How can I find the supplementary unit (e.g. NAR, KGM) associated with a commodity in the uk-tariff-2021-01-01 dataset?

I have looked through all the available tables and relationships relating to measures but struggling to see how this can be done.

Thanks Andrew

SteveCN7 commented 1 year ago

Hi Andrew,

In the dataset, the supplementary units, unit of measurement is held on the measure rather than directly against the commodity code.

Supplementary units can be identified by the measure type from the measures table, this should be 109 (although 110 and 111 are technically usable).

The measures table will have an identifier for the commodity code (goods nomenclature) that identifier is a SID rather than the description, for example

Measure SID = 20203097 Goods Nomenclature SID = 106689 (Comm code is 4015120000 (Of a kind used for medical, surgical, dental or veterinary purposes))

The unit of measurement for the supplementary unit will be held as a measure component or measure condition component as a duty expression. A duty expression can be anything from a unit of measurement to compound duties. The unit of measurement is a unique identifier of three digits such as NAR, KGM etc

You should be able to identify the supplementary units via this table

Please let me know if this helps or not

Many Thanks

ashickton commented 1 year ago

Hi Steve,

Thank you for the pointers. I have managed to work this through.

Just wondering also, what is are the SID columns used for?

Are these static identifiers? Such as 109 to identify the measurement type for a supplementary unit.

Cheers, Andrew

SteveCN7 commented 1 year ago

Hi Andrew - sorry for the delayed response, the SID columns are unique identifiers for specific items. when used in a measure it helps us to identify the measure and the links to it's various component entities. Commodity codes also had SIDs which we use to identify the correct Commodity code where the commodity code has both a suffix 10 and suffix 80 version

adis-io commented 2 months ago

Hi @SteveCN7

I have a couple of questions regarding fetching supplementary units and would be glad if you could answer them.

1) I was working on dataset version v4.0.246 and then discovered another version. Which one should I use to identify supplementary units for commodity codes? The latter seems to contain less data.

2) I've found instances where I can see supplementary unit for commodity code in trade-tariff.service.gov.uk (for example, this), but not in the uk-tariff-2021-01-01 dataset. Which one has the most up-to-date data?

SteveCN7 commented 2 months ago

Hi @adis-io

The Tariff data updates nearly daily depending on various changes to the tariff as changes to legislation, poilcy, administration etc are recorded in the database.

To answer the questions, I've made the assumption that you are using the dataset titled Measures as Defined.

  1. If you update the URL to include /latest/ where the version is present, you'll always be provided the most recent version available on the API, this API also provides the data to the second hyperlink you've identified. We do have a lag issue we're working through.
  2. This is where the Measures as Defined dataset might be the issue. Measures as Defined, shows you where in the commodity code hierarchy the measure has been placed. In the data schema we use (TARIC3) Measures applied at high level in the hierarchy get applied to those below it. So in the example you've provided:
    • Supplementary unit measure present on 3918101010
    • Online Tariff Tool shows this (trade-tariff.service.gov.uk)
    • The measure itself isn't applied directly to 3918101010, but instead to code 3918000000 I've had a look and can see the directly applied measure is missing from the dataset, I think this is because it's a measure we've inherited from before we left the EU. I'm going to raise this for the team to confirm and resolve. The Online Tariff Tool is provided by HMRC, but all tariff data is provided by data from the datasets provided by our platform. So the data should be available at the same time.

Please let me know if I can help further

adis-io commented 2 months ago

Hey @SteveCN7, thank you very much for your prompt response.

I was trying to use the "Measures on declarable commodities" table, but you've already given me an answer for the missing data, indicating it might be related to pre-Brexit inheritance.

For now, I'm considering generating a CSV using the tariff API to cover missing data and then periodically updating it by fetching data from the "Measures on declarable commodities" table. Does this sound like a correct approach?

SteveCN7 commented 2 months ago

Hi @adis-io

Pre-Brexit inheritance seems to be the issue, but I'll confirm with our data engineers that is definitely the case.

That sounds like a sensible approach for now. We're about to engage in an overhaul of our open data provisioning and one thing we're considering is making a "whole" file available and then offering an API end point to provide updates.

We're going to start doing some discovery soon to test viability for this and other potential improvements.

Many Thanks

adis-io commented 2 months ago

Thank you so much @SteveCN7