Investigate getting more up-to-date tender data directly from other source

[struan]

Mostly because at the time of writing the opendata coop data we're using is 2 months old which is not ideal. There do seem to be two sources, one of which (https://ocds-downloads.opendata.coop/source/united_kingdom_contracts_finder_releases) is more up to date.

[JenMysoc]

Who is assigned to this ticket and is it in the plan to be finished by tomorrow? If not I'll move down and we can discuss if it's a definite need for sprint starting 26/9

[struan]

No one is assigned, so no, plan is not to get it finished :|

[struan]

This ticket consists of two part:

1 - find a more up to date source for contract data. It might be we can run the pipeline that is used by the opendata coop data ourselves, or we might have to find an alternative source and write a scraper. 2 - assess how much work there is in changing to the new data source. Ideally this should generate data in the same form that we currently ingest it.

Once we've done that I suggest a new ticket to cover implementing this.

This is partly related to #12, #13, #14

[zarino]

I haven’t checked out the official UK Contracts Finder API yet, but it supposedly returns OCDS format data. It feels like if we’re going to use anything, this would be it. (The only reason I didn’t use it in the prototype was that Open Data Services’ SQLite dump was easier to work with quickly.)

The government also produce daily CSV dumps, which seem to be pretty up to date (latest one was yesterday), but I’m not sure how you’d programmatically "discover" those daily CSVs.

[alexander-griffen]

Following an email from Duncan Dewhurst, we have been pointed to these APIs:

• Contracts Finder OCDS API
  • (This is what Zarino links to in the above comment)
• Find a Tender Service OCDS API
• Public Contracts Scotland OCDS API
• Sell2Wales OCDS API

The reason for the Downloads not being updated is a non-backward-compatible change the government has made to the Contracts Finder OCDS API.

[zarino]

@alexander-griffen and I had a chat with Open Data Services about this. Full notes here, but a summary is:

• If you scrape your contracts data from the Contracts Finder OCDS API and Find a Tender Service OCDS API, then you should get all the contracts for the UK. The Scottish and Welsh APIs just replicate the same data, for those countries.
• Tenders may be duplicated between Contracts Finder OCDS API and Find a Tender Service OCDS API, with different release IDs (!!) so the only way to detect the duplicates is to match on tender names.

[alexander-griffen]

There's an open PR that partially solves this ticket, but I think there's a better way which I've written about in more detail..

When this is implimented, I think it could be automated so that data updates can be done, for example, daily.

Also, when it's implimented, it may shed some light on better ways of dealing with some of the weirdness (like for example when tenders are displayed as having a value of £0), by inspecting the dataset as a whole.

[alexander-griffen]

Closing, because there has been more up to date tender data imported (and opening a new ticket for improvements)