Change how BigQuery instance is created in `platform-output-support`

[andrewhercules]

Currently, our BigQuery open-targets-prod instance has versioned tables for each release (e.g. platform_21_04).

However, for the Google Marketplace public dataset listing programme, our tables should be consistent from release to release and archives stored in GCP buckets.

Can we please update platform-output-support so that our open-targets-prod instance no longer has the data version?

[mkarmona]

@cmalangone translating this into a set of actions so you can activate this

1. after each data release we create in BQ with the Open Targets versioned dataset as we are currently doing but in another project because the Google Marketplace policy needs
2. clean the open-targets-prod BQ database and always recreate the same datasets without versioning in the name so it will always be one DB with the same datasets and same tables in it

Questions you might have?

• what if the schema changes? They don't mind really, they just want the dataset naming the same as they can publish it through their channels
• what if we create a new table inside? It is ok as long as the project and the dataset remains the same

[cmalangone]

@andrewhercules provides us the list of tables that must be listed https://github.com/opentargets/platform-app/blob/main/src/pages/DownloadsPage/dataset-mappings.json

[andrewhercules]

BigQuery updated, ticket closed

[cmalangone]

@andrewhercules I am finishing to add the code to POS :)

[andrewhercules]

No worries - I was too quick with closing! 😅

If possible, can you please also delete the platform_21_06, platform_21_09, and targval_quicksearches datasets?

I'm not sure how the Google public dataset pipeline works, but I figure it would be best if open-targets-prod only contains the one platform dataset.

That being said, I also understand if we use BigQuery to keep a copy of the two most recent releases — just please let me know so I can liaise with Google.

[cmalangone]

I deleted the old bigquery datasets. We will talk about it for shaping the POS output in a better way next release. I am changing the code of POS to generate just platform dataset with a ot_release table with the info about the release.

[andrewhercules]

A user has contacted the helpdesk about the knownDrugsAggregated data from our 21.11 release. The data is not available in Google BigQuery.

@cmalangone can you please check if POS is also adding that dataset to our open-targets-prod.platform BigQuery instance? If not, is this something we can manually do for this release?

[cmalangone]

I've created the dataset manually. The issue was that the file dataset-mappings.json has the dataset filtered as include_in_bq == false @andrewhercules can you please create a ticket?

[andrewhercules]

Thank you @cmalangone! 😄

I have updated #1790 and will adjust the mapping file for the 22.02 release.

[andrewhercules]

Ticket closed as datasets generated for BigQuery.

Additional work will be captured in #1790.