opentargets / issues

Issue tracker for Open Targets Platform and Open Targets Genetics Portal
https://platform.opentargets.org https://genetics.opentargets.org
Apache License 2.0
12 stars 2 forks source link

Migrate pan-UKBB LD matrixes from AWS to GCP #3455

Open d0choa opened 1 week ago

d0choa commented 1 week ago

We would like to have panUKBB LD matrixes to improve our finemapping process.

3 ancestries: EUR, AFR, CSA

s3://pan-ukb-us-east-1/ld_release/UKBB.EUR.ldadj.bm s3://pan-ukb-us-east-1/ld_release/UKBB.AFR.ldadj.bm s3://pan-ukb-us-east-1/ld_release/UKBB.CSA.ldadj.bm and s3://pan-ukb-us-east-1/ld_release/UKBB.EUR.ldadj.variant.ht s3://pan-ukb-us-east-1/ld_release/UKBB.AFR.ldadj.variant.ht s3://pan-ukb-us-east-1/ld_release/UKBB.CSA.ldadj.variant.ht (edited)

@prashantuniyal02 This is going to be costly, so we will need estimates on transfer cost and storage. We can move to auto-class, and the expectation is that we might only access the data once every quarter. But let's try to make this happen

prashantuniyal02 commented 5 days ago

Hi @addramir , I have transferred all the 3 variant.ht (EUR, AFR, CSA) and UKBB.CSA.ldadj.bm to our GCP. You can start with that ancestry. I just need to monitor any associated costs for the next couple of days.

addramir commented 5 days ago

Hi @prashantuniyal02 , thank you! could you please also move EUR (nfe)? It is priority 1.

prashantuniyal02 commented 4 days ago

@addramir , I have also moved the UKBB.EUR.ldadj.bm file to gs://panukbb-ld-matrixes GCP bucket