dcp-793 Added script to fill dcp1 file metadata.

ebi-ait / hca-ebi-dev-team

Repository for hca ebi dev team agile management. See zenhub board

0 stars 0 forks source link

dcp-793 Added script to fill dcp1 file metadata. #508

Closed ESapenaVentura closed 1 year ago

ESapenaVentura commented 2 years ago

Added script to fill DCP1 file metadata when it's missing file size or content type

ESapenaVentura commented 2 years ago

I think I have addressed all comments (Sorry If I missed any!!!)

Let me know what you think now

ESapenaVentura commented 2 years ago

I have added a couple of bits of functionality (Basically, a better query to reduce the number of files requested for azul, a TQDM progress bar for impatient people like me and a function to deal with pagination in azul)

I am testing this script against submission https://contribute.data.humancellatlas.org/submissions/detail?uuid=0948a727-228f-4cfc-857e-6243c6aed08d&project=2043c65a-1cf8-4828-a656-9e247d4e64f1

Will push the changes once it's done patching the files (Inconsistent speed, between 1.33 and 2.5 it/s)

ESapenaVentura commented 1 year ago

@amnonkhen I have:

Made the azul URL pointer always use the config value
Make the function that patches the metadata return a namedtuple
Re-factored the write function so that takes a namedtuple

And left a couple comments on things I did not fully understood your comment

Let me know!

amnonkhen commented 1 year ago

I will schedule a meeting to iron everything out.

ESapenaVentura commented 1 year ago

@amnonkhen I applied the changes, please give green light when possible so that I can merge!