search

ska-sa / katdal

Data access library for the MeerKAT radio telescope
BSD 3-Clause "New" or "Revised" License
12 stars 13 forks source link

Replace underscores with dashes when loading buckets from RDBs #368

Closed ctgschollar closed 6 months ago

ctgschollar commented 1 year ago

The new Ceph installation at MeerKAT has done away with underscores in bucket names. This is to make the ceph s3 comply with the s3 protocol.

That means that some of our legacy data from before 2019 has had their buckets renamed, but the RDBs have not been updated to reflect that.

Eg. http://archive-gw-1.kat.ac.za:7480/1540580961_sdp_l1_flags is now http://archive-gw-1.kat.ac.za:7480/1540580961-sdp-l1-flags

Link to the example in the web-archive. https://archive.sarao.ac.za/search/1540580961/ This data is currently in ceph, so a good test.

We need to make a fix for katdal to replace underscores with dashes for all buckets accessed directly from the archive.

I am happy to test this out on our transfer nodes once you have an update for this.

ludwigschwardt commented 1 year ago

Thanks @ctgschollar! Could you copy and paste this into an SPR1 JIRA ticket as well?

ctgschollar commented 1 year ago

Here you go https://skaafrica.atlassian.net/browse/SPR1-2990

spassmoor commented 1 year ago

Hi @ludwigschwardt This is quite urgent now as the Galactic plane legacy survey us about to be released/published and a large chunk of that data is pre-2019

ludwigschwardt commented 6 months ago

Fixed by #370.