glugeorge
commented
1 year ago @jkingslake I wanted to make sure I'm using the right functions to first upload the individual zarrs. I will be testing this by only uploading one of the subfolders within the Card A folder. From what I can tell, the function you used to upload the previous individual zarrs was xapres_to_zarr_chunked in the to_individual_zarr notebook. Outside of the explicit chunk definition, this function also uses tqdm_slack rather than just tqdm compared to xapres_to_zarr. Could you clarify this difference?
On a separate note, I think I will need to delete and reupload the winter data to a new directory that is outside the Card A directory if we want to use the same functions as before to create zarrs for just the winter data. Do you have experience moving entire folders within google bucket? It seems quite different from moving individual objects.
Currently the raw DAT files are stored on the bucket, within ldeo-glaciology/GL_apres_2022. It is therefore mixed in with the existing data.
My plan is to first do a test upload with just a few DAT files as a new xapres object, and see if it is functional with the old data structure, i.e. if I can properly append. Another option is to re-process the entire folder. Some roadblocks that I anticipate include the different attenuation settings used by the different deployment periods in addition to A104 facing recording issues in February .