search

CDCgov / cfa_azure

Apache License 2.0
9 stars 2 forks source link

change file uploads to keep folder structure in Blob container #85

Closed ryanraaschCDC closed 2 months ago

ryanraaschCDC commented 4 months ago

Currently, files being uploaded are flattened and placed at the root of the blob container. We will want to change this to keep the folder structure that is provided to the upload command.

arik-shurygin commented 3 months ago

Hey Ryan, I saw your development on the branch and it looks good. I was wondering if we can modify the scope of this issue slightly to allow uploads to be to a specific location rather than just top level of the storage blob.

To clarify what I mean lets imagine I have this direcotry structure data/x/file1.csv, data/y/file2.csv. if I want to upload the data folder I would obviously want the x and y folders to be preserved. But I would also like to be able to place data within a folder on the storage blob called job_1 so that I can have versions my csvs for each job I run. So in this example the paths on blob would look like. job_1/data/x/file1.csv and job_1/data/y/file2.csv. This current implementation does not allow me to prepend this job_1 folder without changing my local directory structure, which I would prefer not to do.

Is there a way you can add an argument for location which is prepended to whatever paths are uploaded, it can default to . which would keep the current functionality.

Let me know if this makes sense, appreciate all your hard work!

ryanraaschCDC commented 3 months ago

that should be fairly easy, I suspect! It should just be part of the os.path.join() if a location is provided. I'll get working on it.