search

OpenScienceComputing / unige-pangeo-course

Apache License 2.0
2 stars 1 forks source link

How can I put data on OSN from my computer? #1

Open rsignell opened 5 months ago

rsignell commented 5 months ago

ping @andreall

LorenzoCorgnati commented 5 months ago

Thanks @rsignell and @andreall , I'll be in touch here.

rsignell commented 5 months ago

@LorenzoCorgnati, if you would like to put data from your local computer onto the OSN pod, copy the ~/.aws/credentials file from Nebari onto your local computer (and put it in ~/.aws/credentials .

Your local python environment should include these packages:

Then you should be able to move files (or directories) to OSN using fsspec using the pattern we learned in the class (the simple object storage writing example where we uploaded the air_temperature.nc file). You can use recursive=True to upload a directory.

rsignell commented 5 months ago

So an example code block (after you've added the osn-esip profile to your ~/.aws/credentials file) would be:

import fsspec

osn_endpoint_url = 'https://ncsa.osn.xsede.org'  

fs_write = fsspec.filesystem('s3', profile='osn-esip', skip_instance_cache=True, use_listings_cache=False, client_kwargs={'endpoint_url': osn_endpoint_url})

fs_write.cp('/my/local/data_folder', 'esip/your-user-name/your-data-folder', recursive=True)
mmagaldi77 commented 5 months ago

ok thanks this may be relevant also for issue #4