search

enram / data-repository

Data quality assessment
https://enram.github.io/data-repository/
MIT License
3 stars 1 forks source link

Figure out how to flyway vp data #39

Closed peterdesmet closed 6 years ago

peterdesmet commented 6 years ago

I received the vp data for the flyway analysis from @CeciliaNilsson709 on a USB stick.

@stijnvanhoey, how easily can I run the file_transfer functionality over this to upload these data to the repository?

stijnvanhoey commented 6 years ago

If you have the proper rights to upload to the S3 bucket (aws_access_key_id and aws_secret_access_key in an aws-credentials file):

[lw-enram]
aws_access_key_id=xxxxxxxxxxxxxxxxxxxxx
aws_secret_access_key=xxxxxxxxxxxxxxxxxx

You can upload the code yourself from your local computer:

import datamover as dm
ltos = dm.LocalToS3(filepath="../example_data/", bucket_name="lw-enram", profile_name="lw-enram") # profile name links to the above section in your credentials file
ltos.transfer(name_match="_vp_", overwrite=False, limit=5, verbose=True) # low limit of files set for initial test, remove it when it works
peterdesmet commented 6 years ago

Thanks for the help! Upload is now running for a single radar. It's super slow though. 😴

stijnvanhoey commented 6 years ago

Uploading not optimized for speed as the daily load is rather small. So, no specific design of the code for large data uploads...