Automatically sync aggragate datasets to AWS - Githubissues

int-brain-lab / iblalyx

MIT License

0 stars 0 forks source link

Automatically sync aggragate datasets to AWS #31

Open k1o0 opened 1 year ago

k1o0 commented 1 year ago

The current update_aws management command syncs datasets grouped by session and doesn't support aggregate datasets.

[ ] Create method in management command to sync base aggregates folder*
[ ] Create/modify S3 file records for aggregate datasets after sync; delete datasets with only file record on S3
[ ] Skip session level dataset sync when eid is null (here)

*aws s3 sync /mnt/ibl/aggregates s3://ibl-brain-wide-map-private/aggregates --profile ibladmin --exclude *.log --no-progress --dryrun

oliche commented 12 months ago

Filter datasets per empty sessions will give us the set of data to sync. Then need to go over full architecture to register datasets