Add s5cmd to service - Githubissues

AllenNeuralDynamics / aind-watchdog-service

Data staging service that prepares acquistion data for cloud upload to Amazon S3 and Code Ocean processing.

https://allenneuraldynamics.github.io/aind-watchdog-service/

MIT License

2 stars 1 forks source link

Add s5cmd to service #22

Open arielleleon opened 4 months ago

arielleleon commented 4 months ago

VAST has enabled S3 protocol for data transfers. Preliminary tests on the spims show this is faster than smb for transferring data over the network. s5cmd is a tool that can be installed on win and unix machines to transfer files using the s3 protocol

[ ] Test out on mesoscope
[ ] Add to run_job module

arielleleon commented 3 months ago

s5cmd is integrated into aind-watchdog-service and is on this branch

Currently working with VAST engineers to resolve issues with copy errors. Implemented retries within aind-watchdog-service but this has not resolved the issue.

Work is blocked