search

AllenNeuralDynamics / aind-smartspim-stitch

Stitching and fusion pipeline in the cloud
MIT License
3 stars 1 forks source link

Data Validation #13

Closed miketaormina closed 1 year ago

miketaormina commented 1 year ago

Since we've already seen an instance of an incomplete VAST->AWS transfer cause a failure, I was wondering if we should create a module for data validation to be run either prior to de-striping (as part of a capsule) or prior to capsule-ization.

The simplest case, which I have been checking for the LOCAL->VAST transfer before local deletion, is to make sure that all tile directories have the same number of files (which should be the case for a SmartSPIM acquisition, unless an operator deactivates a tile).

@camilolaiton do you think this should be part of another repo, like the aind_data_transfer one for example?

miketaormina commented 1 year ago

I was going to label this as a question, but either GitHub won't let me or I'm too dense to figure it out

dyf commented 1 year ago

@miketaormina this should live in aind-data-transfer, I think.

camilolaiton commented 1 year ago

I agree with David, this should live in the aind-data-transfer repo. In fact, we noticed this issue some days ago and we're currently looking into it from there. Moving this issue to the aind-data-transfer repo.

Edit: Please, refer to this issue that I created in the aind-data-transfer repo.