search

AllenNeuralDynamics / aind-ephys-pipeline

Code Ocean pipeline for ephys processing with Kilosort2.5
MIT License
5 stars 2 forks source link

Download error #9

Closed bjhardcastle closed 6 months ago

bjhardcastle commented 7 months ago

Run With Parameters 1173866

Failed after 21hr 49min.

end of output 

...
VISUALIZATION time: 399.37s
[capsule-6668112] completed!
[d8/304319] Submitted process > capsule_aind_ephys_results_collector_9 (capsule-4820071)
[d8/304319] NOTE: Process `capsule_aind_ephys_results_collector_9 (capsule-4820071)` failed -- Execution is retried (1)
[b8/3be44b] Re-submitted process > capsule_aind_ephys_results_collector_9 (capsule-4820071)
[b8/3be44b] NOTE: Process `capsule_aind_ephys_results_collector_9 (capsule-4820071)` failed -- Execution is retried (2)
[51/e75b70] Re-submitted process > capsule_aind_ephys_results_collector_9 (capsule-4820071)
[51/e75b70] NOTE: Process `capsule_aind_ephys_results_collector_9 (capsule-4820071)` failed -- Execution is retried (3)
[51/2c0a69] Re-submitted process > capsule_aind_ephys_results_collector_9 (capsule-4820071)
Error executing process > 'capsule_aind_ephys_results_collector_9 (capsule-4820071)'
...
Command error:
  download failed: s3://codeocean-s3batchbucket-16itpvq060udk/1f8f159a-7670-47a9-baf1-078905fc9c2e/56/8d8b2925d41a737781ef34ba2682b2/capsule/results/preprocessed_experiment1_Record Node 104#Neuropix-PXI-100.ProbeF-AP_recording1/traces_cached_seg0.raw to capsule/data/preprocessed_experiment1_Record Node 104#Neuropix-PXI-100.ProbeF-AP_recording1/traces_cached_seg0.raw [Errno 12] Cannot allocate memory
  fatal error: [Errno 2] No such file or directory
  download failed: s3://codeocean-s3batchbucket-16itpvq060udk/1f8f159a-7670-47a9-baf1-078905fc9c2e/a7/75832e232cde18a429752ef1c4dc50/capsule/results/preprocessed_experiment1_Record Node 103#Neuropix-PXI-100.ProbeC-AP_recording1/traces_cached_seg0.raw to capsule/data/preprocessed_experiment1_Record Node 103#Neuropix-PXI-100.ProbeC-AP_recording1/traces_cached_seg0.raw [Errno 2] No such file or directory: '/tmp/nxf.5QTe25X22q/capsule/data/preprocessed_experiment1_Record Node 103#Neuropix-PXI-100.ProbeC-AP_recording1/traces_cached_seg0.raw.167BBCA8' -> '/tmp/nxf.5QTe25X22q/capsule/data/preprocessed_experiment1_Record Node 103#Neuropix-PXI-100.ProbeC-AP_recording1/traces_cached_seg0.raw'

Work dir:
  s3://codeocean-s3batchbucket-16itpvq060udk/1f8f159a-7670-47a9-baf1-078905fc9c2e/51/2c0a6914fe3d73a0fb024ff1a1bf64
alejoe91 commented 6 months ago

@bjhardcastle have you encountered this again? If needed, we can bump up the specs for the cisualization capsule

bjhardcastle commented 6 months ago

It has continued to pop up for a while. I haven't noticed it in the last 5 days, but I've been running far fewer jobs.