search

AllenNeuralDynamics / aind-ephys-pipeline

Code Ocean pipeline for ephys processing with Kilosort2.5
MIT License
5 stars 2 forks source link

Error in preprocessing #4

Closed bjhardcastle closed 7 months ago

bjhardcastle commented 8 months ago

Run 8927683:

N E X T F L O W  ~  version 22.10.0
Launching `main.nf` [comp-e682d535-3a15-40c4-ba3e-a5e6697] DSL1 - revision: c8d71b5fa9
[f4/1cad56] Submitted process > capsule_nwb_packaging_subject_capsule_10 (capsule-1748641)
[1f/0c319f] Submitted process > capsule_aind_ephys_job_dispatch_4 (capsule-5089190)
[capsule-5089190] cloning git repo...
[capsule-5089190] running capsule...
Running job dispatcher with the following parameters:
    CONCATENATE RECORDINGS: False
Session: ecephys_644866_2023-02-08_16-01-11
    Session path from data: ecephys_session - Open Ephys folder: ../data/ecephys_session/ecephys_clipped
    Num. Blocks 1 - Num. streams: 13
    Recording to be processed in parallel:
        experiment1_Record Node 103#Neuropix-PXI-100.ProbeA-AP_recording1 - Duration: 5058.38 s
        experiment1_Record Node 103#Neuropix-PXI-100.ProbeB-AP_recording1 - Duration: 5058.36 s
        experiment1_Record Node 103#Neuropix-PXI-100.ProbeC-AP_recording1 - Duration: 5058.43 s
        experiment1_Record Node 104#Neuropix-PXI-100.ProbeD-AP_recording1 - Duration: 5058.34 s
        experiment1_Record Node 104#Neuropix-PXI-100.ProbeE-AP_recording1 - Duration: 5058.36 s
        experiment1_Record Node 104#Neuropix-PXI-100.ProbeF-AP_recording1 - Duration: 5058.34 s
Generated 6 job config files
[capsule-5089190] completed!
[ce/45bbe9] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[30/a92734] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[87/c753f1] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[0a/802d88] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[65/cc9611] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[6e/974e7e] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[capsule-1748641] cloning git repo...
[capsule-1748641] running capsule...
Backend: zarr
Asset name: ecephys_644866_2023-02-08_16-01-11
Saved ../results/ecephys_644866_2023-02-08_16-01-11.nwb
[capsule-1748641] completed!
[capsule-0874799] cloning git repo...
[capsule-0874799] running capsule...
Running preprocessing with the following parameters:
    DENOISING_STRATEGY: cmr
    REMOVE_OUT_CHANNELS: True
    REMOVE_BAD_CHANNELS: True
    MAX BAD CHANNEL FRACTION: 0.5
    COMPUTE_MOTION: True
    APPLY_MOTION: False
    MOTION PRESET: None
    N_JOBS: 16
Found 1 json configurations

PREPROCESSING
Preprocessing recording: ecephys_644866_2023-02-08_16-01-11 - experiment1_Record Node 103#Neuropix-PXI-100.ProbeB-AP_recording1
    Duration: 5058.36 s
    Bad channel detection:
        - dead channels - 91
        - noise channels - 90
        - out channels - 101
    More than 50.0% bad channels (282). Skipping further processing for this recording.
PREPROCESSING time: 120.35s
[capsule-0874799] completed!
[12/6c6ad0] Submitted process > capsule_aind_ephys_spikesort_kilosort_25_7 (capsule-2633671)
Error executing process > 'capsule_aind_ephys_spikesort_kilosort_25_7 (capsule-2633671)'

Caused by:
  Missing output file(s) `capsule/results/*` expected by process `capsule_aind_ephys_spikesort_kilosort_25_7 (capsule-2633671)`

Command executed:

  #!/usr/bin/env bash
  set -e

  export CO_CAPSULE_ID=9c169ec3-5933-4b10-808b-6fa4620e37b7
  export CO_CPUS=16
  export CO_MEMORY=65498251264

  mkdir -p capsule
  mkdir -p capsule/data && ln -s $PWD/capsule/data /data
  mkdir -p capsule/results && ln -s $PWD/capsule/results /results
  mkdir -p capsule/scratch && ln -s $PWD/capsule/scratch /scratch

  echo "[capsule-2633671] cloning git repo..."
  git clone "https://$GIT_ACCESS_TOKEN@codeocean.allenneuraldynamics.org/capsule-2633671.git" capsule-repo
  git -C capsule-repo checkout f16003e1518abedc55fe78ef99a4b8e65d1c56c6 --quiet
  mv capsule-repo/code capsule/code
  rm -rf capsule-repo

  echo "[capsule-2633671] running capsule..."
  cd capsule/code
  chmod +x run
  ./run

  echo "[capsule-2633671] completed!"

Command exit status:
  0

Command output:
  [capsule-2633671] cloning git repo...
  [capsule-2633671] running capsule...

  SPIKE SORTING
  SPIKE SORTING time: 0.0s
  [capsule-2633671] completed!

Command error:
  Cloning into 'capsule-repo'...
  + python -u run_capsule.py

Work dir:
  s3://codeocean-s3batchbucket-16itpvq060udk/1f8f159a-7670-47a9-baf1-078905fc9c2e/12/6c6ad02df18dd440f70416c45f6e40

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

WARN: Killing running tasks (5)

Run 8797934:

N E X T F L O W  ~  version 22.10.0
Launching `main.nf` [comp-4e2f9cea-25ff-48e4-9ddb-aaa0d86] DSL1 - revision: c8d71b5fa9
[4b/cec9bb] Submitted process > capsule_nwb_packaging_subject_capsule_10 (capsule-1748641)
[d2/b93970] Submitted process > capsule_aind_ephys_job_dispatch_4 (capsule-5089190)
[capsule-5089190] cloning git repo...
[capsule-5089190] running capsule...
Running job dispatcher with the following parameters:
    CONCATENATE RECORDINGS: False
Session: ecephys_666986_2023-08-14_11-38-55
    Session path from data: ecephys_session - Open Ephys folder: ../data/ecephys_session/ecephys_clipped
    Num. Blocks 1 - Num. streams: 13
    Recording to be processed in parallel:
        experiment1_Record Node 102#Neuropix-PXI-100.ProbeA-AP_recording1 - Duration: 7441.63 s
        experiment1_Record Node 102#Neuropix-PXI-100.ProbeB-AP_recording1 - Duration: 7441.63 s
        experiment1_Record Node 102#Neuropix-PXI-100.ProbeC-AP_recording1 - Duration: 7441.68 s
        experiment1_Record Node 103#Neuropix-PXI-100.ProbeD-AP_recording1 - Duration: 7441.59 s
        experiment1_Record Node 103#Neuropix-PXI-100.ProbeE-AP_recording1 - Duration: 7441.62 s
        experiment1_Record Node 103#Neuropix-PXI-100.ProbeF-AP_recording1 - Duration: 7441.59 s
Generated 6 job config files
[capsule-5089190] completed!
[31/1a3065] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[ef/4ee0d0] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[63/b3c860] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[99/9d6714] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[e5/f4d2a3] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[19/83190f] Submitted process > capsule_aind_ephys_preprocessing_1 (capsule-0874799)
[capsule-1748641] cloning git repo...
[capsule-1748641] running capsule...
Backend: zarr
Asset name: ecephys_666986_2023-08-14_11-38-55
Saved ../results/ecephys_666986_2023-08-14_11-38-55.nwb
[capsule-1748641] completed!
[capsule-0874799] cloning git repo...
[capsule-0874799] running capsule...
Running preprocessing with the following parameters:
    DENOISING_STRATEGY: cmr
    REMOVE_OUT_CHANNELS: True
    REMOVE_BAD_CHANNELS: True
    MAX BAD CHANNEL FRACTION: 0.5
    COMPUTE_MOTION: True
    APPLY_MOTION: False
    MOTION PRESET: None
    N_JOBS: 16
Found 1 json configurations

PREPROCESSING
Preprocessing recording: ecephys_666986_2023-08-14_11-38-55 - experiment1_Record Node 102#Neuropix-PXI-100.ProbeB-AP_recording1
    Duration: 7441.63 s
    Bad channel detection:
        - dead channels - 88
        - noise channels - 90
        - out channels - 101
    More than 50.0% bad channels (279). Skipping further processing for this recording.
PREPROCESSING time: 114.92s
[capsule-0874799] completed!
[b2/5b3f6d] Submitted process > capsule_aind_ephys_spikesort_kilosort_25_7 (capsule-2633671)
Error executing process > 'capsule_aind_ephys_spikesort_kilosort_25_7 (capsule-2633671)'

Caused by:
  Missing output file(s) `capsule/results/*` expected by process `capsule_aind_ephys_spikesort_kilosort_25_7 (capsule-2633671)`

Command executed:

  #!/usr/bin/env bash
  set -e

  export CO_CAPSULE_ID=9c169ec3-5933-4b10-808b-6fa4620e37b7
  export CO_CPUS=16
  export CO_MEMORY=65498251264

  mkdir -p capsule
  mkdir -p capsule/data && ln -s $PWD/capsule/data /data
  mkdir -p capsule/results && ln -s $PWD/capsule/results /results
  mkdir -p capsule/scratch && ln -s $PWD/capsule/scratch /scratch

  echo "[capsule-2633671] cloning git repo..."
  git clone "https://$GIT_ACCESS_TOKEN@codeocean.allenneuraldynamics.org/capsule-2633671.git" capsule-repo
  git -C capsule-repo checkout f16003e1518abedc55fe78ef99a4b8e65d1c56c6 --quiet
  mv capsule-repo/code capsule/code
  rm -rf capsule-repo

  echo "[capsule-2633671] running capsule..."
  cd capsule/code
  chmod +x run
  ./run

  echo "[capsule-2633671] completed!"

Command exit status:
  0

Command output:
  [capsule-2633671] cloning git repo...
  [capsule-2633671] running capsule...

  SPIKE SORTING
  SPIKE SORTING time: 0.0s
  [capsule-2633671] completed!

Command error:
  Cloning into 'capsule-repo'...
  + python -u run_capsule.py

Work dir:
  s3://codeocean-s3batchbucket-16itpvq060udk/1f8f159a-7670-47a9-baf1-078905fc9c2e/b2/5b3f6dd34080318948f33e4b5e05ed

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

WARN: Killing running tasks (5)
alejoe91 commented 8 months ago

Hi @bjhardcastle

This is due to the fact that there are too many bad channels and the preprocessing is not producing any output files. Nextflow doesn't like this.

I'll fix it soon by creating a mock output file in case this happens :)

alejoe91 commented 7 months ago

This should be fixed with these commits to the preprocessing + sorting capsule:

Basically, the problem was that each step of the pipeline is expected to produce an output. In this case, with too many bad channels, some dummy outputs are now produced (and will be ignored later). Testing the failing job now after the fix

alejoe91 commented 7 months ago

@bjhardcastle closing this because it's been taken care of :)