I am spike sorting data sets on our local cluster (which uses SLURM) with mountainlab-js, making use of the different processors ms4alg.sort, ms4alg.create_label_map, ms4alg.apply_label_map. I run them as a part of a snakemake pipeline. Snakemake is a workflow management system which allows me to run large parameter scans easily. Each rule in a snakemake workflow is submitted as an individual job to the queuing system on the cluster, and hence works independently.
Of late, I have been seeing these errors randomly when running the processors from the ms4alg package.
(node:7572) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'original_checksum' of undefined
at /mnt/beegfs/home/s.sridhar/scripts/pipelines/ml_pipeline/.snakemake/conda/1f2f2c8f/lib/node_modules/mountainlab/mlproc/prv_utils.js:202:24
at /mnt/beegfs/home/s.sridhar/scripts/pipelines/ml_pipeline/.snakemake/conda/1f2f2c8f/lib/node_modules/mountainlab/mlproc/prv_utils.js:192:7
(node:7572) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:7572) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:8167) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'original_checksum' of undefined
at /mnt/beegfs/home/s.sridhar/scripts/pipelines/ml_pipeline/.snakemake/conda/1f2f2c8f/lib/node_modules/mountainlab/mlproc/prv_utils.js:202:24
at /mnt/beegfs/home/s.sridhar/scripts/pipelines/ml_pipeline/.snakemake/conda/1f2f2c8f/lib/node_modules/mountainlab/mlproc/prv_utils.js:192:7
(node:8167) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:8167) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Since this is being run on the cluster, the corresponding output looks like this:
[ Getting processor spec... ]
[ Checking inputs and substituting prvs ... ]
[ Computing process signature ... ]
Process signature: cf7bd2ec46045c17c672c8bc4ddbf9075ea1d480
[ Checking outputs... ]
{"label_map_out":"/tmp/ml_create_label_map/i140703-001_50_6_35_0.5_0.92_0.2_0.5/ml_label_map_out.mda"}
Processing ouput - /tmp/ml_create_label_map/i140703-001_50_6_35_0.5_0.92_0.2_0.5/ml_label_map_out.mda
false
{"label_map_out":"/tmp/ml_create_label_map/i140703-001_50_6_35_0.5_0.92_0.2_0.5/ml_label_map_out.mda"}
[ Checking process cache ... ]
[ Creating temporary directory ... ]
[ Creating links to input files... ]
[ Preparing temporary outputs... ]
Processing ouput - /tmp/ml_create_label_map/i140703-001_50_6_35_0.5_0.92_0.2_0.5/ml_label_map_out.mda
false
[ Initializing process ... ]
[ Running ... ] /mnt/beegfs/home/s.sridhar/scripts/pipelines/ml_pipeline/.snakemake/conda/1f2f2c8f/bin/python3 /mnt/beegfs/home/s.sridhar/scripts/pipelines/ml_pipeline/.snakemake/conda/1f2f2c8f/etc/mountainlab/packages/ml_ms4alg/curation_spec.py.mp ms4alg.create_label_map --_tempdir=/tmp/mountainlab-tmp/tempdir_cf7bd2ec46_UfI18k --metrics=/tmp/mountainlab-tmp/tempdir_cf7bd2ec46_UfI18k/input_metrics_RMvnkPQS.json --label_map_out=/tmp/mountainlab-tmp/tempdir_cf7bd2ec46_UfI18k/output_label_map_out.mda --firing_rate_thresh=0.5 --isolation_thresh=0.92 --noise_overlap_thresh=0.2 --peak_snr_thresh=0.5
Elapsed time for processor ms4alg.create_label_map: 3.447 sec
Finalizing output label_map_out
[ Saving to process cache ... ]
[ Getting processor spec... ]
[ Checking inputs and substituting prvs ... ]
[ Computing process signature ... ]
The script that is originally run looks like this:
The slightly strange formatting is due to the wildcards system that snakemake follows. It fills in the wildcard entries automatically for different values that I request, and runs this script for each such such parameter set. This is an example of the create_label_map+apply_label_map step. A very similar script is also deployed for the sort step, and also yields the same error.
This error appears randomly, that is, if I run the script for the same configuration (same parameter set, for instance) again, it doesn't necessarily reappear. The conda environment that snakemake creates and uses has the following packages installed:
Any clue what might be happening? Anything else you need for debugging this?
Thanks!
PS I posted the same issue in the mountainlab-js repository, but I figured this was probably a better place to ask the question, sorry! If you think I close/remove the issue from mountainlab-js, please let me know!
Hello,
I am spike sorting data sets on our local cluster (which uses SLURM) with mountainlab-js, making use of the different processors
ms4alg.sort,ms4alg.create_label_map,ms4alg.apply_label_map. I run them as a part of a snakemake pipeline. Snakemake is a workflow management system which allows me to run large parameter scans easily. Each rule in a snakemake workflow is submitted as an individual job to the queuing system on the cluster, and hence works independently.Of late, I have been seeing these errors randomly when running the processors from the
ms4algpackage.Since this is being run on the cluster, the corresponding output looks like this:
The script that is originally run looks like this:
The slightly strange formatting is due to the wildcards system that snakemake follows. It fills in the wildcard entries automatically for different values that I request, and runs this script for each such such parameter set. This is an example of the create_label_map+apply_label_map step. A very similar script is also deployed for the sort step, and also yields the same error.
This error appears randomly, that is, if I run the script for the same configuration (same parameter set, for instance) again, it doesn't necessarily reappear. The conda environment that snakemake creates and uses has the following packages installed:
Any clue what might be happening? Anything else you need for debugging this?
Thanks!
PS I posted the same issue in the mountainlab-js repository, but I figured this was probably a better place to ask the question, sorry! If you think I close/remove the issue from mountainlab-js, please let me know!