ml-run-process ms4alg.sort, Warning: unable to rename file

[bl202]

Hi, I just installed mountainlab-js through conda and am trying out your algorithm but I keep getting the following error

RUNNING: ml-run-process ephys.bandpass_filter --inputs timeseries:/home/bl202/Documents/sample_data/raw.mda --parameters freq_max:6000 freq_min:300 samplerate:30000 --outputs timeseries_out:/home/bl202/Documents/sample_data/out/filt.mda.prv [ Getting processor spec... ] [ Checking inputs and substituting prvs ... ] [ Computing process signature ... ] [ Checking outputs... ] [ Checking process cache ... ] [ Process ephys.bandpass_filter already completed. ] [ Creating output prv for timeseries_out ... ] [ Done. ] RUNNING: ml-run-process ephys.whiten --inputs timeseries:/home/bl202/Documents/sample_data/out/filt.mda.prv --parameters --outputs timeseries_out:/home/bl202/Documents/sample_data/out/pre.mda.prv [ Getting processor spec... ] [ Checking inputs and substituting prvs ... ] Locating /home/bl202/Documents/sample_data/out/filt.mda.prv ... [ Computing process signature ... ] [ Checking outputs... ] [ Checking process cache ... ] [ Process ephys.whiten already completed. ] [ Creating output prv for timeseries_out ... ] [ Done. ] RUNNING: ml-run-process ms4alg.sort --inputs geom:/home/bl202/Documents/sample_data/geom.csv timeseries:/home/bl202/Documents/sample_data/out/pre.mda.prv --parameters adjacency_radius:100 detect_sign:-1 detect_threshold:-1 --outputs firings_out:output/firings.mda [ Getting processor spec... ] Warning: unable to rename file /home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda -> /home/bl202/Documents/output/firings.mda . Perhaps temporary directory is not on the same device as the output file directory. [ Checking inputs and substituting prvs ... ] Error renaming file /home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda -> /home/bl202/Documents/output/firings.mda: ENOENT: no such file or directory, copyfile '/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda' -> '/home/bl202/Documents/output/firings.mda' Locating /home/bl202/Documents/sample_data/out/pre.mda.prv ... [ Computing process signature ... ] [ Checking outputs... ] [ Checking process cache ... ] [ Creating temporary directory ... ] [ Preparing temporary outputs... ] [ Initializing process ... ] [ Running ... ] /home/bl202/anaconda3/envs/mlab/bin/python3 /home/bl202/anaconda3/envs/mlab/etc/mountainlab/packages/ml_ms4alg/ms4alg_spec.py.mp ms4alg.sort --geom=/home/bl202/Documents/sample_data/geom.csv --timeseries=/home/bl202/Documents/tmp-mlab/output_fc01c6308d08b94c019ddc06dc9ff6e6debf8494_timeseries_out.mda --firings_out=/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda --adjacency_radius=100 --detect_sign=-1 --detect_threshold=-1 --_tempdir=/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC Using tempdir=/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC Preparing /home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/timeseries.hdf5... Preparing neighborhood sorters...

Detecting events on channel 1 (phase1)... Detecting events on channel 2 (phase1)...

Detecting events on channel 4 (phase1)... Detecting events on channel 3 (phase1)...

Computing PCA features for channel 3 (phase1)...

Computing PCA features for channel 4 (phase1)...

Computing PCA features for channel 1 (phase1)...

Computing PCA features for channel 2 (phase1)...

Clustering for channel 3 (phase1)...

Clustering for channel 2 (phase1)...

Clustering for channel 1 (phase1)...

Clustering for channel 4 (phase1)...

Found 3 clusters for channel 1 (phase1)... Computing templates for channel 1 (phase1)...

Found 2 clusters for channel 4 (phase1)... Computing templates for channel 4 (phase1)...

Found 2 clusters for channel 3 (phase1)... Computing templates for channel 3 (phase1)...

Found 2 clusters for channel 2 (phase1)... Computing templates for channel 2 (phase1)...

Re-assigning events for channel 1 (phase1)...

Re-assigning events for channel 4 (phase1)...

Re-assigning events for channel 3 (phase1)...

Re-assigning events for channel 2 (phase1)...

Computing PCA features for channel 3 (phase2)...

Computing PCA features for channel 1 (phase2)...

Computing PCA features for channel 4 (phase2)...

Computing PCA features for channel 2 (phase2)...

Clustering for channel 3 (phase2)...

Clustering for channel 1 (phase2)...

Clustering for channel 4 (phase2)...

Clustering for channel 2 (phase2)...

Found 4 clusters for channel 3 (phase2)...

Found 6 clusters for channel 1 (phase2)...

Found 12 clusters for channel 4 (phase2)...

Found 9 clusters for channel 2 (phase2)...

Preparing output...

Writing firings file...

Done.

Elapsed time for processor ms4alg.sort: 99.194 sec Finalizing output firings_out [ Removing temporary directory ... ] Traceback (most recent call last):

**File "", line 1, in runfile('/home/bl202/Documents/temp.py', wdir='/home/bl202/Documents')

File "/home/bl202/anaconda3/envs/mlab/lib/python3.6/site-packages/spyder_kernels/customize/spydercustomize.py", line 678, in runfile execfile(filename, namespace)

File "/home/bl202/anaconda3/envs/mlab/lib/python3.6/site-packages/spyder_kernels/customize/spydercustomize.py", line 106, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "/home/bl202/Documents/temp.py", line 259, in detect_threshold=-1,

File "/home/bl202/Documents/temp.py", line 54, in sort_dataset opts=opts

File "/home/bl202/Documents/temp.py", line 125, in ms4alg_sort opts

File "/home/bl202/anaconda3/envs/mlab/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlproc_impl.py", line 235, in runProcess return P.run(inputs,outputs,parameters,opts)

File "/home/bl202/anaconda3/envs/mlab/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlproc_impl.py", line 189, in run raise Exception('Non-zero exit code for {}'.format(self.name()))

Exception: Non-zero exit code for ms4alg.sort**

---

As you can see, the temporary directory is in fact in the same device as the output file directory. Do you have any pointers on what I'm doing wrong?

Thanks in advance

[tjd2002]

This probably related to a recent bug fix. Whichever version of Mountainsort you are using (4.1.0 or 4.2.0), can you try the other one and report back if the behavior is the same? Something like: conda install -c flatiron 'mountainsort=4.2.0' (There should be single-quotes around Mountainsort=4.2.0)

Also show output of “conda list” Sent from my phone

On Jul 30, 2018, at 2:43 PM, bl202 notifications@github.com wrote:

Hi, I just installed mountainlab-js through conda and am trying out your algorithm but I keep getting the following error

RUNNING: ml-run-process ephys.bandpass_filter --inputs timeseries:/home/bl202/Documents/sample_data/raw.mda --parameters freq_max:6000 freq_min:300 samplerate:30000 --outputs timeseries_out:/home/bl202/Documents/sample_data/out/filt.mda.prv [ Getting processor spec... ] [ Checking inputs and substituting prvs ... ] [ Computing process signature ... ] [ Checking outputs... ] [ Checking process cache ... ] [ Process ephys.bandpass_filter already completed. ] [ Creating output prv for timeseries_out ... ] [ Done. ] RUNNING: ml-run-process ephys.whiten --inputs timeseries:/home/bl202/Documents/sample_data/out/filt.mda.prv --parameters --outputs timeseries_out:/home/bl202/Documents/sample_data/out/pre.mda.prv [ Getting processor spec... ] [ Checking inputs and substituting prvs ... ] Locating /home/bl202/Documents/sample_data/out/filt.mda.prv ... [ Computing process signature ... ] [ Checking outputs... ] [ Checking process cache ... ] [ Process ephys.whiten already completed. ] [ Creating output prv for timeseries_out ... ] [ Done. ] RUNNING: ml-run-process ms4alg.sort --inputs geom:/home/bl202/Documents/sample_data/geom.csv timeseries:/home/bl202/Documents/sample_data/out/pre.mda.prv --parameters adjacency_radius:100 detect_sign:-1 detect_threshold:-1 --outputs firings_out:output/firings.mda [ Getting processor spec... ] Warning: unable to rename file /home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda -> /home/bl202/Documents/output/firings.mda . Perhaps temporary directory is not on the same device as the output file directory. [ Checking inputs and substituting prvs ... ] Error renaming file /home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda -> /home/bl202/Documents/output/firings.mda: ENOENT: no such file or directory, copyfile '/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda' -> '/home/bl202/Documents/output/firings.mda' Locating /home/bl202/Documents/sample_data/out/pre.mda.prv ... [ Computing process signature ... ] [ Checking outputs... ] [ Checking process cache ... ] [ Creating temporary directory ... ] [ Preparing temporary outputs... ] [ Initializing process ... ] [ Running ... ] /home/bl202/anaconda3/envs/mlab/bin/python3 /home/bl202/anaconda3/envs/mlab/etc/mountainlab/packages/ml_ms4alg/ms4alg_spec.py.mp ms4alg.sort --geom=/home/bl202/Documents/sample_data/geom.csv --timeseries=/home/bl202/Documents/tmp-mlab/output_fc01c6308d08b94c019ddc06dc9ff6e6debf8494_timeseries_out.mda --firings_out=/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda --adjacency_radius=100 --detect_sign=-1 --detect_threshold=-1 --_tempdir=/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC Using tempdir=/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC Preparing /home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/timeseries.hdf5... Preparing neighborhood sorters...

Detecting events on channel 1 (phase1)... Detecting events on channel 2 (phase1)...

Detecting events on channel 4 (phase1)... Detecting events on channel 3 (phase1)...

Computing PCA features for channel 3 (phase1)...

Computing PCA features for channel 4 (phase1)...

Computing PCA features for channel 1 (phase1)...

Computing PCA features for channel 2 (phase1)...

Clustering for channel 3 (phase1)...

Clustering for channel 2 (phase1)...

Clustering for channel 1 (phase1)...

Clustering for channel 4 (phase1)...

Found 3 clusters for channel 1 (phase1)... Computing templates for channel 1 (phase1)...

Found 2 clusters for channel 4 (phase1)... Computing templates for channel 4 (phase1)...

Found 2 clusters for channel 3 (phase1)... Computing templates for channel 3 (phase1)...

Found 2 clusters for channel 2 (phase1)... Computing templates for channel 2 (phase1)...

Re-assigning events for channel 1 (phase1)...

Re-assigning events for channel 4 (phase1)...

Re-assigning events for channel 3 (phase1)...

Re-assigning events for channel 2 (phase1)...

Computing PCA features for channel 3 (phase2)...

Computing PCA features for channel 1 (phase2)...

Computing PCA features for channel 4 (phase2)...

Computing PCA features for channel 2 (phase2)...

Clustering for channel 3 (phase2)...

Clustering for channel 1 (phase2)...

Clustering for channel 4 (phase2)...

Clustering for channel 2 (phase2)...

Found 4 clusters for channel 3 (phase2)...

Found 6 clusters for channel 1 (phase2)...

Found 12 clusters for channel 4 (phase2)...

Found 9 clusters for channel 2 (phase2)...

Preparing output...

Writing firings file...

Done.

Elapsed time for processor ms4alg.sort: 99.194 sec Finalizing output firings_out [ Removing temporary directory ... ] Traceback (most recent call last):

**File "", line 1, in runfile('/home/bl202/Documents/temp.py', wdir='/home/bl202/Documents')

File "/home/bl202/anaconda3/envs/mlab/lib/python3.6/site-packages/spyder_kernels/customize/spydercustomize.py", line 678, in runfile execfile(filename, namespace)

File "/home/bl202/anaconda3/envs/mlab/lib/python3.6/site-packages/spyder_kernels/customize/spydercustomize.py", line 106, in execfile exec(compile(f.read(), filename, 'exec'), namespace)

File "/home/bl202/Documents/temp.py", line 259, in detect_threshold=-1,

File "/home/bl202/Documents/temp.py", line 54, in sort_dataset opts=opts

File "/home/bl202/Documents/temp.py", line 125, in ms4alg_sort opts

File "/home/bl202/anaconda3/envs/mlab/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlproc_impl.py", line 235, in runProcess return P.run(inputs,outputs,parameters,opts)

File "/home/bl202/anaconda3/envs/mlab/lib/python3.6/site-packages/mountainlab_pytools/mlproc/mlproc_impl.py", line 189, in run raise Exception('Non-zero exit code for {}'.format(self.name()))

Exception: Non-zero exit code for ms4alg.sort**

As you can see, the temporary directory is in fact in the same device as the output file directory. Do you have any pointers on what I'm doing wrong?

Thanks in advance

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[tjd2002]

The previous issue is #33

[bl202]

Thanks for the reply. Was using Mountainsort 4.1.0, just installed 4.2.0 and I get the same error. I'm attaching output of conda list.

conda_packs.log console_ms4.2.log

[magland]

Could you please try this with a different temporary directory? Try using one that is on the same device as your output files. (see ml-config)

[bl202]

Hi, I have tried other output directories (through defining the environment variable in mountainlab.env), and currently both the output and the tmp directories are in the same device:

/home/bl202/Documents/tmp-mlab/tempdir_d5c04fe186_mR0sjC/output_firings_out.mda /home/bl202/Documents/output/firings.mda

[tjd2002]

Hmm. Unfortunately I’m out this week so can’t push new conda versions. I’d still like to rule out that this bug has already been fixed.

Within your conda env can you run:

npm update -g mountainlab;
pip install mountainlab-pytools -U

After that you should have mountainlab 0.13 or higher (check with “npm list -g”—conda list won’t be aware of this update and will still show 0.10.5) and mountainlab_pytools version 0.3.0 or higher (I think “conda list” will pick this up)

(Mixing and matching conda and npm/pip like this is not ideal, but it should let you pull in the latest versions for now. After we get this bug sorted and have conda packages autogenerated with new versions, this won’t be necessary; you may need to blow away and re-create your conda env.)