Preprocessing crashing not recoverable

[catubc]

This issue has to do with being able to recover (i.e. continue where left off) during preprocessing stages. It shouldn't be diffcult to fix, it involves mostly returning the right path/param variables to the preprocessing.run pipeline when a particular step is being skipped (because it was already completed previously).

For example, post filtering, I had to restart yass and on restart the "path" variable returned by the butterworth() call in process.run() was an object rather than a string. So the next function (standardize) crashed because of this. Also see issue #178 re: propagation of skip_file flags. But after fixing this I'm getting the same error for whitening step, and after fixing that path I get an linalg.SVD error see the second output below.

def process.run()
...
 path = os.path.join(CONFIG.data.root_folder, CONFIG.data.recordings)
...
    # optionally filter the data
    if CONFIG.preprocess.filter:
        path, params = butterworth(path,
                                   params['dtype'],
                                   params['n_channels'],
                                   params['data_format'],
                                   CONFIG.filter.low_pass_freq,
                                   CONFIG.filter.high_factor,
                                   CONFIG.filter.order,
                                   CONFIG.recordings.sampling_rate,
                                   CONFIG.resources.max_memory,
                                   TMP,
                                   OUTPUT_DTYPE,
                                   if_file_exists=if_file_exists)

    path = os.path.join(CONFIG.data.root_folder, CONFIG.data.recordings)  #Cat: redefined the 
path string otherwise the path returned from above on file_skip is an object...

    # standarize
    (standarized_path,
        standarized_params) = standarize(path,
                                         params['dtype'],
                                         params['n_channels'],
                                         params['data_format'],
                                         CONFIG.recordings.sampling_rate,
                                         CONFIG.resources.max_memory,
                                         TMP,
                                         OUTPUT_DTYPE,
                                         if_file_exists=if_file_exists)

Here's the error after trying to fix the whiten path:

File "/usr/local/lib/python2.7/dist-packages/yass_algorithm-0.7.dev0-py2.7.egg/yass/preprocess/whiten.py", line 88, in matrix whiten_filter = _matrix(first_batch, channel_index, spike_size) File "/usr/local/lib/python2.7/dist-packages/yass_algorithm-0.7.dev0-py2.7.egg/yass/preprocess/whiten.py", line 164, in matrix V, D, = np.linalg.svd(M[ch_idx, :][:, ch_idx]) File "/usr/local/lib/python2.7/dist-packages/numpy/linalg/linalg.py", line 1444, in svd u, s, vh = gufunc(a, signature=signature, extobj=extobj) File "/usr/local/lib/python2.7/dist-packages/numpy/linalg/linalg.py", line 98, in _raise_linalgerror_svd_nonconvergence raise LinAlgError("SVD did not converge") numpy.linalg.linalg.LinAlgError: SVD did not converge

[catubc]

Edit: If crash occurs between filtering and standardization, standardize seems to run over space rather than time. It seems there is a transpose or C/F order flag that is not reloaded at the standardization step.

[edublancas]

This should be fixed since I fixed #178, can you try?

[catubc]

Hopefully it's fixed, let's close and reopen if still issue.