Multiple Calls to MLC.addProcess in for loop -> error

[alexmorley]

e.g.

async function convert_firings(MLC, dataset_dirname, output_dirname, params, temp_dir) {
  for (tet = 1; tet < params.tetrodes.length+1; tet++) {
    MLC.addProcess({
      processor_name: 'dd.convert_firings',
      inputs: {
        firings: output_dirname + '/firings.mda',
        params: temp_dir + 'params.json'
      },
      outputs: {
        res_fname: output_dirname + '/res.' + tet,
        clu_fname: output_dirname + '/clu.' + tet
      },
      parameters: {
        tetrode:tet
      }
    })
  };
  await MLC.run();
}

Errors like

[...]
Error in find: Unexpected token { in JSON at position 10746 
[...]

I think its a race condition. Maybe two processes trying to write to the database at the same time? Or is this not the intended use?

Is there currently a lock on the database?

[magland]

I found the same error. I didn't realize that diskdb does not work with multiple processes simultaneously accessing the same file. Working on a solution now.

[alexmorley]

Yeah, you'd have thought that'd be like no. 1 requirement for any database...

Don't know what progress you've made yet but the easiest fix I can think of is for a process to:

1. check for a db.lock file, if its present then wait
2. write a db.lock file
3. then write to the actual db
4. then remove db.lock.

[magland]

Thanks Alex. After exploring various solutions, looking at the source code for diskdb (and having some painful experiences)... I decided to roll my own (which does use a locking mechanism similar to what you suggest).

This needs to be tested!

I've just pushed the changes, and you can see the source code here: https://github.com/flatironinstitute/mountainlab-js/blob/master/mlproc/safedatabase.js

[magland]

This passed, which is good news: https://travis-ci.org/flatironinstitute/mountainlab-js-tests

[tjd2002]

I don’t think it would be too painful to go back to Mongo (user-local installs are easy to spin up) if this turns out to be a bigger pain point.

Sent from my phone

On Jul 25, 2018, at 7:56 AM, Jeremy Magland notifications@github.com wrote:

This passed, which is good news: https://travis-ci.org/flatironinstitute/mountainlab-js-tests

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.