[initialisation step] Integration of the initialisation as the first step of the Run pipeline - Temprorary patch waiting for #249 fix

[servoz]

The database indexing issue is a broad topic that has already been discussed extensively in several now closed or still open ticket.

It is a fundamental operation in Mia and has many impacts on various essential elements of Mia, depending on how and when it is done.

There is currently a consensus that the best solution would be to perform this indexing at runtime with the use of a temporary database at initialisation time (exactly as described in the first post of the #249 ticket).

However, it seems that the work involved goes far beyond the scope of Mia (it would also involve soma_workflow, capsul?) and given the few forces we have at our disposal at the moment, it seems wiser to adopt a temporary solution, which is lighter to implement and will solve some of the problems encountered with the current operation:

• The initialisation is rather done at runtime (the Run pipeline integrates the initialisation as a first step). We comment on the Initialize pipeline buttons. There is only one Run pipeline buttons and if the initialisation fails, the database is not indexed (as it is now).
• If the Run pipeline fails, we delete the indexed data (question: everything or just what was not done?).
  • It might be interesting to keep the data already obtained before the Run pipeline crashes. There is a meticulous management of the indexing of the database and the real existence of the data to do (as long as we don't have the indexing of the database done properly at the time of the run and after each pass through a successful brick (as in #249 ticket), we will live with an indexing in the database coming from the future...). If the Run pipeline crashes, we should check what happens if we restart the run (which should be the wish of a user if the pipeline is not computed to the end) with the already existing data (in the database and on the hard disk) before to code in this direction (and beyond that we have to answer the previous question: everything or just what was not done?).

In short, this is a lighter solution than fixing the #249 ticket, but it is not completely trivial either ....

[LStruber]

Closed with commit f1e29bfd7a233aa9d4756c599e2dbb2b04a03d86. To sum up:

• Initialization button has been removed (commented), and Start button is now always clickable,
• At the beginning of run, initialization step is performed,
• If the Run pipeline fails (tested by manually stopping a running pipeline), data that have not been created yet on hard drive is removed (using cleanup_orphan_bricks() in postprocess_pipeline_execution() function in pipeline_manager_tab.py).
• If we restart the run, with already existing data in the database and on hard disk, the whole pipeline is re-run, overwriting previous saved data.

Note that all addition and comments have been surrounded by "# Commented on January, 4th 2020" and "# End - commented on January, 4th 2020" to easily allow putting the button back in place