Populse db 2 is now working and passing tests

[denisri]

(it has been long and painful). Travis-ci still segfaults randomly, and seems to hang in some builds, but I cannot restart them (I probably don't have permissions to to it). It needs (of course) populse_db branch "2.0", which forbids for now install via pip.

[servoz]

Hi, thank you very much for the hard work (the last 3 PR!). As you've done everything, I have to take some time to understand a bit and do some tests before merging ... I did some tests last week and I observed some things that didn't work yet (but it was with the capsulengine branch). This might be a good time to fix the last little things. So as I understand it, this PR will merge the populse_db_2 branch of populse_mia with the master branch and it needs the 2.0 branch of populse_db to work properly. Concerning Travis-ci, it's very boring, I restart the jobs and sometimes it works!

So I do some tests and I'll be back!

[servoz]

Ok all the Travis-ci are now ok. it's very annyoning . I am not sure but this random segfaults could comes from the virtual environment in Travis. Indeed the segfaults is always coming with this kind of message: .qt.qpa.xcb: QXcbConnection: XCB error: 8 (BadMatch), sequence: 1732, resource id: 2097212, major code: 42 (SetInputFocus), minor code: 0 We nether observed this problem on physical machine or with appveyor.

[servoz]

I've started the tests. Sorry I'm not quick because I'm working on other cases at the same time (our trainee for two months has started his internship). So i use this config:

• populse_mia in populse_db_2 branch (populse_db in 2.0 branch ; all other populse packages on the master branch; No populse packages installed on the station, use in dev mode).
  • first thing i see when launching mia. In Pipeline Manager, i see the init, run, stop and status icons. Does that mean the capsulengine branch was merged into the populse_db_2 branch? I thought those button only existed in the capsulengine branch? (ah no it can't be the merge from capsulengine because it's still the ControlerView of mia and not capsul on the right in the pipeline manager ... !)

[servoz]

I'm going to make all the remarks I observe during the test. Some of them will be minor (can be left for later, others major (rather to be settled before merging the PR!).

• in pipeline manager the little triangles hiding or showing the bricks inside seem to have a different behavior than before (if i am not wrong, before it let see all the bricks by default, now it looks like it's the opposite. Do you have any idea why? (minor remark)

[servoz]

• old format database make a crash

  Traceback (most recent call last):
  File "/data/Git_Projects/populse_mia/python/populse_mia/user_interface/main_window.py", line 600, in open_recent_project
  self.switch_project(relative_path, name)
  File "/data/Git_Projects/populse_mia/python/populse_mia/user_interface/main_window.py", line 1007, in switch_project
  self.update_project(file_path)
  File "/data/Git_Projects/populse_mia/python/populse_mia/user_interface/main_window.py", line 1161, in update_project
  self.data_browser.table_data.update_table()  # Table updated
  File "/data/Git_Projects/populse_mia/python/populse_mia/user_interface/data_browser/data_browser.py", line 2284, in update_table
  self.fill_headers(take_tags_to_update)
  File "/data/Git_Projects/populse_mia/python/populse_mia/user_interface/data_browser/data_browser.py", line 1509, in fill_headers
  tags.remove(TAG_CHECKSUM)
  ValueError: list.remove(x): x not in list

  this is not a big problem, because we can assume that nobody have already started a serious calculation with populse_mia and we just have to create a new database in populse_db v2.0 format (very minor remark)

[servoz]

• run step is crashing (major remark) When the pipeline was launched, the following exception was raised:

  
  Error during workflow execution. Status=('finished_regularly', 1, None, None).
  The workflow has not been removed from soma_workflow and must be deleted manually. 
  Failed jobs: [24]
  Aborted/killed jobs: []
  ============================================
  ---- failed job info ---
  * job: 24: smooth1
  * exit status: finished_regularly
  * commandline:
  'python3' '-c' 'from capsul.api import Process; Process.run_from_commandline("mia_processes.preprocess.spm.spatial_preprocessing.Smooth")'
  * exit value: 1
  * term signal: None
  ---- env ----
  None
  ---- configuration ----
  {'capsul_engine': {'uses': {'capsul.engine.module.spm': 'any', 'capsul.engine.module.matlab': 'any'}}, 'capsul.engine.module.spm': {'config_id': 'spm12', 'config_environment': 'global', 'directory': '/usr/local/SPM/spm12', 'version': '12', 'standalone': False}, 'capsul.engine.module.matlab': {'config_id': 'matlab', 'config_environment': 'global', 'executable': '/usr/local/MATLAB/R2019b/bin/matlab'}}
  ---- stdout ----

---- stderr ---- Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'capsul' ...

If i remember well this is what i observed with the capsulengine branch last week, so i guess it is not exclusively linked with this PR.
We should pass the sys.path to the command` 'python3' '-c' 'from capsul.api import Process; Process.run_from_commandline("mia_processes.preprocess.spm.spatial_preprocessing.Smooth")' `

(if you remember, after a meeting I made a modification in the main.py to add in the sys.path the packages in development instead of using the ones in the site-package. So I removed everything from the site-package on my station to avoid starting with one package and continuing with another when running mia ...).

[servoz]

I just done a pull request in capsul to fix this problem of loosing the sys.path for dev mode in populse_mia. BUT, now this is the next crash at run time:

data/Git_Projects/soma-workflow/python/soma_workflow/client_types.py:307: UserWarning: import sys; sys.path = ['/data/Git_Projects/populse_mia/python', '/data/Git_Projects/capsul', '/data/Git_Projects/soma-base/python', '/data/Git_Projects/soma-workflow/python', '/data/Git_Projects/populse_db/python', '/data/Git_Projects/mia_processes/python', '/data/Git_Projects/populse_mia/python/populse_mia', '/usr/lib64/python37.zip', '/usr/lib64/python3.7', '/usr/lib64/python3.7/lib-dynload', '/home/econdami/.local/lib/python3.7/site-packages', '/home/econdami/.local/lib/python3.7/site-packages/scikit_image-0.15.0-py3.7-linux-x86_64.egg', '/home/econdami/.local/lib/python3.7/site-packages/etelemetry-0.2.0-py3.7.egg', '/home/econdami/.local/lib/python3.7/site-packages/ci_info-0.1.1-py3.7.egg', '/home/econdami/.local/lib/python3.7/site-packages/six-1.14.0-py3.7.egg', '/home/econdami/.local/lib/python3.7/site-packages/rdflib-5.0.0-py3.7.egg', '/home/econdami/.local/lib/python3.7/site-packages/sip-5.0.1-py3.7-linux-x86_64.egg', '/home/econdami/.local/lib/python3.7/site-packages/nipype-1.6.0.dev0+ge1ea1980c-py3.7.egg', '/home/econdami/.local/lib/python3.7/site-packages/csv2report-0.1.0-py3.7.egg', '/usr/lib64/python3.7/site-packages', '/usr/lib/python3.7/site-packages', '/data/Git_Projects/populse_mia/processes']; from capsul.api import Process; Process.run_from_commandline("mia_processes.preprocess.spm.spatial_preprocessing.Smooth") contains single quote. It could fail using DRMAA
  % command_elem, UserWarning)
soma-workflow starting in light mode
Workflow controller initialised

 When the pipeline was launched, the following exception was raised: Error during workflow execution. Status=('finished_regularly', 1, None, None).
The workflow has not been removed from soma_workflow and must be deleted manually. 
Failed jobs: [42]
Aborted/killed jobs: []
============================================
---- failed job info ---
* job: 42: smooth1
* exit status: finished_regularly
* commandline:
'python3' '-c' 'import sys; sys.path = [\'/data/Git_Projects/populse_mia/python\', \'/data/Git_Projects/capsul\', \'/data/Git_Projects/soma-base/python\', \'/data/Git_Projects/soma-workflow/python\', \'/data/Git_Projects/populse_db/python\', \'/data/Git_Projects/mia_processes/python\', \'/data/Git_Projects/populse_mia/python/populse_mia\', \'/usr/lib64/python37.zip\', \'/usr/lib64/python3.7\', \'/usr/lib64/python3.7/lib-dynload\', \'/home/econdami/.local/lib/python3.7/site-packages\', \'/home/econdami/.local/lib/python3.7/site-packages/scikit_image-0.15.0-py3.7-linux-x86_64.egg\', \'/home/econdami/.local/lib/python3.7/site-packages/etelemetry-0.2.0-py3.7.egg\', \'/home/econdami/.local/lib/python3.7/site-packages/ci_info-0.1.1-py3.7.egg\', \'/home/econdami/.local/lib/python3.7/site-packages/six-1.14.0-py3.7.egg\', \'/home/econdami/.local/lib/python3.7/site-packages/rdflib-5.0.0-py3.7.egg\', \'/home/econdami/.local/lib/python3.7/site-packages/sip-5.0.1-py3.7-linux-x86_64.egg\', \'/home/econdami/.local/lib/python3.7/site-packages/nipype-1.6.0.dev0+ge1ea1980c-py3.7.egg\', \'/home/econdami/.local/lib/python3.7/site-packages/csv2report-0.1.0-py3.7.egg\', \'/usr/lib64/python3.7/site-packages\', \'/usr/lib/python3.7/site-packages\', \'/data/Git_Projects/populse_mia/processes\']; from capsul.api import Process; Process.run_from_commandline("mia_processes.preprocess.spm.spatial_preprocessing.Smooth")'
* exit value: 1
* term signal: None
---- env ----
None
---- configuration ----
{'capsul_engine': {'uses': {'capsul.engine.module.spm': 'any', 'capsul.engine.module.matlab': 'any'}}, 'capsul.engine.module.spm': {'config_id': 'spm12', 'config_environment': 'global', 'directory': '/usr/local/SPM/spm12', 'version': '12', 'standalone': False}, 'capsul.engine.module.matlab': {'config_id': 'matlab', 'config_environment': 'global', 'executable': '/usr/local/MATLAB/R2019b/bin/matlab'}}
---- stdout ----

---- stderr ----
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/Git_Projects/capsul/capsul/process/process.py", line 732, in run_from_commandline
    result = ce.study_config.run(process, configuration_dict=configuration)
  File "/data/Git_Projects/capsul/capsul/study_config/study_config.py", line 417, in run
    configuration_dict=configuration_dict)
  File "/data/Git_Projects/capsul/capsul/study_config/run.py", line 84, in run_process
    engine.activate_configuration(configuration_dict)
  File "/data/Git_Projects/capsul/capsul/engine/__init__.py", line 558, in activate_configuration
    activate_module(m)
  File "/data/Git_Projects/capsul/capsul/engine/__init__.py", line 580, in activate_module
    raise EnvironmentError(error)
OSError: MATLAB_EXECUTABLE is not defined

BUT if I remember well, we reached an agreement not to force the user to use environment variables?

[denisri]

I'll open new issues to discuss the different points/questions raised here, because the PR interface does not allow to answer individual comments and there are several topics here. Globally, yes, there are 3 PRs (#165, #166 and #167). #165 and #166 are independent, both starting from the master branch, so should not be both merged actually. #167 includes (merges) both #165 and #166, and adds some modifs to efficiently use populse_db2 with the new capsulengine system. So merging only #167 would be enough, but I propose the 3 branches if you wish to test things individually - I'll remove the other ones once everything is merged.

165 and #167 thus need populse_db with its 2.0 branch, yes.

[denisri]

@servoz said:

first thing i see when launching mia. In Pipeline Manager, i see the init, run, stop and status icons. Does that mean the capsulengine branch was merged into the populse_db_2 branch? I thought those button only existed in the capsulengine branch? (ah no it can't be the merge from capsulengine because it's still the ControlerView of mia and not capsul on the right in the pipeline manager ... !)

The buttons were added in the master branch if I remember. But in master processing takes place in a thread (as it was before) whereas in capsulengine it is done in soma-workflow (separate job processes and distributed calculation). Before the buttons were here, the thread was running and the GUI was just waiting for this thread to end while displaying a busy progressbar. No user interaction was possible. The buttons just added this stop/status possibility in the GUI thread, without changing the way pipelines were run.