PIPELINE: It's the end of HoloPipelines as we know it

[nielsboecker]

This PR reflects the gigantic #45 refactoring that I did to the HoloPipelines. I did a whole bunch of things and nothing is like it used to be. Hallelujah!

Some major points:

• complete re-architecture of all components
• implement several features that were kinda half-baked before
• fix numerous issues and problems
• implement dynamic pipeline loading
• do some basic error handling
• use multiprocessing instead of subprocess
• job state in-memory instead of POSTing to self
• job-specific data kept in job's own directory
• automatic garbage collection
• add logging to file per job and serve that at a new endpoint
• add type hints and docstrings
• add documentation
• dockerise application
• add Postman collection
• identify further problems and inconsistencies, for which I left TODOs or opened issues

Disclaimer 1: There are still many TODOs in the code. The test coverage is now zero. This must be worked on within following issues and PRs.

Disclaimer 2: I have mostly worked with the bone_segmentation pipeline and can confirm that it works end-too-end. The other ones @ansonwong9695 is still working at confirming nothing broke.

[boonwj]

holopipelines-core_1 | /usr/local/lib/python3.7/site-packages/dotenv/main.py:52: UserWarning: File doesn't exist holopipelines-core_1 | warnings.warn("File doesn't exist {}".format(self.dotenv_path)) # type: ignore

This warning will be thrown in docker-compose. There are no .env files in docker compose as the env variables are passed in straight during docker's run. We might want to document this in the README somewhere.

[boonwj]

Noticed that pipelines and models subprocesses are timing out with exceptions. This happened when I left the images up during testing. Is this expected?

I am not sure if this actually causes any issues with the actual workflow.

[nielsboecker]

I didn't really encounter this as I mostly developed local (when I start the Docker daemon I can't do anything else anymore due to my PC being an old man).

Looks scary though. Could it be related to this or this?

[boonwj]

holopipelines-core_1                             | 2019-08-27 13:52:16 | root | INFO  | Success! Created hologram: {'hid': 'ef9ce292-7312-4d02-9243-46486105bc79', 'title': 'asdf', 'contentType': 'model/gltf-binary', 'fileSizeInKb': 2011, 'creationDate': '2019-08-27T13:52:04Z', 'creationMode': 'GENERATE_FROM_IMAGING_STUDY', 'creationDescription': 'Generated by HoloPipelines with the abdominal_organs_segmentation pipeline', 'aid': 'a102', 'pid': 'p100'}
holopipelines-core_1                             | 2019-08-27 13:52:16 | 450c08a31d1b4e3b | INFO  | [450c08a31d1b4e3b] Finished state DISPATCHING_OUTPUT in 11.991149663925171 seconds
holopipelines-core_1                             | 2019-08-27 13:52:16 | 450c08a31d1b4e3b | INFO  | [450c08a31d1b4e3b] Entering state => FINISHED
holopipelines-core_1                             | Exception in thread Thread-3:
holopipelines-core_1                             | Traceback (most recent call last):
holopipelines-core_1                             |   File "/usr/local/lib/python3.7/threading.py", line 926, in _bootstrap_inner
holopipelines-core_1                             |     self.run()
holopipelines-core_1                             |   File "/usr/local/lib/python3.7/threading.py", line 870, in run
holopipelines-core_1                             |     self._target(*self._args, **self._kwargs)
holopipelines-core_1                             |   File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 486, in _handle_results
holopipelines-core_1                             |     cache[job]._set(i, obj)
holopipelines-core_1                             |   File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 662, in _set
holopipelines-core_1                             |     self._callback(self._value)
holopipelines-core_1                             | TypeError: job_success_callback() takes 0 positional arguments but 1 was given
holopipelines-core_1                             | 
holopipelines-core_1                             | 2019-08-27 13:52:29 | root | INFO  | Global state | 1 jobs active: ['450c08a31d1b4e3b']
holopipelines-core_1                             | 2019-08-27 13:52:29 | root | INFO  | Garbage collection | Removing job '450c08a31d1b4e3b' (success=True)
holopipelines-core_1                             | 2019-08-27 13:52:29 | root | INFO  | Removing temporary files for job '450c08a31d1b4e3b
holopipelines-core_1                             | 2019-08-27 13:52:29 | root | INFO  | Moving './__jobs__/450c08a31d1b4e3b' to './__finished_jobs__/450c08a31d1b4e3b'

There's this one error that occurs at the end of the abdominal pipeline. Will test it with the 2 other pipelines too.

[nielsboecker]

Hey @boonwj that is because the abdominal_pipeline was the only one to (unreasonably, because unused) return something at the end of its run. This will confuse the success callback handler in jobs_controller. I analysed this issue when you mentioned it yesterday; the only thing confusing me is that the state in the major-refactoring branch (here) already has the return statement removed, so this shouldn't occur if you built the container from that state?

[boonwj]

Alright, I shall remove all my current images and rebuild again