PIPELINE: Major refactoring needed [WIP]

[nielsboecker]

Description

Our HoloPipelines code base has some major flaws, which already lead to architectural and maintanance-wise problems:

• we barely distinguish between what we are doing ("resample an image") and technical details, which are not necessarily related to the tas ("dicom2numpy")
• similarly, the pipelines that are physically in our code ("nifty2glb") do not obviously reflect the semantic use cases that we offer to the user ("Bone segmentation pipeline")
• we are misusing the concept of "components", as everything is now just a component
• we are inconsistent in our naming conventions and often break Python style guides

Suggested solution

Major refactoring! :rocket:

• A new naming scheme: foo_bar.task.py instead of compFooBar.py
• Different types of components:
  • tasks instead of components
  • services for more complex stuff, like format conversion
  • wrappers for external tools and existing solutions
  • clients for communications like HTTP
  • adapters to abstract similar http calls, e.g. to the NiftyNet API we defined in the /models directory
• Clear separation of concerns for what used to be "components"
  • pipelines are made up entirely from tasks
  • tasks can use wrappers for external services, services for complex code, clients if needed
  • tasks can implement some functionality directly inside them if appropriate, but it is preferred to extract as much as possible to other types of components
• Rename methods and variables to have snake_case

Suggested new structure

/core/
   pipelines.controller.py                  # Only for direct local invocation
   main.server.py                           # Just to listen to endpoints and then
                                            # delegate to other files, like
                                            # jobs.controller or jobs_status
   jobs.controller.py                       # Start pipelines in production
   jobs_status.py                           # Keep track of jobs (could maybe
                                            # be kept in jobs.controller instead)
   garbage_collector.py                     # Perform the "clean up" for jobs

   /pipelines/
      bone_segmentation.pipeline.py
      abdominal_organs_segmentation.pipeline.py
      lung_segmentation.pipeline.py
      airway_segmentation.pipeline.py
      README.md                             # Descriptions of each pipeline, like 
                                            # in HoloPipelines/models/README.md

      /config/
         pipelines_list.json
         io_paths.py
         config.ini

      /tasks/
         /shared/
            receive_input.task.py
            dispatch_output.task.py
            resample_numpy.task.py
            rotate_numpy.task.py
            crop_numpy.task.py
            downscale_numpy.task.py
            convert_format.task.py
            simplify_mesh.task.py           # fast quadratic mesh simplification

         /bone_segmentation/
            segment_bone.task.py

         /abdominal_organs_segmentation/
            segment_abdominal_organs.task.py
            split_to_separate_organs.task.py  # probably out-of-scope
            colour_organs.task.py             # probably out-of-scope
            merge_separate_organs.task.py     # probably out-of-scope

         /lung_segmentation/
            segment_lung.task.py

         /airway_segmentation/
            segment_airway.task.py

      /services/
         format_conversion.service.py       # contains methods: 
                                            # - transform_dicom_to_nifty,
                                            # - transform_dicom_to_numpy, 
                                            # - transform_nifty_to_numpy, 
                                            # - transform_numpy_to_obj

      /wrappers/
         obj2gltf.wrapper.py
         lung_and_airway_segmentation.wrapper.py # Even though it's python code, we use
                                            # existing work as-is, so let's wrap it
         simplifier.wrapper.py              # Yet to implement

      /clients/
         http.client.py                     # does the technical network calls

      /adapters/
         niftynet.adapter.py                # uses http client to call "/model"

Files to delete

• /tools/showHU.py: not used, not needed
• the whole lungSegment folder, which is just copied foreign code except for the main.py
  • Add https://github.com/wanwanbeen/ct_lung_segmentation in requirements.txt instead: See this thread
  • if anything needs to be done before calling this library (what was done in your main.py until now), use lung_and_airway_segmentation.wrapper.py instead
• pipelineNifty2Glb and pipelineNity2Obj
  • replace actual use cases with concrete pipeline, like the bone_segmentation

Open questions

• Is it possible for all tasks to have the same I/O? So can they implement exactly the same API, so that they can easily be shuffled around?
  • If yes, could we maybe have a Task class, and have the other tasks be sub-classes of that?

[nielsboecker]

@boonwj Would appreciate your feedback on my ideas, and also if I forgot any key issues.

@UdomkarnBoonyaprasert you know this component best. Think about any potential issues this refactoring could create and how we could solve them, please.

[nielsboecker]

Update: The foo_bar.type.py naming scheme won't work because not valid python. Just leave out the postfix, through the directory / module name, it is obvious what kind of component it is.