Add headless pipeline test to github actions

[callum-jpg]

Goal: Run pipelines composed of plugin modules in a github action to ensure there are no glaring errors.

Still to do:

• split up installations into individual jobs that inherit the initial cellprofiler installation
• fix stardist
• add other plugins

[callum-jpg]

Currently, there's an issue with the testing strategy. When CellProfiler runs in headless and a pipeline fails to load, the pipeline still runs and does not send a shell exit code that causes a fail in the github action. For this reason, we rely on the file test_run.sh to search the headless mode logs to see if the plugin actually ran or not. However, this strategy isn't ideal and it doesn't seem to work.

For example, here's the terminal output that passed, when it really should have failed:

Fri Jun  2 21:15:18 2023: Image # 1, module Images # 1: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Fri Jun  2 21:15:18 2023: Image # 1, module Metadata # 2: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Getting image reader for: DNA, None, file:/home/runner/work/CellProfiler-plugins/CellProfiler-plugins/tests/headless_test/test_pipeline_img/skimage-mitosis-img.tiff
Falling back to Java reader.
Getting image reader for: None, None, file:/home/runner/work/CellProfiler-plugins/CellProfiler-plugins/tests/headless_test/test_pipeline_img/skimage-mitosis-img.tiff
Fri Jun  2 21:15:18 2023: Image # 1, module NamesAndTypes # 3: CPU_time = 0.41 secs, Wall_time = 0.22 secs
Fri Jun  2 21:15:18 2023: Image # 1, module Groups # 4: CPU_time = 0.00 secs, Wall_time = 0.00 secs
Getting image reader for: DNA, None, file:/home/runner/work/CellProfiler-plugins/CellProfiler-plugins/tests/headless_test/test_pipeline_img/skimage-mitosis-img.tiff
Error detected during run of module RunStarDist
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/cellprofiler_core/pipeline/_pipeline.py", line 1016, in run_with_yield
    self.run_module(module, workspace)
  File "/opt/hostedtoolcache/Python/3.8.16/x64/lib/python3.8/site-packages/cellprofiler_core/pipeline/_pipeline.py", line 1349, in run_module
    module.run(workspace)
  File "/home/runner/work/CellProfiler-plugins/CellProfiler-plugins/./active_plugins/runstardist.py", line 294, in run
    data, probs = model.predict_instances(
UnboundLocalError: local variable 'model' referenced before assignment
Fri Jun  2 21:15:18 2023: Image # 1, module RunStarDist # 5: CPU_time = 0.00 secs, Wall_time = 0.00 secs
HDF5Dict.flush(): /tmp/Cpmeasurementsvbo86ahg.hdf5, temporary=True
HDF5Dict.__del__(): /tmp/Cpmeasurementsvbo86ahg.hdf5, temporary=True
Closing reader <bioformats.formatreader.ImageReader object at 0x7f17f851d4c0>
pipeline_exception
Error in sys.excepthook:

Original exception was:
Pipeline ran successfully

So it seems like CellProfiler still 'ran' RunStarDist (since we can see Fri Jun 2 21:15:18 2023: Image # 1, module RunStarDist in the terminal output), despite encountering an error within the module. The cause of the above error is due to a setting the the cppipe file being incorrect, but it's obviously not ideal that the github action + shell script is not able to detect this. Ideal solution: create some sort of flag that causes cellprofiler in headless mode to exit out on error

[ErinWeisbart]

A hacky solution we could use until we get something nicer implemented: Determine which module number the module-of-interest is and then look for for the module after that in the output. (Obviously would require that our test pipeline has a module after all of modules we're testing in the the pipelines we're using)

[callum-jpg]

An alternative method of testing that @bethac07 mentioned was to use the CellProfiler done file, which is returned when you add the -d flag in headless mode.

Examples: Make done file - https://github.com/DistributedScience/Distributed-CellProfiler/blob/master/worker/cp-worker.py#L249 Read done file https://github.com/DistributedScience/Distributed-CellProfiler/blob/master/worker/cp-worker.py#LL296C32-L296C45

So if the pipeline has run successfully, the done file will reflect this. However, in some quick local testing using a pipeline that contained a plugin that was incorrect (ie the plugin wasn't loading, but the pipeline still ran) I was getting a done file that said the pipeline was a success when it wasn't. However, there may be some nuance I'm missing in making the done file work as expected.

The updated cellprofiler headless command in the test.yml file is: python -m cellprofiler -c -r -p ./tests/headless_test/4.2.5_plugins_test_pipeline_BASIC.cppipe -i ./tests/headless_test/test_pipeline_img -o . --plugins-directory=./active_plugins -d "./cp.is.done"

[callum-jpg]

Documentation checklist for contribution of new plugins

1. Create a pipeline that uses the new plugin
2. Add plugin requirements to setup.py
3. Create a plugin_name.yml test file, based on test.yml that runs your plugin pipeline using CellProfiler in headless mode
4. TBC: ensure the pipeline has been run correctly by checking the Done file or checking logging output with test_run.sh

Additional note: I think it's a good idea to split up the test.yml file into distinct files for each pipeline. Ie. having a test_basic_plugins.yml, test_cellpose_plugin.yml etc. Currently, a single file is used and the environment continually updated to check the next plugin in the sequence. Thus, the testing environment isn't "fresh" and can become quite tricky to maintain as more plugins are added.