sjeknic commented 2 years ago

This PR introduces the last set of fixes and changes before release v0.0.0. All future changes to origin/main will have to be packaged as their own release. This probably isn't a comprehensive list of changes, but let me try:

Names of Operations are changed. The issue was that Track the file type and Track the Operation could not be imported at the same time. So instead, Track, Segment etc are now changed to new names. Probably won't keep the names, but for now, it's fine.
Another API-breaking change in Operation.add_function_to_operation(). save_name kwarg is changed to save_as and both output_type and save_as must be passed as named args. In fact, the only positional arg allowed is the name of the function, no positional arguments can be passed to that function.
save=True by default in Operation. Intermediates are now saved if save_as is provided, regardless of the value of save.
A new folder to save data_frame.hdf5 is no longer made.
ExperimentArrays can now merge conditions. It does this using the required metric position_id which can either be manually set using a dictionary passed to Orchestrator, using ConditionArray.set_position_id(), or by letting Orchestrator attempt to guess. Merged conditions are stacked and accessed via the condition name.
Moved several files around. Created core directory for a lot of the files that the user won't see.
Operations no longer use __slots__. If the user sub-classed Operations, but did not include __slots__ (which seems likely), then they wouldn't have been used anyways. It's a little cleaner this way, and prevents AttributeErrors in some cases.
61 All Operations can now load all inputs. They get passed to super().__init__() by default. They are also renamed to remove the input_ prefix.
Some of #71. SlurmController gets several small updates. It now records more information (including the partition) and cleans up files after it is finished running.
_metrics is moved to extract.py. This makes it more visible to the user. Additionally, bbox and centroid have much more understandable names.
ConditionArray squeezes arrays before running, but ensures that the array is at least 2D. If only one frame, then np.expand_dims(axis=-1) is used before return.
46 Derived metrics, such as ratios of intensities in different channels/regions, is added. The user defines the functions, placeholders are added during extract, and then the metrics are calculated. For now, only numpy functions can be used. Otherwise, the Pipeline cannot be saved/loaded as a yaml file.
73 Filters are also added, and used similarly to derived metrics above. Extract metrics and what not are calculated, then ConditionArray.generate_mask() and ConditionArray.filter_cells() is used to remove cells defined by the filter. Currently only supports functions in utils.filter_utils.
ExperimentArray can be indexed with multiple keys. The result is not a list, but a custom object that can be indexed in the same way that ConditionArray is indexed.
custom_function decorator is deleted.
More functions have been added to various Operations. Important ones include getting skimage.level_set in Segmenter and wavelet-based background subtraction in Processor.
A number of basic operations for changing the type of inputs and applying boolean masks are added and inherited by all Operations.
54 Warnings from the warnings package are now also handled by any logger from utils.log_utils.
A variety of small bug fixes. Examples include force_rerun not being honored in all cases and Operations not saving all important info in dictionaries.
unet-predict is in both Processor and Segmenter now.

sjeknic commented 2 years ago

74 is partially addressed by the most recent commits. `Operation.get_inputs_and_outputs()` now accounts for the value of `force_rerun`. It's still not ideal, and the inputs and outputs could be more consolidated. But going to close the issue, and reference this PR.

sjeknic commented 2 years ago

Okay, I am now at least 95% confident that I have addressed #76 in this PR. See the issue for more details.

sjeknic / CellTK

Pre-release changes #75

61 All `Operations` can now load all inputs. They get passed to `super().init()` by default. They are also renamed to remove the `input_` prefix.

73 Filters are also added, and used similarly to derived metrics above. Extract metrics and what not are calculated, then `ConditionArray.generate_mask()` and `ConditionArray.filter_cells()` is used to remove cells defined by the filter. Currently only supports functions in `utils.filter_utils`.

54 Warnings from the `warnings` package are now also handled by any logger from `utils.log_utils`.

74 is partially addressed by the most recent commits. `Operation.get_inputs_and_outputs()` now accounts for the value of `force_rerun`. It's still not ideal, and the inputs and outputs could be more consolidated. But going to close the issue, and reference this PR.

sjeknic / CellTK

Pre-release changes #75

61 All Operations can now load all inputs. They get passed to super().__init__() by default. They are also renamed to remove the input_ prefix.

73 Filters are also added, and used similarly to derived metrics above. Extract metrics and what not are calculated, then ConditionArray.generate_mask() and ConditionArray.filter_cells() is used to remove cells defined by the filter. Currently only supports functions in utils.filter_utils.

54 Warnings from the warnings package are now also handled by any logger from utils.log_utils.

74 is partially addressed by the most recent commits. Operation.get_inputs_and_outputs() now accounts for the value of force_rerun. It's still not ideal, and the inputs and outputs could be more consolidated. But going to close the issue, and reference this PR.

61 All `Operations` can now load all inputs. They get passed to `super().init()` by default. They are also renamed to remove the `input_` prefix.

73 Filters are also added, and used similarly to derived metrics above. Extract metrics and what not are calculated, then `ConditionArray.generate_mask()` and `ConditionArray.filter_cells()` is used to remove cells defined by the filter. Currently only supports functions in `utils.filter_utils`.

54 Warnings from the `warnings` package are now also handled by any logger from `utils.log_utils`.

74 is partially addressed by the most recent commits. `Operation.get_inputs_and_outputs()` now accounts for the value of `force_rerun`. It's still not ideal, and the inputs and outputs could be more consolidated. But going to close the issue, and reference this PR.