Refactoring of the main classes

lauraporta commented 4 months ago

Description

What is this PR

[x] Bug fix
[x] Addition of a new feature
[x] Other

Why is this PR needed? Refactor the main analysis code.

What does this PR do? Legend: ❌: removes 💭: changes ➕: adds

❌ Removes a lot of modules that were useful in the exploration stages;
❌ Removes contrast enhancement (see: #11 ) from the pipeline and related methods;
❌ Removes "adjust increment" (1) boolean as I think I should not give the user this flexibility, increments should be always corrected to achieve a full 360 rotation, as this improves the overall derotation;
💭 Renames a few methods;
💭 Replaces np.nonzero with np.where (just personal preference, they seem equivalent for my use-case);
💭 Changes handling of number of rotations. Now it is estimated from the rotation signal;
➕ Adds automated creation of folders for saving purposes;
➕ Adds a few comments to separate the methods in groups (signal processing, derotation, saving);
💭 Improve usage of pandas syntax
➕ Overwrites the method check_rotation_number_after_interpolation in the IncrementalPipeline class

I can provide some datasets to test the pipeline locally.

Explanations

(1) : with "increments" I refer to the angle resolution of the motor, i.e. the smallest increment in angles that motor is able to report. 0.2 deg in our case.

How has this PR been tested?

Tested locally with tests that were made before.

Is this a breaking change?

No

Does this PR require an update to the documentation?

Not for now.

Checklist:

[x] The code has been tested locally
[ ] Tests have been added to cover all new functionality
[ ] The documentation has been updated to reflect any changes
[x] The code has been formatted with pre-commit

JoeZiminski commented 3 months ago

(derotation-ev) C:\Users\Joe\work\git-repos\derotation>python examples/derotate.py
2024-07-25 14:42:50 PM INFO     2024-07-25 14:42:50 PM - INFO - MainProcess fancylog.py:326 - Starting logging                                                                                                          fancylog.py:326
                       INFO     2024-07-25 14:42:50 PM - INFO - MainProcess fancylog.py:327 - Not logging multiple processes                                                                                            fancylog.py:327
                       INFO     2024-07-25 14:42:50 PM - INFO - MainProcess full_rotation_pipeline.py:122 - Loading data...                                                                               full_rotation_pipeline.py:122
(derotation-ev) C:\Users\Joe\work\git-repos\derotation>python examples/derotate.py
2024-07-25 14:42:58 PM INFO     2024-07-25 14:42:58 PM - INFO - MainProcess fancylog.py:326 - Starting logging                                                                                                          fancylog.py:326
                       INFO     2024-07-25 14:42:58 PM - INFO - MainProcess fancylog.py:327 - Not logging multiple processes                                                                                            fancylog.py:327
                       INFO     2024-07-25 14:42:58 PM - INFO - MainProcess full_rotation_pipeline.py:122 - Loading data...                                                                               full_rotation_pipeline.py:122
2024-07-25 14:42:59 PM WARNING  2024-07-25 14:42:59 PM - WARNING - MainProcess tifffile.py:4539 - <tifffile.TiffFile 'rotation_00001.tif'> <asarray> failed to reshape (8258, 256, 256) to (8500, 256, 256), raised    tifffile.py:4539
                                ValueError('cannot reshape array of size 541196288 into shape (8500,256,256)')
Traceback (most recent call last):
  File "C:\Users\Joe\work\git-repos\derotation\examples\derotate.py", line 3, in <module>
    derotate = FullPipeline("full_rotation")
  File "C:\Users\Joe\work\git-repos\derotation\derotation\analysis\full_rotation_pipeline.py", line 46, in __init__
    self.load_data()
  File "C:\Users\Joe\work\git-repos\derotation\derotation\analysis\full_rotation_pipeline.py", line 144, in load_data
    ) = get_analog_signals(
  File "C:\Users\Joe\work\git-repos\derotation\derotation\load_data\custom_data_loaders.py", line 72, in get_analog_signals
    data = convert_to_volts(data)
  File "C:\Users\Joe\work\git-repos\derotation\derotation\load_data\custom_data_loaders.py", line 49, in convert_to_volts
    data = -10 + 20 * (data + 2**15) / 2**16
OverflowError: Python integer 32768 out of bounds for int16

Config file

``` paths_read: path_to_randperm: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/example_derotation_data/stimlus_randperm.mat" path_to_aux: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/example_derotation_data/aux_stim/230802_CAA_1120182_rotation_1_001.bin" path_to_tif: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/example_derotation_data/imaging/rotation_00001.tif" paths_write: debug_plots_folder: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/output/debug_plots/" logs_folder: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/output/logs/" derotated_tiff_folder: "C:/Users/Joe/work/git-repos/derotation/joe-example-data/output/data_folder/" saving_name: "derotated_image_stack" channel_names: [ "camera", "scanimage_frameclock", "scanimage_lineclock", "photodiode2", "PI_rotCW", "PI_rotCCW", ] # camera,scanimage_frameclock,scanimage_lineclock,photodiode2,PI_rotCW,PI_rotCCW rotation_increment: 0.2 adjust_increment: True rot_deg: 360 debugging_plots: True contrast_enhancement: 0.35 analog_signals_processing: find_rotation_ticks_peaks: height: 4 distance: 20 squared_pulse_k: 0 inter_rotation_interval_min_len: 1000 angle_interpolation_artifact_threshold: 0.15 interpolation: line_use_start: True frame_use_start: True ```

lauraporta commented 3 months ago

Thank you Joe, it seems that this issue might be related to a newer version of numpy. I just upgraded it and found the same exception.

JoeZiminski commented 3 months ago

Otherwise was working okay on numpy 1.23.1, will try on 2.0 after the fix. I had a couple of warnings I will dump here in case they are relevant:

2024-07-25 16:18:18 PM INFO     2024-07-25 16:18:18 PM - INFO - MainProcess fancylog.py:326 - Starting logging                                                                                                          fancylog.py:326
                       INFO     2024-07-25 16:18:18 PM - INFO - MainProcess fancylog.py:327 - Not logging multiple processes                                                                                            fancylog.py:327
                       INFO     2024-07-25 16:18:18 PM - INFO - MainProcess full_rotation_pipeline.py:121 - Loading data...                                                                               full_rotation_pipeline.py:121
                       WARNING  2024-07-25 16:18:18 PM - WARNING - MainProcess tifffile.py:4539 - <tifffile.TiffFile 'rotation_00001.tif'> <asarray> failed to reshape (8258, 256, 256) to (8500, 256, 256), raised    tifffile.py:4539
                                ValueError('cannot reshape array of size 541196288 into shape (8500,256,256)')
2024-07-25 16:18:20 PM INFO     2024-07-25 16:18:20 PM - INFO - MainProcess full_rotation_pipeline.py:170 - Dataset rotation_00001 loaded                                                                 full_rotation_pipeline.py:170
                       INFO     2024-07-25 16:18:20 PM - INFO - MainProcess full_rotation_pipeline.py:171 - Filename: derotated_image_stack                                                               full_rotation_pipeline.py:171
                       INFO     2024-07-25 16:18:20 PM - INFO - MainProcess full_rotation_pipeline.py:249 - Finding rotation ticks peaks...                                                               full_rotation_pipeline.py:249
2024-07-25 16:18:22 PM INFO     2024-07-25 16:18:22 PM - INFO - MainProcess full_rotation_pipeline.py:324 - Cleaning start and end rotation signal...                                                     full_rotation_pipeline.py:324
                       INFO     2024-07-25 16:18:22 PM - INFO - MainProcess full_rotation_pipeline.py:349 - Creating signed rotation array...                                                             full_rotation_pipeline.py:349
                       INFO     2024-07-25 16:18:22 PM - INFO - MainProcess full_rotation_pipeline.py:371 - Dropping ticks outside of the rotation period...                                              full_rotation_pipeline.py:371
2024-07-25 16:18:25 PM INFO     2024-07-25 16:18:25 PM - INFO - MainProcess full_rotation_pipeline.py:407 - Ticks dropped: 8.                                                                             full_rotation_pipeline.py:407
                                Ticks remaining: 137958
                       INFO     2024-07-25 16:18:25 PM - INFO - MainProcess full_rotation_pipeline.py:436 - Number of rotations is as expected                                                            full_rotation_pipeline.py:436
                       WARNING  2024-07-25 16:18:25 PM - WARNING - MainProcess full_rotation_pipeline.py:457 - Number of ticks is not as expected: 137958.                                                full_rotation_pipeline.py:457
                                Expected ticks: 144000.0
                                Delta: -6042.0
                       INFO     2024-07-25 16:18:25 PM - INFO - MainProcess full_rotation_pipeline.py:509 - New increment example: 0.209                                                                  full_rotation_pipeline.py:509
                       INFO     2024-07-25 16:18:25 PM - INFO - MainProcess full_rotation_pipeline.py:522 - Interpolating angles...                                                                       full_rotation_pipeline.py:522
2024-07-25 16:18:27 PM INFO     2024-07-25 16:18:27 PM - INFO - MainProcess full_rotation_pipeline.py:562 - Cleaning interpolated angles...                                                               full_rotation_pipeline.py:562
2024-07-25 16:18:29 PM INFO     2024-07-25 16:18:29 PM - INFO - MainProcess full_rotation_pipeline.py:616 - Calculating angles by line and frame...                                                       full_rotation_pipeline.py:616
                       INFO     2024-07-25 16:18:29 PM - INFO - MainProcess full_rotation_pipeline.py:688 - Plotting rotation ticks and rotation on signal...                                             full_rotation_pipeline.py:688
2024-07-25 16:18:34 PM INFO     2024-07-25 16:18:34 PM - INFO - MainProcess full_rotation_pipeline.py:732 - Plotting rotation angles...                                                                   full_rotation_pipeline.py:732
2024-07-25 16:18:35 PM INFO     2024-07-25 16:18:35 PM - INFO - MainProcess full_rotation_pipeline.py:236 - ✨ Analog signals processed ✨                                                                full_rotation_pipeline.py:236  
                       INFO     2024-07-25 16:18:35 PM - INFO - MainProcess full_rotation_pipeline.py:822 - Starting derotation by line...                                                                full_rotation_pipeline.py:822
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2114195/2114195 [10:20<00:00, 3408.86it/s] 
2024-07-25 16:28:55 PM INFO     2024-07-25 16:28:55 PM - INFO - MainProcess full_rotation_pipeline.py:832 - ✨ Image stack rotated ✨                                                                     full_rotation_pipeline.py:832
2024-07-25 16:28:57 PM INFO     2024-07-25 16:28:57 PM - INFO - MainProcess full_rotation_pipeline.py:941 - Masked image saved in                                                                         full_rotation_pipeline.py:941
                                C:/Users/Joe/work/git-repos/derotation/joe-example-data/output/data_folder/

neuroinformatics-unit / derotation