Water Calibration overhaul

[alexpiet]

Pull Request instructions:

• Please follow the update protocol
• Answer the questions below in detail. Your responses will be emailed to experimenters.
• If the experimenters must do anything new, provide detailed step by step instructions on the wiki
• If computer maintainers need to manually update anything, provide detailed step by step instructions
• Use markdown syntax in order for your comments to be rendered reliably in the email: "1." instead of "1)", use four spaces for indents.
• If you use the keyword "skip email" in the title, it will skip the email updates
• Merges from "develop" into "production_testing" should use the keyword "production merge" in the title for reliable indexing of updates
• Merges from "production_testing" into "main" should use the keyword "update main"

Describe changes:

• Resizes the water calibration dialog to fit on a half screen
• Allows spot checks to be saved without interfering with full calibrations
• Changes default calibration settings to a narrower range of values
• Increases the calibration speed by reducing the interval time
• Streamlines logic for calibration, by removing fields and removing notion of saving calibration parameters
• Makes it easier to repeat a measurement
• Emergency stop closes the valve and cancels the current measurement

What issues or discussions does this update address?

• partially addresses #474
• resolves #456, by only using QInputDialog for inputs
• resolves #297
• resolves https://github.com/AllenNeuralDynamics/aind-behavior-blog/issues/347
• resolves https://github.com/AllenNeuralDynamics/aind-behavior-blog/issues/337

Describe the expected change in behavior from the perspective of the experimenter

• calibration should be significantly faster and easier
• Users should never need to manually delete data

When you open the water calibration dialog, it will look like this:

After pressing Calibrate Left, you will be prompted to enter the before weight:

The calibration will run and provide progress reports in purple text. When its finished, you will be prompted to enter the after weight.

You can then press "Continue" to perform the next measurement. "Repeat" to repeat the measurement of the valve open time currently selected. Or "Finish" to stop measurements. You can perform repeats immediately, or do all measurements and then repeat any measurements you like.

Describe any manual update steps for task computers

• You may delete the WaterCalibrationPar_#.json file in ForagingSettings on each rig. This is not required
• [x] 446/447
• [ ] 428
• [x] Ephys1
• [ ] Ephys3
• [x] To decide: how often to do full calibration, and spot checks

Was this update tested in 446/447?

• [x] tested in 447

[alexpiet]

Consistency of new calibration code with old calibration code

In order to test the new calibration system. I calibrated 1A several times, the results are plotted here. The labels are confusing:

• 2024-06-06_left valve, using the new system, with 0.01s intervals
• 2024-06-07_left valve, using the new system, with 0.10s intervals
• 2024-06-08_left valve, using the new system, with 0.25s intervals
• 2024-06-09_left valve, using the new system, with 0.5s intervals
• 2024-06-10_left valve, using the OLD system, with 0.5s intervals All other labels reflect previous calibrations

We observe a difference in slope between the calibrations (all done today, in a row), and previous calibrations. But all of today's calibrations on old and new system are in agreement. I conclude the new system works, and 0.1s intervals are fine

[alexpiet]

Consistency of calibration across 8 measurements

To assess what threshold we should use for valve drift, I performed 8 spot checks in a row targetting 2uL volumes. I got back: 2.1 +/- 0.0299 mg (mean +/- std). None of the samples would have been flagged by my proposed tolerance of 10% error ( < 1.8, or > 2.2 mg).

If we had used the old calibration settings to target the 2uL volumes, then the results would have been flagged.

I conclude the spot checks are valuable, and the tolerance I used is ok.

I attach to this comment the results of the water calibration test

calibration_notes.txt

Test_1_WaterCalibration_1.json

[alexpiet]

@XX-Yin @hanhou This is ready for review. See two validation test results in the comments

[XX-Yin]

@alexpiet Overall it looks good. I don't have time to review it this week. I will review it next week so we can merge it next week.

[alexpiet]

@hanhou @XX-Yin I added two additional features, and this is ready to review again:

• I added the spot checks to the metadata. Only the spot checks since the last calibration are included.
• I added the spot checks to the calibration plots, but they do not count against the "show last" counter. Only spot checks after the normal calibrations that are plotted are shown.

[XX-Yin]

@alexpiet It's better the GUI can be open without the WaterCalibrationPar_#.json. We can set some default parameters.

[alexpiet]

@alexpiet It's better the GUI can be open without the WaterCalibrationPar_#.json. We can set some default parameters.

Good call @XX-Yin. I updated the code to:

• set default fields for WaterCalibrationPar_#.json
• No longer log an error if WaterCalibrationPar_#.json is not provided, or cannot load
• Only add additional calibration types from WaterCalibrationPar_#.json if they have the required fields.