Backup data

[XX-Yin]

Pull Request instructions:

• Please follow the update protocol
• Answer the questions below in detail. Your responses will be emailed to experimenters.
• If the experimenters must do anything new, provide detailed step by step instructions on the wiki
• If computer maintainers need to manually update anything, provide detailed step by step instructions
• Use markdown syntax in order for your comments to be rendered reliably in the email: "1." instead of "1)", use four spaces for indents.
• If you use the keyword "skip email" in the title, it will skip the email updates
• Merges from "develop" into "production_testing" should use the keyword "production merge" in the title for reliable indexing of updates
• Merges from "production_testing" into "main" should use the keyword "update main"

Describe changes:

1. Back up the data each trial.
2. The code for saving backup data runs in a different thread and does not affect the operation of behavior propagation in the main thread.
3. The data format is the same as normal saving process.

What issues or discussions does this update address?

Back up the data in case the software crashes

Describe the expected change in behavior from the perspective of the experimenter

No

The backup is turned on by default. If you want to turn it off, set save_each_trial in the ForagingSettings.json to False.

Describe any manual update steps for task computers

No

Was this update tested in 446/447?

• [x] Tested on my own computer
• [x] Tested in 447
• [x] Tested in 323_EPHYS3

[XX-Yin]

@yonibrowning @alexpiet @hanhou @galenlynch

I think we should merge this first to use this backup function. If it's really necessary to save the single trial, we can update it later.

[XX-Yin]

Please also make sure that this backup saving mechanism does not interfere with the normal saving process at the end of the session.

Specifically:

• If the session crashes, what part of data will be saved and what part will be missing (at least the post weight)? How about meta data? If the session crashes, the data not entered to the GUI will not be saved (including the post weight). The meta data will be saved. The backup saving triggers the same saving function, so all fields provided to the GUI should be saved.

• If the session runs fine, will the final saving process overwrite the backup file with all correct information included (weight info, e.g.)? Yes, the final saving will overwrite the backup file with all correct information.

• What if the user decides not to save the session in the end? Would you delete the backup file? If the user decides not to save the session, I think it's better to delete the entire session folder.

They are all good questions. The backup saving will not interfere the normal saving steps.

Please see inline one-to-one response

[hanhou]

Please also make sure that this backup saving mechanism does not interfere with the normal saving process at the end of the session. Specifically:

• If the session crashes, what part of data will be saved and what part will be missing (at least the post weight)? How about meta data? If the session crashes, the data not entered to the GUI will not be saved (including the post weight). The meta data will be saved. The backup saving triggers the same saving function, so all fields provided to the GUI should be saved.

• If the session runs fine, will the final saving process overwrite the backup file with all correct information included (weight info, e.g.)? Yes, the final saving will overwrite the backup file with all correct information.

• What if the user decides not to save the session in the end? Would you delete the backup file? If the user decides not to save the session, I think it's better to delete the entire session folder.

They are all good questions. The backup saving will not interfere the normal saving steps.

Please see inline one-to-one response

Sounds good.

I think it's better to delete the entire session folder.

Does this need to be executed by the code?

Or shall we explicitly separate the normally saved file and the backup file like this:

1. The backup thread always saves the data to xxx.backup
2. At the end of the session, if the user saves the data or choose to restart without saving, always delete the backup file.
3. If the session crashes, the backup file remains

This would give them even less opportunity to interfere with each other, and we'll easily know which sessions have crashed.

[XX-Yin]

Please also make sure that this backup saving mechanism does not interfere with the normal saving process at the end of the session. Specifically:

• If the session crashes, what part of data will be saved and what part will be missing (at least the post weight)? How about meta data? If the session crashes, the data not entered to the GUI will not be saved (including the post weight). The meta data will be saved. The backup saving triggers the same saving function, so all fields provided to the GUI should be saved.

• If the session runs fine, will the final saving process overwrite the backup file with all correct information included (weight info, e.g.)? Yes, the final saving will overwrite the backup file with all correct information.

• What if the user decides not to save the session in the end? Would you delete the backup file? If the user decides not to save the session, I think it's better to delete the entire session folder.

They are all good questions. The backup saving will not interfere the normal saving steps. Please see inline one-to-one response

Sounds good.

I think it's better to delete the entire session folder.

Does this need to be executed by the code?

Or shall we explicitly separate the normally saved file and the backup file like this:

1. The backup thread always saves the data to xxx.backup
2. At the end of the session, if the user saves the data or choose to restart without saving, always delete the backup file.
3. If the session crashes, the backup file remains

This would give them even less opportunity to interfere with each other, and we'll easily know which sessions have crashed.

To simplify steps, I just added a tag saving_type_label to differentiate normal saving and backup saving. If the user clicks the Save button, it will overwrite the existing file with label changed from backup saving to normal saving.

[XX-Yin]

@yonibrowning has agreed to merge this for use in the ephys rig, although he thought we need more discussions in the future.

Can we approve it so that we can merge it this week?

[alexpiet]

Have we tested this with FIP running?

[XX-Yin]

Have we tested this with FIP running?

Not yet, I don't stop the fib photometry for backup saving. It shouldn't influence the fiber photometry, but I will test it.

[XX-Yin]

@alexpiet I tested the fiber photometry, it worked.

[alexpiet]

@XX-Yin I will review tomorrow, and we can merge before friday