New data file format review

[alustig3]

• fixed broken custom variable dialogs. fix was needed because state_machine.py no longer returns a string repr() of the value, but instead returns the value itself

• removed the tsv/txt "file type" dropdowns and instead made a "file type" setting in the settings dialog / user_settings.json

  • I don't imagine users would be going back and forth between file formats, but instead will adopt one format and have it apply across all tasks/experiments.
  • this does remove the possibility of specifying format per each experiment, as it is no longer a parameter saved in the experiment configuration. But I don't think this is a problem (see above point)
  • right now the the default is tsv, but maybe it should default to txt? I can see reasons for either.

• I think it makes sense to always print variables at the start of the task.

  • This makes it easy to know the initial state of all variables, instead of having to infer what the values are based on what the task has as defaults / what was overwritten by a persistent variable or from configure experiment's variables table.
  • It is possible to adjust variables once an experiment is "Run" but before an individual setup/task starts. These variable adjustments are not logged to the datafile, so evidence of these variable modifications become lost information. Printing the state of all variables at the start of a task run fixes this problem of information being lost.

• fixed a bug that also exists on the dev branch, where changing what subjects are being run in the experiment will cause the program to freeze because the pyControl would be trying to access a subject.plot from a previous experiment run that had been deleted. error below

  2023-05-26 10:54:39,841 Traceback (most recent call last):
  File "/Users/lustiga/Dropbox (HHMI)/Code/karpova_lab/pyControl/pyControl_official/gui/configure_experiment_tab.py", line 354, in run_experiment
  self.GUI_main.run_experiment_tab.setup_experiment(experiment)
  File "/Users/lustiga/Dropbox (HHMI)/Code/karpova_lab/pyControl/pyControl_official/gui/run_experiment_tab.py", line 135, in setup_experiment
  self.experiment_plot.set_state_machine(self.subjectboxes[0].board.sm_info)
  File "/Users/lustiga/Dropbox (HHMI)/Code/karpova_lab/pyControl/pyControl_official/gui/plotting.py", line 303, in set_state_machine
  subject_plot.set_state_machine(sm_info)
  File "/Users/lustiga/Dropbox (HHMI)/Code/karpova_lab/pyControl/pyControl_official/gui/plotting.py", line 48, in set_state_machine
  self.states_plot.set_state_machine(sm_info)
  File "/Users/lustiga/Dropbox (HHMI)/Code/karpova_lab/pyControl/pyControl_official/gui/plotting.py", line 113, in set_state_machine
  self.axis.clear()
  File "/Users/lustiga/.local/share/virtualenvs/pyControl_official-H2ResLOC/lib/python3.10/site-packages/pyqtgraph/graphicsItems/PlotItem/PlotItem.py", line 603, in clear
  self.removeItem(i)
  File "/Users/lustiga/.local/share/virtualenvs/pyControl_official-H2ResLOC/lib/python3.10/site-packages/pyqtgraph/graphicsItems/PlotItem/PlotItem.py", line 588, in removeItem
  self.vb.removeItem(item)
  File "/Users/lustiga/.local/share/virtualenvs/pyControl_official-H2ResLOC/lib/python3.10/site-packages/pyqtgraph/graphicsItems/ViewBox/ViewBox.py", line 447, in removeItem
  scene = self.scene()
  RuntimeError: wrapped C/C++ object of type ViewBox has been deleted

[ThomasAkam]

Thanks @alustig3 for the code review.

Regarding whether we should have a drop-down for the file type, my thoughs are:

• In the long run we want all users using .tsv not .txt files. The new file format is better and we don't want to support both formats over the long term as it makes maintainance more complicated. I therefore do not like the option of having it as a setting in the settings dialog.
• In the short term I was thinking it might make sense to support both, on the assumption that old projects would continue with .txt but new projects would use .tsv, hence the drop down menu.
• A third option would be to only offer .tsv in the next release. I had been thinking with all the new functionallity added in this release we might want to call it v2.0, and doing so would also indicate that there are some changes that are not fully backward compatible. I do want to ensure that the data_import.py functionallity can generate as close as possible to identical Session objects or session_dataframes from either .tsv or .txt, to make it straightforward to mix and match file types in the same project without having to change analysis code (assuming the analysis uses the provided data import functions).

What do you think?

[alustig3]

I agree @ThomasAkam about it being best to avoid having to support both formats long term.

Doing a v2.0 release with only .tsv seems reasonable to me, especially if we did a v2.0rc version beforehand to hopefully get some more feedback/testing from others.

[ThomasAkam]

Sounds good, I'm happy with dropping the .txt format for the next v2.0 release. Agree a v2.0rc for testing is a good idea.

Please could you remove the stuff from the settings dialog for setting the file type and then I'll merge this.

[alustig3]

@ThomasAkam I removed the stuff from the settings dialog

I hardcoded the data logger to use tsv. This was the quickest and easiest path, but maybe you want to also remove the "file_type" argument from data_logger's open_data_file() all together and get rid of the self.file_type from the Data_logger class?

[ThomasAkam]

Thanks, I'll take out the functionallity for generating the .txt files once I have merged this.

I noticed there is some code in dialogs.py defining a Radio_setter class which does not appear to be used anywhere. Should this be removed?

[ThomasAkam]

Thanks Andy!