Providing logs for users

[juliomateoslangerak]

Going back to the discussion early this week about logging tools on cockpit. Apart from the developer logs and the more structured image metadata, we should provide users with experimental logs.

When doing multisite experiments or, looking forward, with more complex experiments through some custom scripts, the user will be happy to keep a human readable log of what happened. That comes back to saving into a file what is currently printed into the stout panel. It provides some traceability but also some confidence to the user about what happened last night. I think it would it make sense to print into this user log file the path to the developer log.

[iandobbie]

This issue has come up in the past and I advocated for a text log with each experiment. An alternative suggestion was to store this data in the image file and have tools to extract the data from the image file. I see the idea behind this to keep the data associated with the file, but we don't have a sensible plase to put this data in the current format. I dont think the free form 800 char area in the header is a sensible location. Also I am very keen on a plain text file as this is easy to open and read.

So what information do you think should go into this file?

Suggestions: Start time. Light sources - power cameras - settings, exposure time, imaging modes, binning, roi etc... experimental params - z stack, repetitions, delays...... Image file name and location XYZ coords of initial position

Any other suggestions?

[juliomateoslangerak]

I also would advocate for a text file associated with the experiment (also, one file per multi-site experiment). While it is a better practice add this info to the metadata sections of the image, in practice, the per experiment log file (that is indeed what I would like) is very practical and usable by the user. Maybe this is something to add in the context of #881, as the NGFF allows dumping arbitrary files into its file structure.

As for what to put in there, difficult to say. What I find my self regularly checking is the timestamps. Did the scope have time to cycle through all positions in time? I like the current format that is printed in multisite experiments. I like the story aspect of it. Maybe in yaml?

experiment parameters:
- exposure_times:
  - light_x: 100ms
  - light_y: 150ms
 ...

events:
  ...
  short_timestamp: Going to site 5
  short_timestamp: started stack
  short_timestamp: finished stack
  ...
  short_timestamp: waiting 5.3 seconds until next cycle
  short_timestamp: Going to site 1
  ...

[iandobbie]

Good thoughts. I'm going to test the extended metadata code that is basically done then I will think about this. I think the implementation could be relatively easy. Just open a text file along with the image files in the datasaver and dump data along with the images that arrive.

[carandraug]

This is also duplicate of #179.