Add text status file to archive

[g7gpr]

Create a text file, in an extensible format, containing salient data of previous observation containing information such as

#RMS_update date and the commit hash from the last RMS_Update on the following lines:
Date:   Mon Jul 22 10:47:01 2024 -0400   
commit: 3a20e04d8fe8104b02af82b3fc34d3e00ace2bd4 (HEAD -> master, origin/master, origin/HEAD, dalava/master)                            
Merge: 3344c26 2aee258           
pi_version: Raspberry Pi 4 Model B Rev 1.1     
os_version: Debian GNU/Linux 11 (bullseye)     
storage_space: 117G      
fits_files: 3079         
fits_missing: 0          
minutes_missing: 0   
storage_status: 4      
real_meteors: 43       
number_fits_detected: 37    
photometry_status: good   
dropped_frames:    2    
jitter_quality: 100.0%    
dropped_frame_rate: 0.0%          
last_calculated_FPS: 24.980993   
sensor : IMX291
field_of_view : 90 x 50

add to the archive to be uploaded.

[peschman]

Yes, including camera sensor type, lens and capture method might be good. Luc is planning to add 'protocol: udp' in config at some point in the near future.

[g7gpr]

protocol udp is in test at the moment

[g7gpr]

date                                 : 20240730_064325
commit                               : 24481635d80ec742e3a9f01bc946f803cef1dc74
merge_parents                        : 073138b2989b7039913e7833470a9291d1eb0f63
hardware_version                     : x86_64
os_version                           : Linux-5.10.0-29-amd64-x86_64-with-glibc2.36
storage_total                        : 232.73GB
storage_used                         : 166.08GB
storage_free                         : 54.76GB
fits_files                           : None
fits_missing                         : None
minutes_missing                      : None
capture_directories                  : None
detections_before_ml                 : None
detections_after_ml                  : None
number_fits_detected                 : None
number_detections_before_ml          : None
number_detections_after_ml           : None
number_fits_detected                 : None
photometry_good                      : None
dropped_frames                       : None
jitter_quality                       : None
dropped_frame_rate                   : None%
last_calculated_fps                  : None
sensor_type                          : None
lens                                 : None
protocol                             : None
media_backend                        : None

Blocked out the concept.

[g7gpr]

repo                            : g7gpr/feature/summary_file
date                            : 20240730_102327
commit                          : a68c4928b15da37e2f626804a59ba8dbfdb0ba1d
merge_parents                   : 54e5a9c4c341c852075698131599f20327ecc1ec
hardware_version                : raspberry pi 4 model b rev 1.5^@
storage_total                   : 117GB
storage_used                    : 85GB
storage_free                    : 28GB
fits_files                      : None
fits_missing                    : None
minutes_missing                 : None
capture_directories             : 0
detections_before_ml            : 0
detections_after_ml             : 0
number_fits_detected            : 0
number_detections_before_ml     : 0
number_detections_after_ml      : 0
number_fits_detected            : 0
photometry_good                 : False
dropped_frames                  : None
jitter_quality                  : 97.368
dropped_frame_rate              : 0.000%
last_calculated_fps             : None
sensor_type                     : None
camera_pointing                 : az 85.24°, alt 30.65°
camera_fov                      : horizontal 88.62°, vertical 47.10°
lens                            : 4mm
protocol                        : None
media_backend                   : None

This is missing some attributes, because it was a reprocess rather than a capture. Is this heading in the right direction?

[peschman]

Yes, I think things are developing nicely, thanks!

[nzstevem]

For completeness perhaps include the camera id, and status. Also suggest splitting the pointing and fov into seperate variables on their own line for ease of subsequent processing. Also2 including the units in the variable name rather than the rhs will make things much easier downstream. i.e. storage_total_gb: 117, lens_mm: 4

[g7gpr]

Hi nzstevem

There will also a be a json file for downstream processing, this text file is more intended for human consumption, unless peschman wants to comment here?

[g7gpr]

Sorry nzstevem - what do you mean by status? This is coming from the station - the station does not know the camera status, this is generated at the server.

[nzstevem]

Correct. My mistake. I didn't understand. I was thinking that someone might want to take all these files and do comparison statistics, especially after a software change, and they would probably need to exclude some cameras - especially the ones that have not uploaded for some time or those that were beset by a cloudy night. The date can be used for filtering out older files and if the cam id is there its always possible to go back to the weblog and get the status from there. :-)

[g7gpr]

You've made a good point - the date in this file is the commit date; not the observation date. Needs clarifying, thank you.

[peschman]

I agree that a parallel JSON version would be a good idea.

[peschman]

I think we may want to offer the owner/operator the possibility of saving these files in a sub directory under their username on the Pi or Linux box they are running. Once we have a local store of summary data for each night, one could build these into a database allowing summary of certain results over time. The summary filename would need to include station ID and UTC data recording date.

[g7gpr]

Why not write straight to a database? Sqlite is compiled into all versions of Python used on RMS.

[peschman]

If writing to a database is not hard to do, then it would be an added benefit. Would there be a database on each station, or would this be centralized? We started out with target that had been stated when Denis and I were first discussing it: "a file that is human and machine readable"

[g7gpr]

Writing to a database is trivial, just requires robust error handling.

[g7gpr]

Database would be per station.

[nzstevem]

I wonder a little what the point of a (sqlite) database is. Who is going to use it and for what?

To be of use to an operator they are going to either need to have programming knowledge or a utility like dbbrowser installed. And understand some SQL if they want to do anything more than browse.

It might be better to hold off until a broader strategy of what a local database will hold, and how it will be maintained and preserved is established.

Another consideration is what to do in a multi-cam system (which I think will proliferate as PI5's become the norm)? Is there one database on the system or one per camera? If there is a database held locally and the future intended use case is some sort of local operational dashboard I think you'd probably want a single instance so you could use the sqlite engine's filtering, sorting, and aggregate functions across multiple cameras.

[g7gpr]

IMHO, one database per camera. I can see the advantages of a single database covering all cameras, but I see stronger advantages in keeping the potential for interactions between cameras on a single station to minimum possible.

My use case would be to look for changes in the latest commit associated with some change in performance drop off. I'd find this much easier to do with a a bit of SQL, rather than searching through text files.

As for the future use cases, if I can put in the basic architecture, imperfect though it may be, others can build on that if they choose.

[nzstevem]

For your use case it would presumably be useful to have a software version (or commit id) attribute on the row - if this was possible.

[dvida]

I tend to agree with Steve, for web purposes (folks might want to build their own dashboards) and the ease of reading, a simple JSON will do for now. We can modify things in the future if needed. As for the DB, we could host one on the server and simply fill it in with all the JSONs.

[g7gpr]

Do people want the json to have the same name for every observation session for ease of import?

[g7gpr]

observation_summary.json

[dvida]

Dave, this looks great. Can you add line breaks and automated indents in the JSON file? See how RMS is generally saving them for a template. I suggest that all non-permanent files (permanent files are the config, platepar, and mask), have names stamped with the station code and the date of the run. E.g. AU0001_20240814_012345_observation_summary.json. We can then link these on the weblog to a static name.

[g7gpr]

OK - the text file is saved with the RMS standard convention, I'll do the same with the json.

[g7gpr]

With indentation style copied from https://github.com/CroatianMeteorNetwork/RMS/blob/dc8b91e85e665ec493b499c5c88a5b079c817830/RMS/Astrometry/ApplyRecalibrate.py#L577

{
    "Duration": "40970.398",
    "StartTime": "2024-08-12 23:30:17.554072",
    "capture_duration_from_fits": "40970.398",
    "captured_directories": "0",
    "commit_date": "20240812_213307",
    "commit_hash": "31067605b4868f8aeb4679efde4205bd09949ab0",
    "detections_after_ml": "0",
    "dropped_frame_rate": "None",
    "fits_file_shortfall": "3897",
    "fits_file_shortfall_as_time": "11:05:05.280000",
    "fits_files_from_duration": "4001.0154296875003",
    "hardware_version": "x86_64",
    "jitter_quality": "None",
    "storage_free_gb": "176.27",
    "storage_total_gb": "238.31",
    "storage_used_gb": "49.86",
    "time_first_fits_file": "2024-08-11 10:39:05.995000",
    "time_last_fits_file": "2024-08-11 22:01:56.393000",
    "total_expected_fits": "4001",
    "total_fits": "104"
}

[nzstevem]

On naming convention ... does this mean they will match the captured, detected etc date/time prefixes ? which would be good.

[nzstevem]

There's a hidden feature in the status viewer to display this test file enabled by an ?options=T suffix on the url.

i.e. https://globalmeteornetwork.org/status/?options=T

the "=" on the camera panel will display the test file.

It's dynamic so any added details should just appear.

[g7gpr]

AU0006_20240811_101142_903530_observation_summary.json AU0006_20240811_101142_903530_observation_summary.txt

Good that it is dynamic; I've restructured the code to make adding new keys easier.

[dvida]

The formatting looks great now. And yes, the idea is that they have a common name to other files, as Dave's examples show. As far as I'm concerned, this is good to go. Let me know if you're working on anything else. If not, I'm merging.

[g7gpr]

Don't merge yet please. The code needs much more stress testing. How does it handle reprocess, broken captures etc. Many of the cameras in Western Australia are running this now, so the server mods could be started to move the json into static.

[dvida]

Could you set it as draft then?

[nzstevem]

https://globalmeteornetwork.org/status/?options=T

The ?options=T suffix activates the status viewer test switch which in this case enables the ability to see these files.

A new symbol “=” (it might be better @, or????) is added to the camera panels which when clicked shows this data. Currently it only shows the test file. As soon as the location and naming of these files with respect to the camera file hierarchy is stable I’ll update the code to pick up the correct copy for each camera, and move this out of my test mode.

From: Denis Vida @.> Sent: Tuesday, 13 August 2024 2:25 pm To: CroatianMeteorNetwork/RMS @.> Cc: nzstevem @.>; Comment @.> Subject: Re: [CroatianMeteorNetwork/RMS] Add text status file to archive (Issue #316)

Could you set it as draft then?

— Reply to this email directly, view it on GitHub https://github.com/CroatianMeteorNetwork/RMS/issues/316#issuecomment-2285221982 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AHANPBOHGMRE3ZLASHCWVMLZRFU75AVCNFSM6AAAAABLVEQJIOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBVGIZDCOJYGI . You are receiving this because you commented.Message ID: @.***>

[g7gpr]

I don't even have a PR open on this at the moment. Perhaps you're asking me to open a draft one; which I will do.