Web page, statistics lost on restart / reboot

[g7ruh]

Hi Jasper, thanks for super program. I have recently got into AIS decoding and your software gets better and better! Keep up the good work.

On using the latest version (0.42), I really like the graphs. The problem is that AIS-catcher closed, I think when the USB cable was moved and caused voltage loss to the dongle.

On restart, the plots had reverted to zero. This is OK for the minute graphs but the per hour graphs take days to repopulate. Is it possible to store the graph data every so often, say every hour to save excessive writes to SD cards, and recover that data at restart or when installing an update?

That way the longer timeframe plots would have a gap in them for the missing time, but at least have the plot available to see long term trends.

Roger G7RUH Near Southampton, UK

[jvde-github]

Hi Roger, thanks for using the program. I like the suggestion and will implement this. In fact, it would make it possible to also include daily stats (a bit useless now).

Something like -H FILE yyyy to backup/persist the content of the map and plots in a file yyy.

I am still fiddling a bit with the internal data structures so want to clean that up first, but will definitely add this (and daily stats).

I find them useful as well to compare against the message count in MarineTraffic et al.

Thanks for suggesting, Jasper

[jvde-github]

Hi Roger, please note that the bleeding edge has now the ability to load and save statistics from file. It will not yet save the contents of the database used to populate the map and shipping table.

The command is: AIS-catcher -N 8100 FILE stat.bin

If the program terminates it will save the statistics to file and will read them again (if available) on open. There is no option to periodically write to the file. To achieve this, the best option is to periodically restart the program (e.g. if you run via systemd to provide a max run time).

Please note that this functionality could break down between versions if data structures significantly change but at least it provides some form of basic persistency.

Jasper

[g7ruh]

Hi Jasper, Thanks for doing this upgrade! I like the colours on the graphs, the distinction between message types is useful.

I wonder if the graphs are a little large for the browser / monitor? My comments are made using Firefox and page at 100% zoom Big monitor, and a 'Pi Chromium via VNC for the smaller.

Two different screen sizes and the way the page is rendered at 100% zoom attached to give you some idea of what I am seeing. On the big monitor I do not get the bottom of the lower graph displayed and even when zooming the print gets a lot smaller but the graphs still seem quite large. On the smaller monitor, quite a lot of scrolling is needed.

I have tested the save data function and it does save and restore OK and tells you on restart if it cannot find a file. I have used a full path to store and get the file which works fine too.

I am not sure restarting AIS every so often is viable as the sites I feed seem a little fussy about reconnecting sometimes. Minimising the connection breaks helps this, I think. A save 'every so often' would help too as the program is fairly stable and I prefer to let it run until a new version is available.

As far as ship plot and ship list goes: these are transient in nature and do not need to be stored, just to recover a list or plot. Within 10 minutes they will be back to showing a realistic live state. Having the ability to see short term and long term trends for the other things is very helpful in spotting issues, hence my request.

If things break in future versions, because of changes, that is fine, it is the basic level of persistency, as you say, which is the primary purpose.

As an aside, is there a How-To for JSON formatted config files, or an example using a little more detail than you have at present? It is rather new to me, format-wise and a bit confusing!

Thanks for adding this feature adding detail and colour to the graphs.

Roger

[g7ruh]

closed in error!

[jvde-github]

Hi Roger,

I have added an experimental backup sercice:

AIS-catcher -N 8100 FILE stat.bin BACKUP 60

will initiate a save every 60 minutes. Have a look whether this works for you. Need some fine-tuning but it worked when I tested it on my system. Feedback appreciated!

Thanks, Jasper

ps Thanks for the comments on the plots. I will clean up the plots at some point. I am thinking of fewer plots but with a switch to choose the granularity (daily, hourly, minute, second).

[g7ruh]

Jasper

BACKUP option testing:

I have used BACKUP 10 so as to get more frequent backup file tests, rather than wait an hour between each!

Server: reading statistics from /home/pi/AIS-catcher/stat.bin Statistics: cannor read file. HTML Server running at port 8100 Server: starting backup service every 10 minutes. [R82XX] PLL not locked! Allocating 15 zero-copy buffers I copied the stat.bin file from the previous bleeding edge version and verified it was there, it gave the cannot read file message and created a new one. Did the file format change or is this an error in recovery process? I guess it might have changed given the new circular charts.

Picky comment, spelling error: "cannor" read file in the error line above.

If I am running the web page on a browser and end AIS-catcher to restart for testing, when I restart AIS-catcher it gives me:

Error: Cannot open port in range [8100,8100]

Message and it seems to take a while after I do a web page refresh with AIS-catcher ended for the port to reset and for the web page to give a message saying it cannot find the page (which is expected if AIS-catcher is not running). Maybe I need to wait a little longer for ports to close?

The circular plots for max distance last hour and last day are much better, after last day circular plot, direction becomes less important and a linear chart showing max distance like MarineTraffic does works fine. The circular plots are a little large but that is the same as my earlier comment about overall size / scale.

With the circular charts, there is an issue with blank space, as receivers near the coastline will only get signals from the sea and not from inland. Unless they live on an island, perhaps half the chart will be blank as it represents the land:

In my case, I am unlikely to get any signals from more than 10NMi north of my location so the top half of the chart (in my case) is not needed. Maybe the chart should be trimmed to remove the 'white space'?

I will post again, a bit later with the results of my BACKUP testing after a few AIS-catcher restarts.

Roger

[g7ruh]

Jasper,

BACKUP testing every 10 minutes works. I get the following console messages:

Allocating 15 zero-copy buffers Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin

and after killing AIS-catcher

Allocating 15 zero-copy buffers Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: initiate backup. Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: writing statistics to file /home/pi/AIS-catcher/stat.bin Server: stopping backup service.

After restarting AIS-catcher it loads the backup file and continues, as designed, with the chart as was when AIS-catcher was ended.

Thank you for adding this, makes it so much better for long term trends.

Roger

[g7ruh]

Jasper

about the distance circles, an issue about distance. Do we need a 'sense check' and take out the errors a bit like bell-curve analysis where you drop the highest and lowest data points as they could be noise or errors in the stats.

Like this which looks like it is a packet located at the 'origin' as in 0 deg lat and 0 deg lon which is 3046NMi from my location.

On OpenCPN I have set a max distance of 300 NMi so I get propagation RX but don't get strange ones. VHF-wise anything greater than 300NMi is unusual and you would have to be watching it to see it.

Roger

[jvde-github]

Roger,

there is a validation logic for signals that are a bit further out. Unfortunately, I put in a placeholder for now (which is a check on having seen the vessel at least twice) which does not work (if for example lat/lon are corrupted in one message). I have quickly extended it with a requirement that we need to see the vessel within a distance of 5nmi between two messages to have it validated. To be refined though and waiting final implementation.

To work around the port not being free when systemd is restart you can use an experimental option -N REUSE_PORT which will ensure the port is not blocked for a following run, even if there are connections in TIME_WAIT.

Note that I changed the format again of the saved statistics so will be reset when starting the latest. I needed to make the statistics handling thread safe.

Thanks, Jasper

[g7ruh]

Jasper,

Testing the -N REUSE_PORT option give me an error, without this option the new bleeding edge version works OK

This start command AIS-catcher -d 00000111 -p 25 -gr tuner 32.8 rtlagc off -u 127.0.0.1 -u 31. -q -N 8100 FILE /home/pi/AIS-catcher/stat.bin BACKUP 60 -N LAT 50.12345 LON -1.54321 &

gives this startup information:

AIS-catcher (build Jan 10 2023) v0.42 (C) Copyright 2021-2023 jvde-github and other contributors This is free software; see the source for copying conditions.There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Searching for device with SN 00000111. Found Rafael Micro R820T tuner UDP: open socket for host: 127.0.0.1, port: 10106, filter: OFF UDP: open socket for host: 31.172.90.245, port: 59794, filter: OFF Server: reading statistics from /home/pi/AIS-catcher/stat.bin HTML Server running at port 8100 Server: starting backup service every 60 minutes. [R82XX] PLL not locked! Allocating 15 zero-copy buffers

This startup command AIS-catcher -d 00000111 -p 25 -gr tuner 32.8 rtlagc off -u 127.0.0.1 -u -q -N 8100 FILE /home/pi/AIS-catcher/stat.bin BACKUP 60 -N REUSE_PORT -N LAT 50.12345 LON -1.54321 &

gives this response

$ AIS-catcher (build Jan 10 2023) v0.42 (C) Copyright 2021-2023 jvde-github and other contributors This is free software; see the source for copying conditions.There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Error: expected a number

and adding a port number after the option

AIS-catcher -d 00000111 -p 25 -gr tuner 32.8 rtlagc off -u 127.0.0.1 -u -q -N 8100 FILE /home/pi/AIS-catcher/stat.bin BACKUP 60 -N REUSE_PORT 8100 -N LAT 50.12345 LON -1.54321 &

gives this

AIS-catcher (build Jan 10 2023) v0.42 (C) Copyright 2021-2023 jvde-github and other contributors This is free software; see the source for copying conditions.There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Error: unknown switch "8100"

Am I doing something wrong? If so please correcnt my interpretation of your REUSE option by correcting my startup options string.

thanks Roger

[jvde-github]

Ah, sorry, was a bit imprecise and quick in my reply, the full discussion is here: https://github.com/jvde-github/AIS-catcher/issues/50

The flag is -N REUSE_PORT on or -N REUSE_PORT off (default)

So, you need -N REUSE_PORT on.

I see that the error message can also be a bit more prescriptive.

Try this one:

AIS-catcher -N 8100 FILE /home/pi/AIS-catcher/stat.bin BACKUP 60 REUSE_PORT on LAT 50.0 LON -1.5

ps I would edit the post and remove any port numbers and lat/lon that are irrelevant and if sensitive

[g7ruh]

Jasper,

all working now. Now why did I not think to use On and OFF, well I will try that next time.

I amended LAT and LON to 12345 and 54321 before I posted and have removed port info. The Lat and Lon in the post is in the middle of the English Channel between my location and Cherbourg. Thanks.

Roger

[jvde-github]

Suggest we close this issue and open a new one for future issues if you are ok? Thanks Roger for testing and using the program.

Jasper

ps The port is still in the AIS-catcher output.