Store tar1090 data for replay

[harpermann]

We would like to have ADS-B position replays available.

• Can they be stored from local tar1090 process running?
• Possibly instead or in addition, write up to FlightAware or FlightRadar24
• Maybe check with either service if that works for them and how to do it properly
• Best if replays are instantly available but may be restricted in how the records are written if not immediately

[eastham]

I just forked a tar1090 branch where we can initially investigate this, and invited you. If you have a tar1090 instance running, could you see if the /?replay url has that 30 minute lag I mentioned, or is it always up to date? My raspi is being used for other things at the moment...

On Fri, Mar 29, 2024 at 10:11 PM Harper Mann @.***> wrote:

Assigned #5 https://github.com/eastham/adsb_actions/issues/5 to @eastham https://github.com/eastham.

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#event-12299657732, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC4UPHXGYULQ4T7ZI2EI6LY2ZCQRAVCNFSM6AAAAABFPIQQX6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSGI4TSNRVG43TGMQ . You are receiving this because you were assigned.Message ID: @.***>

[harpermann]

Sure, will doHarper Mann510-599-2075 cellOn Mar 30, 2024, at 9:15 AM, eastham @.***> wrote: I just forked a tar1090 instance where we can initially investigate this,

and invited you. If you have a tar1090 instance running, could you see if

the /?replay url has that 30 minute lag I mentioned, or is it always up to

date? My raspi is being used for other things at the moment...

On Fri, Mar 29, 2024 at 10:11 PM Harper Mann @.***>

wrote:

Assigned #5 https://github.com/eastham/adsb_actions/issues/5 to @eastham

https://github.com/eastham.

—

Reply to this email directly, view it on GitHub

https://github.com/eastham/adsb_actions/issues/5#event-12299657732, or

unsubscribe

https://github.com/notifications/unsubscribe-auth/AAC4UPHXGYULQ4T7ZI2EI6LY2ZCQRAVCNFSM6AAAAABFPIQQX6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSGI4TSNRVG43TGMQ

.

You are receiving this because you were assigned.Message ID:

@.***>

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were assigned.Message ID: @.***>

[harpermann]

Replay setup is in the README.md under the destroy SD card section. Here is another howto: https://wiki.oarc.uk/flight:adsb-server-writeup. Not working yet on mine but working on it. Looks like it needs nginx, not default lighthttp. It creates files,

-rw-r--r-- 1 readsb nogroup 25 Mar 30 10:26 blob_65.zstl -rw-r--r-- 1 readsb nogroup 25 Mar 30 10:26 blob_66.zstl -rw-r--r-- 1 readsb nogroup 25 Mar 30 10:27 blob_67.zstl root@rpi4:/var/globe_history/internal_state#

But haven't figure out how to display them. Switching to nginx

[eastham]

Hmm, all I did was turn on the destroy sd card thing and it worked, I think. Definitely nothing complicated needed.

On Sat, Mar 30, 2024, 10:29 AM Harper Mann @.***> wrote:

Replay setup is in the README.md under the destroy SD card section. Here is another howto: https://wiki.oarc.uk/flight:adsb-server-writeup. Not working yet on mine but working on it. Looks like it needs nginx, not default lighthttp. It creates files,

-rw-r--r-- 1 readsb nogroup 25 Mar 30 10:26 blob_65.zstl -rw-r--r-- 1 readsb nogroup 25 Mar 30 10:26 blob_66.zstl -rw-r--r-- 1 readsb nogroup 25 Mar 30 10:27 blob_67.zstl @.***:/var/globe_history/internal_state#

But haven't figure out how to display them. Switching to nginx

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2028337770, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC4UPCBF6QDN5OCEEFTJK3Y23Y6JAVCNFSM6AAAAABFPIQQX6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRYGMZTONZXGA . You are receiving this because you were mentioned.Message ID: @.***>

[harpermann]

I got it working and there is no lag. Also, not immediately filling up the micro-sd. May have taken a complicated route but I'll document.
BTW: we're leaving Monday for a 3 week RV trip to see the big eclipse. Running around a bit the next two days.

[eastham]

I think what we agreed on for this one: use the "Search" window on the right hand side of the display if we need to pull up short-term history for a given aircraft. This seems to work fine. So all that's left here is to add it to the training for people working in the tower.

[harpermann]

From a discussion with HP, we will create a script to check time and / or new files in the tar1090 save directory and put status into the Linux Node Exporter so we can record problems and post alerts as needed. Need to create a rule in Prom after we get the info in the dB.

[eastham]

Would it be easier for the script that monitors the files to also be a native prom client?

On Tue, Jul 16, 2024 at 12:20 PM Harper Mann @.***> wrote:

From a discussion with HP, we will create a script to check time and / or new files in the tar1090 save directory and put status into the Linux Node Exporter so we can record problems and post alerts as needed. Need to create a rule in Prom after we get the info in the dB.

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231670901, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC4UPFE7NNBXZGQPO5LVO3ZMVXBRAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGY3TAOJQGE . You are receiving this because you were mentioned.Message ID: @.***>

[harpermann]

Good thought. Here is a filestat exporter I'm setting up on my tar1090 dB directory to verify it works. It'll allow an alert if files haven't been updated in a some setpoint time. https://github.com/michael-doubez/filestat_exporter. Given the priority of these files, we should back them up, probably to google drive or something. Do we have drive mounted for other things?

Harper Mann 510-599-2075 mobile

On Tue, Jul 16, 2024 at 12:31 PM Paul Eastham @.***> wrote:

Would it be easier for the script that monitors the files to also be a native prom client?

On Tue, Jul 16, 2024 at 12:20 PM Harper Mann @.***> wrote:

From a discussion with HP, we will create a script to check time and / or new files in the tar1090 save directory and put status into the Linux Node Exporter so we can record problems and post alerts as needed. Need to create a rule in Prom after we get the info in the dB.

— Reply to this email directly, view it on GitHub < https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231670901>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAC4UPFE7NNBXZGQPO5LVO3ZMVXBRAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGY3TAOJQGE>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231686367, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATOAZBSRI3BVEM5KZ4SWQTZMVYHXAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGY4DMMZWG4 . You are receiving this because you were assigned.Message ID: @.***>

[harpermann]

Oh come to think of it, we need gdrive to store airrband recordings.

Harper Mann 510-599-2075 mobile

On Tue, Jul 16, 2024 at 1:10 PM Harper Mann @.***> wrote:

Good thought. Here is a filestat exporter I'm setting up on my tar1090 dB directory to verify it works. It'll allow an alert if files haven't been updated in a some setpoint time. https://github.com/michael-doubez/filestat_exporter. Given the priority of these files, we should back them up, probably to google drive or something. Do we have drive mounted for other things?

Harper Mann 510-599-2075 mobile

On Tue, Jul 16, 2024 at 12:31 PM Paul Eastham @.***> wrote:

Would it be easier for the script that monitors the files to also be a native prom client?

On Tue, Jul 16, 2024 at 12:20 PM Harper Mann @.***> wrote:

From a discussion with HP, we will create a script to check time and / or new files in the tar1090 save directory and put status into the Linux Node Exporter so we can record problems and post alerts as needed. Need to create a rule in Prom after we get the info in the dB.

— Reply to this email directly, view it on GitHub < https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231670901>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAC4UPFE7NNBXZGQPO5LVO3ZMVXBRAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGY3TAOJQGE>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231686367, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATOAZBSRI3BVEM5KZ4SWQTZMVYHXAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGY4DMMZWG4 . You are receiving this because you were assigned.Message ID: @.***>

[eastham]

Hmm, I'm not sure if that exporter will work, since the files that are being written to are constantly changing (it starts writing a new set of files every day).

How about something like this? https://www.robustperception.io/monitoring-directory-sizes-with-the-textfile-collector/

Regarding storage / backups, that was partly my motivation for having the two redundant ADS-B boxes -- they will have their own (slightly-different) copy of the data, should be close enough. gdrive backup would be great too -- unless you know of an easy way to do it though, I'm worried we have more work ahead of us than calendar time...

On Tue, Jul 16, 2024 at 1:11 PM Harper Mann @.***> wrote:

Oh come to think of it, we need gdrive to store airrband recordings.

Harper Mann 510-599-2075 mobile

On Tue, Jul 16, 2024 at 1:10 PM Harper Mann @.***> wrote:

Good thought. Here is a filestat exporter I'm setting up on my tar1090 dB directory to verify it works. It'll allow an alert if files haven't been updated in a some setpoint time. https://github.com/michael-doubez/filestat_exporter. Given the priority of these files, we should back them up, probably to google drive or something. Do we have drive mounted for other things?

Harper Mann 510-599-2075 mobile

On Tue, Jul 16, 2024 at 12:31 PM Paul Eastham @.***> wrote:

Would it be easier for the script that monitors the files to also be a native prom client?

On Tue, Jul 16, 2024 at 12:20 PM Harper Mann @.***> wrote:

From a discussion with HP, we will create a script to check time and / or new files in the tar1090 save directory and put status into the Linux Node Exporter so we can record problems and post alerts as needed. Need to create a rule in Prom after we get the info in the dB.

— Reply to this email directly, view it on GitHub <

https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231670901>,

or unsubscribe <

https://github.com/notifications/unsubscribe-auth/AAC4UPFE7NNBXZGQPO5LVO3ZMVXBRAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGY3TAOJQGE>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub < https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231686367>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AATOAZBSRI3BVEM5KZ4SWQTZMVYHXAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRGY4DMMZWG4>

. You are receiving this because you were assigned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231746392, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC4UPG2P3TOK4PTZZNUSSTZMV477AVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRG42DMMZZGI . You are receiving this because you were mentioned.Message ID: @.***>

[harpermann]

I think filestats is easier than du to alert from and I got it working. Here's a graph of 2 files in /var/run/tar1090 whose names don't change. The plot is seconds old. We can alert on > 90 seconds or longer.

[image: image.png]

Harper Mann 510-599-2075 mobile

[eastham]

(image doesn't seem to attach via github)

[harpermann]

[harpermann]

Yep, email image no but paste yes

[eastham]

Ok, I see what you are doing. Makes sense! Not sure exactly what that file does, but hopefully it is a good proxy for health...

On Tue, Jul 16, 2024 at 3:48 PM Harper Mann @.***> wrote:

Yep, email image no but paste yes

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231944460, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC4UPCATQM3CQIH2JL74OTZMWPMTAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRHE2DINBWGA . You are receiving this because you were mentioned.Message ID: @.***>

[harpermann]

We will need to move the only scanner to the fail-over host when it fail happens. It's fine to write local files but if the whole ssd borks, we'd lose them, so I want to copy to gdrive.

Gdrive works fine. Here's how I did it with rclone. https://ostechnix.com/how-to-mount-google-drive-locally-as-virtual-file-system-in-linux/

It's on my @. google account. I'll try it on @. We'd need to talk directory size and likely need to purchase some unless the buringman.org account has space already.

We could write directly to gdrive with ffmpeg for recording files. Do you think our internet is reliable enough for that? The gdrive can be mounted from more than one host (of course) so we can fail-over-easy.

Otherwise, we could have ffmpeg write to a local directory and run rsync on a cron to copy the files to gdrive, verify, then purge the local recording to keep from running out of local disk space.

Harper Mann 510-599-2075 mobile

On Tue, Jul 16, 2024 at 4:03 PM Paul Eastham @.***> wrote:

Ok, I see what you are doing. Makes sense! Not sure exactly what that file does, but hopefully it is a good proxy for health...

On Tue, Jul 16, 2024 at 3:48 PM Harper Mann @.***> wrote:

Yep, email image no but paste yes

— Reply to this email directly, view it on GitHub < https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231944460>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/AAC4UPCATQM3CQIH2JL74OTZMWPMTAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRHE2DINBWGA>

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2231956431, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATOAZFZMXUBB7D66YPXPE3ZMWRCLAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRHE2TMNBTGE . You are receiving this because you were assigned.Message ID: @.***>

[eastham]

Cool, I didn't know you could mount google drive like that. The burningman.org account should have plenty of space -- looks like there is no pre-set limit, and I just uploaded that 13 GB SD Card image with no problem...

Regarding writing directly to it: even if the internet is fast, I am not sure I'd want to rely on Google Drive for reliable / low-latency writes (I worked with some of the people who built that thing -- it wasn't pretty). The danger is that it wedges the whole thing and makes it unusable or unstable.

I'd be fine manually copying this stuff over every few evenings, but if you feel rsync won't be too painful to get going, that could work too. Definitely would want to do it off-hours. You're the expert here...

Message ID: @.***>

[harpermann]

I came to the same thought; direct to gdrive is less reliable. Let’s write airband recordings local then rsync on cron. Rsync is simple.  😎. Let’s check it but I don’t think we’ll need to wait for evenings. It’s not that much data we’re talking each recording is about 55 mb per hour.  I’ll try adding rsync to the record script so when it closes the current recording on the hour, it’ll fork rsync to pick up the newest file. Never more than an hour without backup. Harper Mann510-599-2075 cellOn Jul 17, 2024, at 11:04 AM, Paul Eastham @.***> wrote: Cool, I didn't know you could mount google drive like that. The burningman.org account should have plenty of space -- looks like there is no pre-set limit, and I just uploaded that 13 GB SD Card image with no problem...

Regarding writing directly to it: even if the internet is fast, I am not sure I'd want to rely on Google Drive for reliable / low-latency writes (I worked with some of the people who built that thing -- it wasn't pretty). The danger is that it wedges the whole thing and makes it unusable or unstable.

I'd be fine manually copying this stuff over every few evenings, but if you feel rsync won't be too painful to get going, that could work too. Definitely would want to do it off-hours. You're the expert here...

Message ID: @.***>

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were assigned.Message ID: @.***>

[harpermann]

Hit a snag. I don't have access to https://console.cloud.google.com/apis/ on the burningman.org account which is where tokens are set up for access from the command line to gdrive. Do we have a service account or some such that has access to the APIs? I can add enough storage on my regular harper.mann@gmail.com google account if it's complicated to use an org one.

[eastham]

Hmm, yeah, I don't have access either. How much storage do you think we need all told? The ADS-B data is only 150MB.

[harpermann]

The Airband recording data is running 55mb / hour so like 1.5 gb / day and something like 10 gb for 7 days. Not that much. Maybe 30 gb would cover everything?

Harper Mann 510-599-2075 mobile

On Thu, Jul 18, 2024 at 2:57 PM Paul Eastham @.***> wrote:

Hmm, yeah, I don't have access either. How much storage do you think we need all told? The ADS-B data is only 150MB.

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2237690054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATOAZGZQKMYUEO7AONVAXTZNA26BAVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZXGY4TAMBVGQ . You are receiving this because you were assigned.Message ID: @.***>

[eastham]

Could be half that, we only need to record 12 hours a day when the airport is open. Regardless, I've got that much room in my personal account, do you want to use that?

[harpermann]

Either way, yours or mine. Let's set it up on the production host next Weds when we meet up.

Harper Mann 510-599-2075 mobile

On Thu, Jul 18, 2024 at 3:22 PM Paul Eastham @.***> wrote:

Could be half that, we only need to record 12 hours a day when the airport is open. Regardless, I've got that much room in my personal account, do you want to use that?

— Reply to this email directly, view it on GitHub https://github.com/eastham/adsb_actions/issues/5#issuecomment-2237721453, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATOAZA7XZLRFD4Z54Z5UZDZNA533AVCNFSM6AAAAABK7GRCS6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZXG4ZDCNBVGM . You are receiving this because you were assigned.Message ID: @.***>