Enhancement: Support for more retry options on failures & notifications

[sumocomputers]

I am on 0.1.33 using Homebrew on macOS.

I have run into certain issues where locast2tuner fails for some reason, and since my Mac Mini is headless I have no idea whether locast2tuner is running or not. And let me tell you, when you can't provide your wife with reliable Jeopardy recordings, life is slightly less sweet.

For example, in my logs down below, I had an issue with my internal PiHole DNS servers briefly, and locast2tuner just bombed out immediately.

Here is what I am thinking:

• Some sort of option to have locast2tuner retry when it encounters an error. It could be number of retries, a time span, or maybe the option to combine both.

• Some sort of notification mechanism. Ideally when locast2tuner starts and stops. Email is probably a PITA, so maybe something like Pushover, or maybe something even cooler that I don't know about.

• In line with the above, there could be later enhancement with a "heartbeat" added that just sends out a message like "Running", "Stopped", "Starting", for use in something like a live dashboard (I have several iPads throughout the house).

FYI, there are no labels here in GitHub for enhancement, bug, question, etc.

May 09 13:35:06.325 INFO Login succeeded!
May 09 13:35:13.209 INFO Loading stations for Cleveland (cache: true, cache timeout: 3600, days: 30)..
May 09 14:32:13.230 INFO Reloading FCC facilities..
May 09 14:35:13.141 INFO Login token expired: Mutex { is_locked: true, has_waiters: false }
May 09 14:35:13.142 INFO Logging in with myobfuscatedemail@something.com
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: reqwest::Error { kind: Request, url: Url { scheme: "https", username: "", password: None, host: Some(Domain("api.locastnet.org")), port: None, path: "/api/dma", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("dns error", Custom { kind: Other, error: "failed to lookup address information: nodename nor servname provided, or not known" })) }', src/fcc_facilities.rs:112:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Abort trap: 6

[wouterdebie]

I hear you. First of all, how are you running locast2tuner? Using docker or an ubuntu package?

Secondly, it's hard to determine which errors are transient and which are not. In your case, locast2tuner couldn't even login correctly, so I'm not sure what it's supposed to do. To be honest, I don't feel like implementing a lot of restart or notification behavior, since there are other systems that do this much better. E.g. if you run using the ubuntu package, systemd takes care of starting and stopping services, which can also handle restarting on failure (check https://ma.ttias.be/auto-restart-crashed-service-systemd/). The same can be done when running docker images. This also leads to additional systems alerting you when the service is up or down (e.g. nagios). And depending on what your setup is, you can get notifications through email, pushover, sms, or whatever.

[Croq360]

FWIW, the Windows Service solution I'm currently using has some restart parameters set too.

[eiddor]

Agreed with that approach both for full process restart and especially for notifications. I know @wouterdebie has added some retry behavior/error handling inside of the app itself and I'm sure that's always a WIP.

[sumocomputers]

I am using HomeBrew on macOS. I hear you on not wanting to reinvent the wheel.

For at least the restart functionality, would the article linked/Ubuntu apply to macOS via HomeBrew?

If not, I guess I could look at Docker, though I have never really done anything with it. The HomeBrew method seemed very easy and comfortable to me, so I just stuck with it.

[eiddor]

So, I haven't tried the HomeBrew version yet and I'm not really sure how it operates, but I wonder if you can run it as a service as in this guide:

https://thoughtbot.com/blog/starting-and-stopping-background-services-with-homebrew

Docker is easy, too, FWIW. I think you have some options.

[sumocomputers]

So, I haven't tried the HomeBrew version yet and I'm not really sure how it operates, but I wonder if you can run it as a service as in this guide:

https://thoughtbot.com/blog/starting-and-stopping-background-services-with-homebrew

Docker is easy, too, FWIW. I think you have some options.

Thanks for the info. I think I will try to get a Docker version up and running; sounds like it opens up a lot of options

[eiddor]

Thanks for the info. I think I will try to get a Docker version up and running; sounds like it opens up a lot of options

Honestly once you start messing with it a lot of ideas/possibilities will open up in your mind. My entire media ecosystem is running in Docker, like 10-12 containers.

[wouterdebie]

Thanks for the info y'all! locast2dvr included a launchctl script that we could reuse. I'm all for instructions or examples on how to run stuff as a service and do retries and such, as long as there are other systems we use to do those things. That said, I'd be happy to change things around where locast2tuner crashes when it shouldn't (like I fixed a while ago where it would crash if a single stream would crash).

[sumocomputers]

A little off-topic, but I have a 2018 Mac mini with i5 (it runs home automation, Plex, and a few other critical services for the house)

I also have a couple of mid-level Synology NAS (DS1019+, DS1817+).

Just wondered if there is a preference on which one to run Docker?

[eiddor]

I'd say if your NAS has the resources, running Plex/locast2tuner together in Docker on that is ideal, but it really depends on what you want to accomplish and how much you like to tinker/manage.

[sumocomputers]

I'd say if your NAS has the resources, running Plex/locast2tuner together in Docker on that is ideal, but it really depends on what you want to accomplish and how much you like to tinker/manage.

I don't think they have the power, especially for Plex, so maybe I will start with the Mac.

If I do run locast2tuner on Docker, do I also need to run Plex in Docker?

And for this scenario, I want to tinker/manage just enough to record Jeopardy, lol.

[eiddor]

If I do run locast2tuner on Docker, do I also need to run Plex in Docker?

No, not at all. Docker will forward the locast2tuner ports to the container and everything will behave the same.

[eiddor]

I don't want to clutter this issue up anymore, so feel free to ping me directly!

[sumocomputers]

I will close this, since looks like other configurations can accomplish what I need. Thanks @wouterdebie and others for the help!

[wouterdebie]

This is probably a good thread for our Gitter :)