Testing Notes 2

[andrewfer000]

This is a new issue to start fresh from Testing Notes.

[markjfine]

Starting this new issue with: Find out why/how Discogs is 403-ing search requests via Cloudflare. It's not being done by IP or User-Agent. Also, find out why logos aren't showing right after being downloaded when not cached in stationLogos.json - noticed this after Discogs put the kabosh on automated cover art.

[markjfine]

While investigating the logo issue, I found a station that used 'jpeg' instead of 'jpg' for album art. The regex was only looking for jpg, png, and txt. So I knocked that one out straight away. Still investigating the other part tho.

[markjfine]

I've determined that the issue of why the logos don't show when downloaded at the start has to do with the XHDR going to 0 with no logo nor a cover downloaded yet. I'll add logic that automatically applies and displays the logo if it's downloaded, but there's no image showing at all, regardless of the XHDR setting. Could be tricky given the current way the images display.

[markjfine]

Fixed.

[andrewfer000]

Awesome! I'll test it out tomorrow.

[markjfine]

Trying to work TODO 3.a and a precursor to 4.a but reception is not cooperating. Dropping a lot of LOT and XHDR messages. This is like trying to work a radio in a SCIF. If I can't resolve my reception issues I may put everything on hold until I'm back in Virginia and I have my good antennas. This is just too painful.

[markjfine]

After a thoroughly frustrating morning, I think I got this logo thing sorted. Also managed to fix an annoying bug where the covers would show prematurely.

[andrewfer000]

I still need to do some testing, but I think there is a problem with album art that makes it take awhile for them to pop up or not pop up at all when the next song starts. I need to do more testing. Will leave an update tomorrow.

[markjfine]

If you do, you might want to un-comment some of the print statements I've added that show when covers and logos are received, when the LOT messages occur, as well as when the XHDR changes. This way you can see the events as they happen in a terminal window in real time instead of having to review a log file, well after the fact.

So you know what you'd be looking at:

1. The LOT messages are when files are received, including the travel/nav text and image/overlay files.
2. The XHDR messages show what's supposed to be displaying (cover or logo), and the LOT number of the file (logos are always a LOT number of -1 in XHDRs, regardless of the logo's actual lot number at the beginning of the file name).

It's possible that I may have missed a condition where the cover may be delayed, but have seen where covers are received before the XHDR changes while a different cover is showing, while the XHDR says a logo should be displaying, and well after the XHDR has changed to the correct LOT number for the cover when no cover existed.

In the first two cases, it immediately changes to the cover when the XHDR changes using the correct LOT number. However, if the XHDR changes to the correct LOT number before the cover has actually downloaded, it won't show until the station sends another cover (LOT message).

The delay you're seeing may be because:

1. The station hadn't sent the cover or failed to switch the XHDR away from the logo (or previous cover) in time - have seen those instances happen a lot.
2. nrsc5 had a problem with the cover file and threw it out - I assume this is a possibility. The app also has some error checking based on file size against what the LOT message says it should be.
3. nrsc5 didn't echo the appropriate LOT or XHDR status message, either due to a signal dropout or the station failed to send it. When that happens the app just thinks it never happened.

I have seen events 1 & 3 occur yesterday many times, which was frustrating the hell out of me - I assume 2 is additionally possible due to poor reception here. That's what prompted me to start adding the LOT number checks - to prevent showing new covers early, or old covers after the XHDR switched.

Previously, all of the gui apps just detected the XHDR switch to a cover and displayed whatever the last received cover was - not really the way it's supposed to work, and why I want to split cover and logo handling completely. It's just too messy.

This will also allow me to cache covers received for other streams, like I'm doing for logos now, as well as reuse covers in case the LOT numbers change. So if you happen to change streams it will show the appropriate cover and not the logo, if that's what's playing and the cover is available. Opens up a whole lot of other possibilities for reorganizing the cover art by station and the actual filename. Will also allow me to index the LOT number metadata for reference so the aas directory doesn't constantly fill up with duplicate images that have to be pruned.

[andrewfer000]

Okay so I figured out what was wrong and it's the stupidest thing ever. So, when I pull down a new version of nrsc5-dui from the repo, I rename the old version do nrsc5-dui-oldX and I extract the old one to the new location. Now due to some strange file system stuff, I need to restart my terminal for it to notice changed files (or else it references the old ones). Once I noticed that the problem was caused by that I restarted the terminal and the application and it seems like everything is working now.

This is a great fix! It's too bad the album art downloads from Discogs does not work anymore.

[markjfine]

How are you pulling them down? In theory, you should just need to go to the root of where nrsc5-dui.py is then run a git pull of the main branch to update it. The versioning is kept intact, so you can always back the code out to a point if there's a problem, just like any other cvs.

I think the command is something like: 'git pull origin main'

[andrewfer000]

I download the zip file and extract it to my desktop after renaming the old version. Then I copy the new folder to my desktop. After that I copy my AAS, Configs, and Maps over to the new version and that's it.

I like to keep the old versions in case something goes wrong.

[markjfine]

That's a lot of work. I run incremental backups every hour so if something goes wrong with the cfg, aas, or other data I can easily recover.

[markjfine]

btw: cheers for the reddit post. I don't do reddit, personally, but It also got picked up by rtl-sdr.com. Seems like it's getting a good share of viz.

[markjfine]

Note: Discogs uses Cloudflare for their content delivery. Apparently they have this new super duper bot buster. I have a feeling automated cover downloads is going to be a nice idea that can't be fully implemented because of things like super bot busters. Again, from a site security aspect and knowing what nasty things are out there, I can't blame them for doing it. I'd do it in a heartbeat.

Worthwhile to note: I think what it's doing may be similar to how WordFence determines it's a bot or not. If it just scrapes the html file (what this is doing), it's a bot. If it pulls down the associated css, js, and images, it's a browser. They don't seem to give a monkey's about the user-agent (and I block an f-ton of those). I'm just wondering if it's worth it to basically build a browser that caches all that with no display just to get past the bot buster. Would mean creating a tmp directory to save all that nonsense to, then deleting the piece parts when I was done with it. Seems like a wasteful endeavor for very little payoff.

[andrewfer000]

That's true. It's possible and it's a nice feature but it's not super necessary since the broadcaster usually supplies the album art anyway. That feature can be put on the back burner for now and maybe focusing on the smaller Quality of Life stuff should come first. I'm not sure if you noticed, but this program has been getting more attention recently. That's because I shared it on Reddit. I'm hoping this gets us some help in testing and implementing new features as time goes on.

[markjfine]

Looking at MusicBrainz as a possible replacement for Discogs. They also have an API that generates json, so I don't have to go through some ugly kabuki dance parsing web pages. ...and there's a Python package already written for it.

[markjfine]

Tempted to try and implement this now, but I know it'll be painful because of reception issues.

[andrewfer000]

I have good reception so I can do testing. MusicBrainz is a good service since it's like a "Wikipedia for music" and I'm assuming the information is free (as in Wikipedia free) and with a real API things would be more stable and controllable.

I feel like after the update you just did for the station art/album art makes the program feel so much more complete and better.

Also, I tested the program in the car today and let's just say there may be some problems but it might just had to do with the bad signals, my crap R820D based rtl-sdr, and the fact the program was trying to reach Discogs without an internet connection which may have lead to the program freezing I was experiencing.

[markjfine]

Think I'm going to disable all of the Discog stuff it until I get it re-wired for MB. Would rather not get a complaint that it doesn't work.

[markjfine]

Went ahead and added the MusicBrainz stuff as much as possible.

Had trouble with the part that actually downloads the image and saves it so used the MB URL in the old save routine.

Once I get that last part fixed, both urllib3 and OpenSSL will no longer be needed.

However, will still have to pip3 install musicbrainzngs for the new stuff to work.

[andrewfer000]

I like the new album art service so much better. You never cease to amaze me with your skill. Thank you for doing so much work for this project and giving it new life and features. I am super grateful for it.

Also, If you don't mind could you modify the console output to send a warning if the SDR gets disconnected. It would be a huge help when I'm testing different devices and cables.

[markjfine]

Cheers. With me it's really more determination and luck than skill.

The disconnect should turn the pilot red when nrsc5 generates a lost device message. I've noticed that sometimes these radios get locked up for no reason. Unfortunately there's no real good way to detect those.

[andrewfer000]

Hey so after doing a lot of testing today I learned a few things,

1. My SDR or Laptop's USB port is complete shit.
2. The album art download system needs a little tweaking. It cannot find some songs that I know should be in the DB since they are older and even if album art is not available it does not always display the one downloaded from the station.
3. Since Musicbrainz is a database and has a real API for python is there a way to gather the information about the album and genre for a song when the station does not provide it just from the song title and artist?

[andrewfer000]

Hey so I think I found a pattern, the downloaded album art only seems to work when the song was released as a single. I believe the best way to fix this is to get the album name from the DB for the song and use that version of the art. I hope that's possible but I'm not sure.

Edit: Yikes, Looking through the Musicbrainz database is difficult. Is there just a way to search a song + artist and return the first album the song showed up on? Like dang this is complicated!

Also I have seemed fo find an issue on the radio station's side. For example, they put the artist as "Blink 182" when it is "Blink-182" in the database (and that's how the band name is spelled). I think things like that are also causing issues with the Album art down-loader.

[markjfine]

The search parameters need to be tweaked. I just noticed a few things in the search I'm doing, which is different than the way the web site does it. Am currently looking at that while I rewicker the download part, which I'll post in a few. Right now it's focusing on singles, as you surmised. I thought I was looking at all releases that contained the title, which apparently is not the case.

Yes, I can pull the other info easily. It's returned within the json that I get.

[andrewfer000]

That's awesome! I hope everything works in your favor, Now I understand how great yet much of a pain these systems and databases can be

Tbh I find it kind of funny how this program written in python and uses an rtl-sdr works better than most $30,000+ cars with HD Radios. That just puts a smile on my face xD.

[andrewfer000]

so I tested your latest code and its still mostly the same. I know that this feature is going to take a lot of time and testing to prefect.

That being said, I think it would be smart to try and get the album name first (if not provided by the station) and use that as part of the search if possible. Obviously the program does not have eyes of it's own so it's not going to be prefect but something is better than nothing at this point in the feature's development.

[markjfine]

I know, I just uploaded a better query. That one just removed the urllib3 nonsense. Getting the album name is trickier than you think... found a better way.

[markjfine]

So, next thing I'm going to do is clean this puppy up. It's now got more commented code than operational code.

[markjfine]

I should also add that you should delete the covers you don't want to keep, because the search routine won't automatically replace them if they're cached like that.

[markjfine]

Still tweaking the routine even further. Seems there are conditions where it bails out on an error that I'm not properly trapping.

Scrolling back, it seems you've also hit on another issue I've been seeing, and that may be related to how MB wants to see everything in UTF-8. So non-alphanumeric characters like -, #, and ' may be getting misread. So I need to either .encode() the search string or do search-and-replace things on those characters which will be a major pain in the arse.

[markjfine]

Added a fix. Found out MB doesn't always include the piece parts I'm looking for, so had to add checks to make sure the routine continued to the end, as well as not using one big error trap for the MB queries.

[markjfine]

Word of caution tho: Am seeing incorrect meta data for Compliations that are listed as Albums. So stupid K-Tel type crap is getting picked up in my routine as a valid cover image. Nothing I can do about hosed meta data except report it.

[andrewfer000]

That's great! Like I said, something is better than nothing when it comes to this type of feature. Unfortunately I cannot test right now because I am at the shore until tomorrow and even with my Amplified FM/TV antenna I cannot pick anything up well enough to synchronize. When I get home I'll test with ALT-104.5 (WRFF) and 106.1 The Breeze (WSIX) and I'll let you know my results.

[markjfine]

Nice day today... enjoy.

[andrewfer000]

So since I cannot get any radio stations here and it's too cold to go to any beach attractions today (55 and overcast :[ ) I decided to just review your commit. I'm just curious where does the score come from? Is it a type rating used by Musicbrainz? Also how come the score has to be 90 or greater.

[andrewfer000]

Also, if you can figure this program out then you can do testing without having a real radio station since this program can generate a signal which I think you can pipe into rtl_tcp. Personally, I never used GNU Radio before, but if you can figure it out it might be super useful - https://github.com/argilo/gr-nrsc5

[markjfine]

I think MB does some correlation between what you entered and the result and comes up with a score. I picked 90 just as a limit because I wanted it fairly close. Most of the matches it ends up selecting are 100%, so...

I've never really gotten gnuradio to work properly. But in theory you should be able to use anyone's IP address as a surrogate rtl_tcp in nrsc5-dui provided your internet is fast enough. I've not tried it but I really should given the situation here.

[markjfine]

So much for theory. I looked on the SpyServer map and pulled up some remote rtl-sdrs, entered the IPs both with and without the :5555 port (think it defaults to :1234 without) and nrsc5 wasn't able to connect to any of them. Must be a way to do this though.

[markjfine]

Took the requirements for urllib3 and OpenSSL too soon. Need to put it all back... lol

Working on a prototype capability to enter the URL for a station logo image if you can find it online. It then downloads it to the aas directory and saves it as the default logo:

[andrewfer000]

Epic feature! I think once you get this feature in and finish up cleaning the code you can probably do that 2.0 release that you have been working towards. Edit: just checked the about dialog and noticed it has been updated from 1.2.x to 2.1.0.

I think one thing that could be important for the album art is to set it so if MB fails it will fall back to the station's provided album art. So far it's been working well except for a few songs but if the fallback can be setup then that won't be an issue (at least to me).

[andrewfer000]

I think an interesting way to check if an album art is downloading or not is to check if the "Album" field is getting filled, if it's not after the album art download attempt happens and the Album name is not filled from MB it will automatically fall back to what the station sent. (one way you can do this if there is no way to do it already is to use the file's timestamp as the latest file and overwrite the current thing on the artwork display.) Also sometimes when I start a station, MB downloads the album but then the station art gets sent in and overwrites it on the display. Not sure if you noticed that or not. I mean it's rare for the overwrite thing so no biggie.

[markjfine]

Yeah, I just need to package it up to make a formal release. But I think 2.1 final is getting close as long as I stop getting distracted by shiny things.

It already falls back to the station's art by default, but I may put an option that allows you to turn the stations covers off completely (or have it work vice versa, actually). The next step in this is to start building a cache json similar to the station logo file that contains metadata for the downloaded cover art. Then, instead of just looking to see if the file exists, it checks the metadata for the album name and genre if the station hasn't provided it.

The fallback would be if the station provides a cover that hasn't yet been downloaded (or can't be found because MB can be inconsistent) that image will be treated and cached just like a downloaded one, and renamed using the artist-title format. This way it can be reused until an actual download replaces it. The current system saves the file with a lot number, and if the lot number changes for one title, or if another station pushes it, the number of files grows exponentially out of control.

This seems to solve the current problems of consolidating and organizing (and potentially eliminating) station-provided cover art as well as how to retrieve the album and genre from cached files. That json might get very large over time, so I'm weighing alternatives. It would be neat if I could save the metadata within the file itself like a real ID3, which would preclude the need for cache file at all. If the capability to add metadata to images is already in Pillow (hadn't checked) all the better and I can save from having yet another dependency. Then I just need to re-wicker the Album Art tab the way I really want it, and 2.1 will finally be done.

But first, the interface for the URL was rapid-prototyped using a GtkWindow last night to see how it would work - I need to change that into a popup dialog that can run modally (and not pop-behind the main window, which is a real pain in the arse) with an GtkEntry for the URL and two GtkButtons (Ok, Cancel). Might have to actually pull it into Glade in Linux to do that. Til now, I've been doing all the glade interface design by hand in TextEdit in macOS.

[markjfine]

Updated the repo with the station logo thing. Was listening to WPRB last night. They don't have station logos, cover images or track info and felt like it was something sorely needed.

[markjfine]

Added the fallback option in Settings.

[andrewfer000]

The station logo download works but it could be improved since it often goes back to the logo that the station sends over. Also for a few songs/bands, the artwork downloader just does not work at all. I'm going to uncomment your print statements and see what's going on with those or if I can find a common denominator between them.

On an unrelated note, my semester is quickly coming to an end and with both final exams and final projects due I might not be able to review new updates to features as quickly as usual for the next 1-3 weeks. I'll test when I can. but I'm not going to test out every modification like I have been recently.

[markjfine]

Oh! Wasn't aware you were in school. Please, by all means, school comes first. No apologies necessary.

[andrewfer000]

Oh yeah one more thing I wanted to mention is that I feel like the GUI is growing in size, On my desktop/laptop this is not much of an issue but I would like to run this on Linux phones when they become more viable. So maybe the next major feature/change with this program would be to allow resizing and possibly additional layouts if Python and/or GTK permits it.

[markjfine]

The size is actually a function of the base font. It's really tiny on my Linux VM, which has the same screen resolution as my Mac, 2880x1800, but the font is a lot smaller so it's about 1/8th the size.