Chanel Scanning seems bugged for Native HDHR ATSC

[CraziFuzzy]

When doing a channel scan from sage for ATSC tuning, there is some issue in the data returned, resulting in a junked up lineup in sage. Looks like it is not detecting and returning actual program information. Example:

17:04:06.271 [SageTVRequestHandler-123:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG SageTVRequestHandler - SageTV sent: 'AUTOINFOSCAN HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner|10'
17:04:06.272 [SageTVRequestHandler-123:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] INFO  HDHRNativeCaptureDevice - Capture device is was already locked.
17:04:06.272 [SageTVRequestHandler-123:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/channel' value: 'auto:213000000' lockKey: '1062706233' sendLength: 49 address: 192.168.99.31
17:04:06.523 [SageTVRequestHandler-123:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/status' value: 'null' lockKey: '0' sendLength: 25 address: 192.168.99.31
17:04:06.525 [SageTVRequestHandler-123:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/streaminfo' value: 'null' lockKey: '0' sendLength: 29 address: 192.168.99.31
17:04:06.776 [SageTVRequestHandler-123:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/streaminfo' value: 'null' lockKey: '0' sendLength: 29 address: 192.168.99.31
17:04:06.777 [SageTVRequestHandler-123:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG SageTVRequestHandler - Replied: '13-0;13-0;13-0;13-0'

I'll be looking into it, just wanted to see if you've done much (or any) testing of ATSC tuners.

[CraziFuzzy]

FWIW, while watching a channel, looking at /tuner0/streaminfo shows the proper subchannels, so it may just be an issue of waiting long enough for the tuner to detect the programs.

[CraziFuzzy]

Found a shortcoming in the debug logging. It looks like there is no way to see what is returned in HDHomeRunControl.getVariable, since it transfers off to the setVariable function with null in the value, and the setVariable writes it's debug at the beginning when value is still null. Might require adding another debug entry that shows the actual returned value when incoming value is null. Without this it will be difficult to determine why the programs are not being parsed.

[enternoescape]

This is loosely modeled after how the HDHomeRun native code does things. When the value is 'null' that actually means get even though the method says set.

[enternoescape]

You do seem to be having a timing issue though based on what I'm seeing there. Based on how things are parsed it's probably seeing something that looks like 0 4 times. The logging could be a little more verbose at that juncture though, but the reason you don't see more is because it thinks things went well. Also you can't always log what's returned by the HDHomeRun because a lot of the message is not plain text.

[enternoescape]

I reviewed the code that gets you to that place and it really does a lot to try and make sure there should be valid data before it even tries to parse what might be present. I generally have avoided logging certain things that might cause very excessive logging in rapid succession. This returns a lot of lines and could be called a lot in a row, but I can see some value in seeing what it returned, so I'll add the logging.

[enternoescape]

I added logging at what should be the right moment. Let me know if you need any help.

[enternoescape]

I probably just addressed the issue entirely with the commit I just made. I couldn't help just fixing this since I am at least 95% sure the reason you saw 0 is because the HDHomeRun really did essentially say the channel number was 13.0.

[CraziFuzzy]

ahhh man... was going to dig into this this afternoon. We'll see how it works.

[enternoescape]

Sorry, I might have stolen some of your fun, but there's still a 5% chance you might still have something to look into. :)

[CraziFuzzy]

hehe.. I still think there might be an issue with checking streaminfo too soon after tuning, because that's the only reason I can see the HDHR returning the 0's in the first place. I'm guessing this may result in channels being missed - but I've gotta get it built and try it out a bit to check and see.

[enternoescape]

It will wait for I believe at least 12 seconds before it gives up on a frequency which is usually more than enough and it can't wait too long because SageTV will terminate the channel scan prematurely. That includes if the channel ends in .0 which is part of the changes I just made. So if the channel ends in .0, it will continue to poll the same as if no channels were returned.

[CraziFuzzy]

well, first build and run of it looks like it's going weirder now. It's not throwing out the .0's, but at least I'm seeing the new logging from streaminfo. It looks like it is retrying the streaminfo when it gets no streaminfo, but as soon as it gets the 0's, it's good. I have to take off, but I'll try to look at it later and see what the change you made actually does.

17:06:57.847 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG SageTVRequestHandler - SageTV sent: 'AUTOINFOSCAN HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner|10'
17:06:57.847 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] INFO  HDHRNativeCaptureDevice - Capture device is was already locked.
17:06:57.848 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/channel' value: 'auto:213000000' lockKey: '1062706421' sendLength: 49 address: 192.168.99.31
17:06:58.100 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/status' value: 'null' lockKey: '0' sendLength: 25 address: 192.168.99.31
17:06:58.103 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/streaminfo' value: 'null' lockKey: '0' sendLength: 29 address: 192.168.99.31
17:06:58.104 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunTuner - streaminfo returned: none

17:06:58.355 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/streaminfo' value: 'null' lockKey: '0' sendLength: 29 address: 192.168.99.31
17:06:58.356 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunTuner - streaminfo returned: none

17:06:58.606 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunControl - key: '/tuner0/streaminfo' value: 'null' lockKey: '0' sendLength: 29 address: 192.168.99.31
17:06:58.607 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG HDHomeRunTuner - streaminfo returned: 3: 0
4: 0
5: 0
6: 0
7: 0
tsid=0x012D

17:06:58.608 [SageTVRequestHandler-135:HDHomeRun HDHR-US Tuner 10173AD8-0 Digital TV Tuner] DEBUG SageTVRequestHandler - Replied: '13-0;13-0;13-0;13-0;13-0'

[enternoescape]

Interesting. I guess we should just change it to check if the channel is 0. I really thought the channel number would have been in there, but actually now I'm not sure why I thought that.

That should be a very easy edit to /src/main/java/opendct/capture/HDHRNativeCaptureDevice.java on line 1406. Just change program.CHANNEL.endsWith(".0") to program.CHANNEL.equals("0")

[CraziFuzzy]

Okay, looks like checking for CHANNEL.equals("0") works out right. First scan seems to work out right on my test system/tuner - going to throw the JAR at my main server and see how it behaves.

[CraziFuzzy]

Okay, so scan in worked - seems to have retried properly and found the right channels - there is another issue, and I'm not sure if it's opendct related or not - but sage is assigning virtual channel numbers for that are equaling the subchannels - not sure what's going on with that. Perhaps the returned values to sage are not as sage expects. I'll do a little more checking.

https://1drv.ms/i/s!AmVc_jqaZwXviSs75JEmgGm9GVXW

[enternoescape]

I've seen things like that happen before. It's either a bug in SageTV or I never actually figured out what it's expecting. The parsing logic on the SageTV side of things looks fairly straight forward and any time I've messed with the returned results, I just get even more strange remappings. At least when you get these kinds of results, you should be able to "Clear all Remapped Channels" and the channels scan should then match the actual channels on the lineup.

[CraziFuzzy]

I'm looking into the scanning logic in the UI (where the scan info is actually processed), and it appears the api is capable of returning a lot more info. It looks like it will support: RF-Maj-Min(Name)BroadcastType For example: 13-13-1(KCOPDT)ATSC;13-13-2(KCOPDT2)ATSC;13-13-3(KCOPDT3)ATSC I'm going to try adding this to it and see how it performs with it's matching to the lineup.

[CraziFuzzy]

Definite progress - that greatly improves matching - 100% match to the lineup for channels that it actually detected - still having some channels not being detected reliably. I've seen a few responses that did get program data, but didn't detect signal, so it was ignored. DEBUG HDHomeRunTuner - streaminfo returned: 3: 11.1 KTTV-DT (no data) but enabling it anyway does tune in the channel, and while tuned, checking streaminfo shows it correctly. This is probably worthy of another retry, but we seem to be causing a ton of retries here waiting for it to not just get tuned but also parsed for stream info, it may make more sense to simply put in a wait after tuning before getting the streaminfo to begin with. Slower scans overall, but even a half second per ATSC channel (adding 35 seconds to a complete scan) may be faster than the repeated retries with their 250msec delay in between them.

[CraziFuzzy]

298 Seems leagues better

[enternoescape]

From looking at the SageTV side, I concluded something similar to what you pushed, but after looking at what you did, I can see I was really close, but for some reason I thought there were other delimiters in there.

[CraziFuzzy]

I just realized that this improved ATSC scanning program should probably be duplicated over to the QAM_HDHOMERUN case as well. I'm not sure I have and ClearQAM channels to check it with, but I might build up a test and see how it works. In fact, while I obviously wouldn't be able to check it, I'm thinking it would work for the DVB modes as well, since I'm guessing they also include a callsign in the stream, so sending that to sage is likely to aid in mapping the scan to the EPG.

[enternoescape]

ATSC is the easiest of the bunch to detect in real time since you will only ever see things you can actually tune into. QAM is being remapped internally so the same virtual channels you use for the CableCARD are the channels you'll get for QAM (you won't see the actual channel-program mapping).

DVB is a different story. That returns data similar to ATSC and I don't have enough experience (or any really) to know if the ATSC approach works well in that area. So far only one person that I know of has actually used DVB with OpenDCT and it was more of a curiosity than serious full time use.

[CraziFuzzy]

My thoughts on QAM were for users who do not have a cablecard to get the vchannel mapping from. I'd think scanning in that instance would work just like ATSC, where it goes frequency by frequency, tuning, and waiting for a lock, then getting the program name and number from the streams. I'd need to play with my spare HDHR to see if this is presented differently than in ATSC mode, but I had just assumed it would present the same data.

[enternoescape]

Ok, I guess I see the point there. It's not that the HDHomeRun doesn't present the same data as ATSC when scanning, it's that it will present a lot of programs that you can't actually stream because they are encrypted or otherwise inaccessible. The only reliable way to tell if they work or not is to try and stream each one which is what offline scanning will do. That scanning if performed live takes more time than SageTV is willing to wait, so it's really not an option. Although I've seen plenty of QAM scans return lots of channels that don't actually stream when scanning using SageTV (not through OpenDCT), so maybe users are already used to the getting lots of mappings that don't actually stream anything.

[enternoescape]

Either way, I'll see if I can adapt the code involved in this piece and take the opportunity to clean things up a bit with channel detection.

[CraziFuzzy]

I think the streaminfo reports whether it is encrypted or not, and I believe HDHomeRunProgram.isTunable() is already checking for that, and that ATSC logic is checking isTunable and throwing out channels that are not. (HDHRNativeCaptureDevice.java:1441)

It may actually be that we might be able to simply combine these cases all into the same code. Instead of selecting the whole sequence based on tuner type, we may be able to simply select in the few places that would be different (mainly, the string-building at the end for return values).

[enternoescape]

What you're describing is mostly what I'm already planning on doing. All of the functionality that you're describing is just for legacy devices. Non-legacy uses the scan from the web interface of the HDHomeRun which makes all of this fuss unimportant.

What I'm trying to say that is sometimes programs don't get flagged in a manner that would make you think that you can't tune into it. For example, FiOS has a bunch of channels just for advertisements (I presume during VOD) and those channels would be included as tunable in the current format even though you will not actually get anything if you try to tune them in. This is because they are identifiable and not encrypted.

[CraziFuzzy]

Thought those might get thrown out by having a (no data) flag on them.

Regarding using the lineup.html page, I find that to be a needless step if we can have the scan done from within the sage UI via the 'legacy' methods. I know I almost never bother going to the web page of the device if I don't need to (but I only have a prime - which as far as i can tell, doesn't use the lineups page in sage, and a HDHR-US, which is only legacy mode anyway).

For me, the biggest problem with depending on the lineups page, is that it is an extra step, and it will always use all tuners the device has, even if they are currently locked.

[enternoescape]

I might agree with you on your points about legacy, but there are a few advantages that you are not considering. Also OpenDCT could kick off the channel scan on the web interface, but those numbers generally don't change unless you move to another house (with a few exceptions), so it only downloads what is currently reported and returns that. It will never actually initiate a channel scan for you on the HDHomeRun since that would definitely cause some unexpected situations for some users in reference to your comment about taking over tuners that are locked. Since the frequencies don't need to be completely re-scanned, this also speeds up the channel scan dramatically.

If it's clear that the user never ran a channel scan from the web interface of the device itself, OpenDCT will do things the legacy way anyway, so for people who likely never knew this option existed, it will still work well. Pulling this information directly from the device was a very highly desired feature by many users, especially in the case of the Prime.

Unless we are going to add a new capture device type to SageTV so that it understands what a virtual channel is to a CableCARD tuner, we can't do a legacy scan the channels on that kind of device in real time; at least not reliably. The SageTV channel scan times out because it spends too much time per channel to see if it can tune it in or not. Even if a channel scan was kicked off on the device itself across 3 tuners, it still takes too long.

When you use HTTP instead of RTP, in many cases you get much better reliability. If you don't perform a channel scan on the device itself, you will not have any HTTP links and will only be able to use RTP. For myself and others this helped tremendously with packets that would otherwise be dropped due to network load conditions.

I can compromise here and since the intent is clear when you set hdhr.always_tune_legacy to true that you are not interested in the web interface, that with the exception of the Prime (when a CableCARD is inserted), it will do a legacy channel scan.

[CraziFuzzy]

I understand how the CableCard can't just do a per-frequency scan from sage, since it only ever gets told to tune to a vchannel - I can also see how people may have wanted to use the lineup page for qam scans if all the legacy scan was returning to sage was the number of the found channel. However, with the tweaks to the info it returns I made in the ATSC legacy scan, where it tells sage the name, the vchannel (if there is one) etc, I think it makes a more elegant solution.

Keeping in mind, I don't even HAVE a non-legacy tuner at this point, and since I have a cablecard, and limited to no ClearQAM scanning, I'm not one to really visualize what direction it should go - I just know from a new user perspective, as much as can be done completely within the SageTV UI the better.

[CraziFuzzy]

FWIW,I also have never done a scan on my cablecard tuner from within sage and had anything useful come in, so I'm not entirely sure what information it is passing back to sage in this case either.

[enternoescape]

I see what you're saying and I agree with the goal of making as much accessible from SageTV as possible. Unless someone actually adds CableCARD support to the SageTV core, it will always be a special case.

Since you have never done a channel scan on the Prime itself, you will not get anything useful within SageTV. The reasons are as described above. It's just not possible with how SageTV does things. The offline channel scanning feature would be your only way to get any useful results.

[CraziFuzzy]

Don't misunderstand, I have done scans no the web page, at at least some point - but if I remember correctly, doing the scan from in sage to it will end up populating the list in sage with junk channels, instead of matching the scan results with what was provided from the EPG. I'll give it a shot again in a bit and see if that is still the case.

[enternoescape]

...and actually offline channel scanning doesn't do much with a Prime since it doesn't know what channels it can even try.

[enternoescape]

I can look into that more, but if I recall sometimes the CableCARD discovers channels that SageTV doesn't know about from the EPG data and that always results in extra channels.

[CraziFuzzy]

I'll get some good info here in an hour or so (when I get home), as I did do a lineups page scan the day before yesterday on the prime, so it should have fresh info to grab, so I'll see what info is really being sent to sage, and how sage is interpreting that. See if we need to massage how the data is presented to clean things up a bit.

[enternoescape]

I'll be mostly unavailable for the rest of the day, but I look forward to your results especially since you have a better grasp of what the STV is doing with the data.

[CraziFuzzy]

So I just did a quick scan from in sage, and I now remember the shortcoming. even though the lineups list on the HDHR has numbers AND callsigns, the callsign is not sent to sage. This results in some things not matching correctly. The example I have, is vchannel 6 on my lineup is KMEX, which is actually broadcast on vchannel 236 (so the EPG from sage brings it in as 6 - KMEX - [236]). Because the HDHR scan only send '6' to sage, it thinks it's a new channel (because when a scan returns just a number, it's assuming it is the physical number), so sage doesn't match it up with the existing channel that is physical 236, logical remapped to 6, callsign KMEX. I think (and I'll try to make up a test build to try this) that if we also send the CALLSIGN with them (in this case, 6(KMEX), it might match it up properly, instead of creating a new channel that isn't mapped to an EPG listing.

[CraziFuzzy]

So, some changes to the code to add the channel name to the return certainly did reduce the number of mismatches - shown as changes in the lineup - from 170-some to 34. However, it did remap a couple channels incorrectly because of strange differences between the ID in the stream and the guide info. example is: vCHAN EPG HDHR 501 HBOHDP HBOHD 502 HBOHD HBOHE What this did, is that since it prefers to match by name than number, it matched the SCANINFO's 501(HBOHD) to the EPG's 502 - HBOHD, which causes sage to remap guide channel HBOHD (the eastern HBO) to physical and logical channel 501. Not sure of a proper way to handle this cross, as sage really does do what would be expected here.

Aside from this, the only other problems I see are really sage side. Sage doesn't check the number for a match if it doesn't find a match for the name (or a few set variations of the name), so stations that simply don't match (HDHR's INDIS doesn't match with the EPGs ID), and are instead created as new stations not attached to any EPG data. This could be changed on the sage end so that if the scan's results don't match a name in the EPG data, but there IS a channel with that number, then match it up that way. Not sure this would cause any issues.

In the end, it doesn't take long to manually match up the channels in error, and it was far less to go through than without the names entirely.

[CraziFuzzy]

If you want to take a look at the change I made, it's here https://github.com/CraziFuzzy/opendct/commit/388fb88cf206ac7d02f859383c3032075b0df58b

Not sure if you want to merge it or not. I feel it probably should be, since providing all the info from the scan we can seems to be the most correct way to do it. If there are problems with how sage deals with that data, it should be dealt with on the sage end.

[CraziFuzzy]

I did look through the parsing logic, and the QAM parsing looked much more appropriate to CableCard use than the 'Other' parsing (which is essentially for DVB, but is not really refined), which is what is being used now. QAM parsing will look through the existing lineup for the scanned channel by name, and if it finds it, will match that. If it doesn't find it's match by name, it will also compare the number, and if that matches to a station that is a zap2it station (StationID >10000), it will match it to that.

I made a test JAR that adds QAM to the results, to force it to use that method, and the results were pretty reliable. The HBO thing above was still an issue, and it had a real problem with Music Choice channels. This is because before it even tries matching things, the QAM tests drop the digit off the end - so all the MC20-MC29 channels are matching to MC2 by name, and then doing some weird things to add an incrementing number to the end to match it from there - really sort of a pain, and an easy fix in sage would be to test for an exact match (physical number AND name) before doing any massaging of the info.

Conclusion, really, is that a new Broadcast type should probably be set and returned in the results. CC or DCT maybe. Technically, some of the issues I'm seeing with using QAM would be issues for actual QAM scans as well, but I think the 'correct' way is to not call the scan results from a cablecard tuner 'QAM'. Calling them CC or DCT will keep things the same as the status quo (because if Sage doesn't recognize the BroadcastType as ATSC, DVB, or QAM, it will use the Other/Blank logic), but specific improvements could be made on the sage end to deal with these CC scans a bit differently (putting the physical number match as a priority above matching to a name or altered name).

[enternoescape]

I like the commit, it makes sense, I merged it. You have helped me completely wrap my head around what the STV is doing. Honestly, I really wish that channel processing was normal code because the loops and branch logic being performed in the STV would be much better organized as code and my best guess as to why that's in the STV is because when SageTV was closed source, they wanted people to be able to customize the channel scanning, which is now unimportant.

[CraziFuzzy]

I did more work on it after that commit to add the broadcast type to the output as well. I've also tweaked the STV code to do a number-first parsing of the DCT channels. It brought in my scan with both of these in place. I'll try to get it cleaned up and posted as a change to the STV. sometime in the next couple days (no idea with the holidays what my time will look like).

If you want to make a java based parser, that's entirely possible to do - the issue, however - is that the scans are typically a slow process, that needs good progress monitoring - this is really the main reason it resides in the STV. That said, whether its done in java or STV, it's still a convoluted mess of logic that it goes through to try to match things up. DCT results are actually the simplest, because you can be pretty certain that matching by number along is probably correct. ATSC is second best, as the callsigns are an official item, pretty sure to match the EPG data. DVB is a mystery to me, and QAM is a convoluted mess that has so many checks and rechecks and comparisons to deal with a bunch of edge cases to try to find the right channel. Whether done in Java or STV, the entire section of code could use a reimagining and rewrite from top to bottom, better defining the various steps along the way, and tweaking the various steps based on the broadcast type.