Closed enternoescape closed 7 years ago
The fourth argument in the API constructor is different in each one. When it's true, it ends up being an RPC to the server and is executed in the server context instead. It's magic. :-)
Jeff Kardatzke Sent from my Android
On Aug 10, 2016 7:51 PM, "Joseph Shuttlesworth" notifications@github.com wrote:
It's unclear to me how IOUtils.writeStringToFile(f, s) does something different between WriteStringToLocalFile() and WriteStringToFile() in the API. Unless I'm missing a critical detail, they are using the exact same functions, yet one is expected to be from the server's file system and the other is expected to be from the client's file system.
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/161#issuecomment-239061028, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDE_ZV7JMXEZ9cz89pLE6XWkex5Gdks5qeo4rgaJpZM4JVuyx .
It is similar to the other File functions that have a basic one, that always accesses the file on the server filesystem, and a local one that accesses the file on the local filesystem (which means, when accessed in a client context - meaning a full client i.e. SageTVClient.exe) it will be the client's filesystem. On a Server based client like miniclient, placeshifter, or the server's local UI context, the two API calls should do essentially the same thing. For EPG purposes, I don't think you'd ever have a reason to use the Local variant, but for UI functions, it very often requires use of local and server file access for various things.
@Narflex Oh right. I think I actually knew that and forgot about it. :)
I think we might have a small issue with the fact that there isn't any checking in the new API functions to ensure that the file being written is one we should be writing to (perhaps we could just restrict it to writing to folders SageTV is importing and the actual SageTV program directory). This is especially concerning because by default in the Windows service and Linux, SageTV has very high privileges. If used poorly, someone could effectively overwrite a critical file, then reboot into a broken OS.
Do any of the file access API functions check this?
We did that for Google Fiber to secure the box, but the level we went to would not be desirable for the consumer product.
SageTV has always assumed your local network is secure. There's always been the ExecuteProcess API call which allows you to do essentially anything you want to a system. So this new one isn't opening up any holes that were not already there.
Jeff Kardatzke Sent from my Android
On Aug 13, 2016 6:09 PM, "Christopher Piper" notifications@github.com wrote:
Do any of the file access API functions check this?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/161#issuecomment-239650166, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDOatOeiBCLZAtQuSlZe1tCjdUFOVks5qfmq_gaJpZM4JVuyx .
Your changes gave me some inspiration on how I should be implementing certain parts of the SD EPG solution, so even though you were concerned that it would be disruptive, it has actually been very helpful. :)
I thought I would provide some screenshots of what I have designed in studio so far in case I'm going in the wrong direction. All errors are handled similar to how @Narflex did the new error handling in WarlockRipper.
Loosely modeled after the Zipcode screen for WarlockRipper.
One more.
The message in this one is ok and I know you're probably going to suggest that it checks if the lineup is in use by any other lineup first before it will even allow you to remove it, but I can think of at least one situation that would be very annoying to be in if we don't let you remove a lineup from your account that is currently in use by other capture devices and nothing will stop you from removing the lineup via another SageTV server.
We could have it save in use lineup SD URIs to Sage.properties and re-add them if they get removed and a capture device actually needs them. I would probably have that generate a warning so you know something like that just happened and an error if the lineup couldn't be re-added.
I would try to avoid auto-adding a lineup, as they have a daily limit on that. Also, while you are in those screens, there is a lot of text that needs to be changed (the stuff about free guide data at the top, and the text on the warlock selection needs to difrentiate it better.
I agree; it was just a thought on how to keep people from messing up their lineups. There isn't a screenshot for it, but when a lineup is added or removed from the account, a dialog pops up letting you know how many changes you have left.
I'm just so used to the wording on that screen, I forgot that it's still telling you that the guide data is free. I'll think of something if @Narflex doesn't come up with something first.
@Narflex, I'm totally unfamiliar with XMLTV, but is it handled by the WarlockRipper, or is it a different DataSource plugin? I'm wondering if it would make sense to split XMLTV out to it's own datasource option on that screen, to make things clearer.
XMLTV is handled by WarlockRipper. When enabled, it essentially replaces it.
I did not go to great lengths to generify the way EPG providers interact with the API because there are a lot of static methods used by the API in WarlockRipper. I didn't want to move all of that into an interface at least for now, so there would be additional work that would result in some minor changes to EPG.java and large changes to WarlockRipper.java if we really wanted to enable the XMLTV plugins as their own thing. So far I just added 3 new API calls, an EPG getter, EPG setter and String to SHA1 hex for SD passwords. The getter and setter as you may have guessed also provides a means to tell it what EPG source you want to work with.
I'm on vacation again, but I'll be back next week, so I can provide more feedback then. Great progress though!
Jeff Kardatzke Sent from my Android
On Aug 18, 2016 6:23 AM, "Joseph Shuttlesworth" notifications@github.com wrote:
XMLTV is handled by WarlockRipper. When enabled, it essentially replaces it. I did not go to great lengths to generify the way EPG providers interact with the API because there are a lot of static methods used by the API in WarlockRipper. I didn't want to move all of that into an interface at least for now, so there would be additional work that would result in some minor changes to EPG.java and large changes to WarlockRipper.java if we really wanted to enable the XMLTV plugins their own thing. So far I just added 3 new API calls, an EPG getter, EPG setter and String to SHA1 hex for SD passwords. The getter and setter as you may have guessed also provides a means to tell it what EPG source you want to work with.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/161#issuecomment-240777471, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDD5RIsxPBmtrFHVYspubHk8k4DOEks5qhIb8gaJpZM4JVuyx .
More screenshots.
Some countries only have one postal code, so I have a step to verify if the regex provided by SD would only match the postal code example and if that's the case, the postal code example is used and the postal code entry screen is skipped. The example you see next to Ex. is the example provided by SD, so it will be different depending on the country. Also since SD does very little actual validation, but does provide regex for each country, I am checking to make sure the postal code is correct for the country before submitting a request to SD. This check can be disabled in Sage.properties if it becomes a real problem for anyone.
I also think I have a better idea for removing lineups from the account. I'll set it up so if you try to remove a lineup that's in use by any capture device, you'll get that red flashing warning for confirmation that you really want to do this.
Looks great! Awesome work.
Thanks. I'm gaining a lot of Studio experience thanks to this. I think this will be a well used feature, so I'm doing my best to make it look and work well enough that there should be few complaints.
I changed the wording on the EPG provider selection page to read: "SageTV license owners can receive Program Guide data for the United States and Canada with no subscription fees. Users without a SageTV license can use Schedules Direct (which requires an active subscription) or can configure the XMLTV Plugin (which needs to be installed separately). Do you want to configure the " + CaptureDeviceInput + " source with Program Guide data?"
@CraziFuzzy I didn't change the options yet. I like what you suggested, but I think it will be too verbose for a button due to the "merger" of SageTV License guide data and XML guide data.
@Narflex Any hints on how to remove a single item from a String array via the API? There seems to be a race condition between the account data coming down from Schedules Direct and removing a lineup from the account. What I'm doing right now is, right after the lineup is confirmed by SD as removed from the account, I query the account for the current lineups and refresh the table. The problem is that sometimes the removed lineup is still returned with the account if this happens too fast. I could inject a 1 second delay that might work most of the time, but that doesn't feel right. I would like to just remove the lineup from the array and refresh the table instead of re-querying SD.
If I recall, the only way would be to create a new array, and skip adding that one.
@Narflex I've been defining all of the classes for the maps and sets in EPG.java and just more clearly knowing the types being used is starting to clear things up for me quite a bit as to what exactly this class is doing in various places. I'm wondering if you think there might be any value in using re-entrant locks here. I say this mostly because I think I'm seeing a lot more reading than writing.
@CraziFuzzy That's what I thought someone might suggest. I've seen some examples of "for" loops in the STV and while they're not awful looking in any way, maybe I'd be better off with just introducing a one second delayed check in the remove method on the Java side of things to ensure that the lineup no longer comes down with the account lineups before returning.
Well, to clarify... You probably can handle your removal using one of the Filter methods in the database API. But behind the scenes, it will still ultimately be making a new array (if it was a map or collection, it can filter them in-place). If it makes the STV code cleaner, you should be able to. A FilterByBoolMethod call set to not pass on match with an equals function should do what you need I think.
The only other thought I have is to cache the account lineups and add/remove from the cache, but I feel like that's a bad idea since they could be changed by other programs and SageTV wouldn't know it needs to refresh them.
I have a concern about a specific API method that make assumptions that you can't have if we don't have a clear way to determine which source they should be using. It's not a problem so long as the lineup already exists. If the lineup does not exist, it makes some assumptions that if an EPG plugin is selected to use that and if it's not, to use WarlockRipper. I don't have any ideas that won't require global changes to the STV on how to fix this.
public boolean ConfigureInputForEPGDataLineup(String CaptureDeviceInput, String Lineup);
Isn't ConfigureInputForEPGDataLineup only called in one place in the STV? (I didn't check yet) I knew about that one and figured you would just make a new API call for SD if it's using that source.
Jeff Kardatzke Sent from my Android
On Aug 21, 2016 1:10 PM, "Joseph Shuttlesworth" notifications@github.com wrote:
I have a concern about a specific API method that make assumptions that you can't have if we don't have a clear way to determine which source they should be using. It's not a problem so long as the lineup already exists. If the lineup does not exist, it makes some assumptions that if an EPG plugin is selected to use that and if it's not, to use WarlockRipper. I don't have any ideas that won't require global changes to the STV on how to fix this.
public boolean ConfigureInputForEPGDataLineup(String CaptureDeviceInput, String Lineup);
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/161#issuecomment-241278798, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDJw8eTnR0VC5Uo7adyQivzt_F1meks5qiLCkgaJpZM4JVuyx .
I am having a hard time finding what menu it is used on, so I can't determine how to get around it. I must be missing what Find all... does, because I have yet to have it find anything I have ever tried to search for. I did a text search on the XML and found it, but it looked like it could be called if the input was changed which wouldn't know if it was Schedules Direct or not.
Nevermind, I found it. I failed to notice that Find all... doesn't change the screen focus. There's 6 places.
I'll figure something out.
I also can see a remote possibility of the source ID's colliding if one were to use Schedules Direct and WarlockRipper at the same time. Since in Schedules Direct the closest thing to an ID would be the URI they provide with each lineup, I plan on generating a hash specific to their format out of that and following it with a check for existing ID's in use by WarlockRipper to make sure nothing gets overwritten. The URI will be stored in Sage.properties as the actual identifier for the lineup when it reaches out to Schedules Direct to update. If there's a range that you can safely say WarlockRipper won't touch, I can work with that instead and just make sure the hash is always in that range.
Never mind on that either. The ID's are issued as unique from the Wizard and I'm sure you were just thinking that.
So my new plan is to append " (Schedules Direct)" to schedules direct lineups once they are actually to be used by SageTV. That way all I need to do is have ConfigureInputForEPGDataLineup check if the provided lineup ends with " (Schedules Direct)" to know that it needs to go that route, plus this will give better visibility in the UI to the end user about what they are using for EPG.
How long will that make some of the lineup names? Wondering display wise if that might end up being too much. I dont' recall how long sd's lineup names can be, but maybe " (sdepg)" may be more appropriate.
Unrelated, but for the postal code entrance page, are you using the postalCodeExample they return in the countries query for the example in the question?
It's no worse than what happens when you copy a lineup for a network encoder. It's not like the lineup name is displayed in many places and I figure if you're going to display it, you might as well make it look formal. Although on the other hand it might actually be easier to read it abbreviated like that for the particularly long names.
I think I commented on that earlier but yes, I am using the example they provide. If for any reason an example can't be acquired, the example is just removed from the prompt.
I'll get a eventually get a screenshot of what it looks like with either or. I saw one with " (sdepg)" and I was surprised at how bad it looked.
@enternoescape To remove the item from the list in studio: you can call standard java methods from within studio code. So you can use the regular remove method from the java List. You would call it like this from studio code:
java_util_List_remove(listVar, itemToRemove)
The convention is fully qualified classname, underscores instead of periods, and the first argument is the object on which to call the method.
Thanks. I'm using an array so it actually would need to be re-created. I'm not sure if you can use a List for a table in the Studio.
Right now in Schedules Direct, the longest name is German:
"Kabel Deutschland (Vodafone) Berlin - Berlin (Panketal - unausgebaut) - Cable" (77 characters)
which is lineup DEU-0002745-DEFAULT
There are more than 80 that are > 50 characters but only 5 that are over 60 and those are outside of North America. If you can handle the case where the name is < 60 that will be 99% of the lineups you'll encounter.
It's ok if the name is long, it will just wrap around onto the next line. This is really just an aesthetics thing we're debating here.
Provider IDs in SageTV are created by doing a base 36 conversion of the TMS 'headend ID' which is a a 7 character code (and then we add one more character to the end of it to differentiate 'digital' from other types of lineups). This won't ever get anywhere near the max value of longs. So you may just want to set the second highest bit to one for your provider IDs and then we can be sure there is no conflict between the two. Originally, the theory was you had to use the EPG plugin for all your data sources...or use the internal EPG client...you could not mix the two so there would never be conflicts.
On Sun, Aug 21, 2016 at 9:10 PM, Joseph Shuttlesworth < notifications@github.com> wrote:
It's ok if the name is long, it will just wrap around onto the next line. This is really just an ascetics thing we're debating here.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/161#issuecomment-241311272, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDM6_9vjt9mGhOXcRQJmQHXAApRsWks5qiSEhgaJpZM4JVuyx .
Jeffrey Kardatzke jkardatzke@google.com Google, Inc.
And if you have any other pending questions I missed...let me know, I'm back from vacation now. :)
On Mon, Aug 22, 2016 at 3:55 PM, Jeffrey Kardatzke jkardatzke@google.com wrote:
Provider IDs in SageTV are created by doing a base 36 conversion of the TMS 'headend ID' which is a a 7 character code (and then we add one more character to the end of it to differentiate 'digital' from other types of lineups). This won't ever get anywhere near the max value of longs. So you may just want to set the second highest bit to one for your provider IDs and then we can be sure there is no conflict between the two. Originally, the theory was you had to use the EPG plugin for all your data sources...or use the internal EPG client...you could not mix the two so there would never be conflicts.
On Sun, Aug 21, 2016 at 9:10 PM, Joseph Shuttlesworth < notifications@github.com> wrote:
It's ok if the name is long, it will just wrap around onto the next line. This is really just an ascetics thing we're debating here.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/161#issuecomment-241311272, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDM6_9vjt9mGhOXcRQJmQHXAApRsWks5qiSEhgaJpZM4JVuyx .
Jeffrey Kardatzke jkardatzke@google.com Google, Inc.
Jeffrey Kardatzke jkardatzke@google.com Google, Inc.
I have the STV interface 90% there which is enough for me to add a new lineup that will be loaded using Schedules Direct. I already have 99% of the code to parse the JSON into POJO's with reasonable efficiency and a utility class to seamlessly convert certain things into more SageTV friendly formats such as the New Zealand and Australia station ID's. I'm now working on getting the data into the database and I'm using protected boolean extractGuide(long)
in WarlockRipper as a guide. I know I'll need to do a few things differently, such as store the md5 hashes for various objects so we know if something changes on a specific channel and day, so we should update the data we downloaded previously.
There's also optional logging of the entire conversation between Schedules Direct and SageTV to the file sd_epg.log. SD uses compression for the larger pieces of data, so those pieces are decompressed before they get logged.
I didn't look for it yet, but any chance that SageTV can handle an http redirect for images? I was also looking at some code used to validate if a URL is valid and I noticed that https was missing. Schedules Direct is entirely https including images. Is it correct that SageTV doesn't download from https or am I missing something?
@rkulagowski Is the server time returned from any of the REST calls suitable for time synchronization?
@Narflex I think I'm officially confused about what serviceLevel is used for. Line 487, WarlockRipper.java. It doesn't look like I could actually use it with SD, but I don't want to completely overlook it if it's something I should be interested in.
I think service level is something that hasn't really been able to be properly implemented in a very long time, simply because service offerings by various providers are so inconsistent. It really could probably be removed at this point I'm guessing.
It's values are being provided by the SageTV EPG server. It's just unclear to me how exactly the data is/was being used. If it was being used, I'm sure someone would have noticed that when it's values are being loaded from Sage.properties, they come in as <Integer, Map<Integer, Integer>>, but are populated and saved from <Integer, Map<Long, Integer>>. So basically if you restart the server, the values might not actually be parsed back from strings if the saved value was greater than an integer..
I believe it was the 'Basic Service' or 'Expanded Basic Service' choice in the wizard. Essentially used to auto enable the channels you had based on what level of service you subscribed to. With the many packages out there today, I can't imagine the information coming from tribune is in any way useful or relevant anymore. I also don't think they are ever used after the first time a lineup is created and those initial channels are enabled.
Ah, I was starting to think that's what it was, but I couldn't find enough evidence to support my thoughts. Maybe a little down the road, we should eliminate that screen completely if it's not actually all that helpful anyway.
The request for token response and /status API now return the server datetime. I have updated the documentation.
Screen should be pretty easy to remove, as well as the API and methods for it.
I think we can leave the service levels stuff in there, I don't see any reason to remove it. It is what Fuzzy described, it's for selecting Basic vs. Extended Basic (and it has a little more detail than that, but we've never used it). You can just have it skip that screen for SD data sources and not bother populating it for your data sources. It does make setup a little easier because it can automatically enable a larger set of the correct channels.
For the http/https images and redirects, if you're finding any problems there let me know. They should be easily fixable...there was one relating to https downloads that was trivially fixed once SageTV went open source. HTTP redirects should be enabled pretty much anywhere there is http downloading in the code. And I think I found where the https problem is...just search for "http" in MetaImage and ImageUtils and the fixes will be obvious. :)
On Tue, Aug 23, 2016 at 8:42 AM, Christopher Piper <notifications@github.com
wrote:
Screen should be pretty easy to remove, as well as the API and methods for it.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/161#issuecomment-241776834, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDPWpNjpSnC5U4mVqbu92uVvo2PgFks5qixTrgaJpZM4JVuyx .
Jeffrey Kardatzke jkardatzke@google.com Google, Inc.
Where does Tribune get the information for the service levels though? I have not had any cable service that offered just 2 tiers in almost a decade - so the screen has always seemed quite useless to me.
I'm assuming they get it from all the cable companies. This is the specific details on it:
Due to the wide range of service tier packages offered by cable systems, TMS chose to standardize its service tier selections to the basic five: Basic, Premium, Music, Pay-Per-View and Extended Basic. Each station/channel is assigned to one of these service tier selections.
And since Premium is things like HBO, Music is something that most people never use on Cable (at least I don't, I play all my music through SageTV), and Pay-Per-View isn't one a SageTV user would likely enable...we were left with the only 2 that made sense for Basic and Extended Basic.
On Tue, Aug 23, 2016 at 3:32 PM, Christopher Piper <notifications@github.com
wrote:
Where does Tribune get the information for the service levels though? I have not had any cable service that offered just 2 tiers in almost a decade
- so the screen has always seemed quite useless to me.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/google/sagetv/issues/161#issuecomment-241901359, or mute the thread https://github.com/notifications/unsubscribe-auth/ANEIDIeZdWPsaVzAEqGoB5uAOZL2lSI2ks5qi3UNgaJpZM4JVuyx .
Jeffrey Kardatzke jkardatzke@google.com Google, Inc.
This is one of the barriers to entry for new SageTV users. Schedules Direct needs to be a first class citizen fully integrated into SageTV. It needs to be selectable and configurable within the UI.
I have been looking at doing this over the past few weeks and I think it would be best to build everything from the ground up instead of using the plugin. Re-writing this EPG service will result in it being written as a piece of the core, using all that is available internally and not something that is somewhat disconnected. We should also be able to eliminate some library dependencies that we would otherwise need to account for to use the plugin code.
For the JSON parsing, I propose that we use Gson since it's owned by Google already and a compatible license. I will just take the parts we need and change the root class path to sage.epg.sd.gson so it doesn't conflict with any plugins. I would leave the Gson code otherwise unchanged so we could upgrade it if the need were to arise without any serious complications. The path for the schedules direct classes would be sage.epg.sd.
My first goal would be to get everything working as well as the current integrated solution along with UI support. After that is committed, I would take a look at ways we can incorporate additional data provided by SD that SageTV may not currently have a use for, but would provide an opportunity for new options/features.
I propose that we add the following option to the EPG Lineup Setup Wizard.
Use Schedules Direct Guide Data with this Source
1) The next screen would ask for your username if this is your first time, if this is not your first time skip to step 2.
Please enter your username for Schedules Direct.
<text box>
The next screen would ask for your password if this is your first time. Passwords are stored as a sha1 hash in hex somewhere other than Sage.properties.
Please enter your password for Schedules Direct.
<masked text box>
2) If this is not your first time, you are presented with the option to use your existing username and password or change them. Selecting to change then takes you back to 1 with the username already populated in case you just wanted to change your password.
Use Existing Username and Password
Change Username and Password
3) If authentication fails, you will see a message to that effect, then be taken to 2. You are not forced to change the saved username and password, but of course you should probably enter everything again. If the server is offline, you will see a message indicating this and your username and password will be saved, but you will be unable to proceed and will also be taken to 2. If everything goes well, you proceed on.
Authentication with Schedules Direct failed. Please check your username and password.
Schedules Direct not available right now. Please try again in 30 minutes.
4) The API will then query for your existing lineups and allow you to select from what you already have or add a new one. If no lineups currently exist, you are taken to step 5 directly. When removing, it will check to see if any other capture device is using the lineup you are about to remove and will warn you about this fact.
Please select an existing lineup or add a new lineup.
Lineup 1
> prompt if you want to select the lineup or remove it. If you select, you're done.Lineup 2
> prompt if you want to select the lineup or remove it. If you select, you're done.Lineup 3
> prompt if you want to select the lineup or remove it. If you select, you're done.Add a New Lineup
> go to step 4 if you have never added a new lineup. If you have gone through steps 5 through 7 before, you will be shown a dialog asking if you would like to select the same region, country, postal code again which takes you directly to 8 or if you want to select something else, you'll go to step 5.Would you like to use the previously entered Region, Country and Postal Code (North America, United States, 12345)?
Use the Postal Code 12345
> go to step 8.Select a different Region, Country and Postal Code
> go to step 5.The select or remove dialog would be:
Select this Lineup
Remove this Lineup
5) Get the list of regions and countries from SD. List regions to select from, selecting one takes you to step 6.
Please select your region.
North America
Europe
Latin America
etc...
6) List countries for region selected in the previous step. Selecting one takes you to step 7.
Please select your country.
United States
Canada
7) Query for postal code including an example.
Please enter your postal code. (Ex. 12345)
<text box>
After entering the postal code, it will be checked against the regex provided from SD to ensure it's valid. If it's not valid, an error message will appear. Otherwise, we proceed to step 7.
You have entered an invalid postal code.
8) The API will now query for all of the lineups in the provided postal code and present you with a list. At this point, your region, country and postal code have been saved so your last selections can be used to skip these steps for adding the a second lineup. Making a selection here adds the lineup to your account and you're done.
Please select your cable or satellite provider (press a key to jump to lineups starting with that letter):
Provider 1
Provider 2
Provider 3
There will of course be more to discuss, but I just wanted to get a few ideas out there in case anyone has better ideas on how the UI should flow for this feature or if any of my ideas will be harder than I think to implement and what might be a better alternative. For those who do not know, Schedules Direct stores your lineup selections on their server and the UI suggestions I posted here are based on this fact.