Thoughts on enhancements

[hikavdh]

A more properly named issue to continue #49 Error with some ids

[hikavdh]

Oh, and at the end of the json listing it gives the url for the previous and for the next!

I have to look add the code that determines which logo to prefer. It is quit old from when there were only a few sources. But the empty one is weird.

[hikavdh]

I anyhow think it only uses the one from logo_names if there is none come through any of the source grabs.

[kyl416]

Maybe the / in the location of the Radio2 logo is causing problems

[hikavdh]

I forgot a quote in the example!

[kyl416]

I caught that, and added the missing quote

[hikavdh]

I have to review that code. I seems that at present it only accepts source 4 icons from that list.

[hikavdh]

Can you look at those icons. In the wiki is an old list of icons and for source 4 you can get available icons with http://212.142.41.211/ChannelLogos/02/bestandsnamen.txt It would be nice to have a list in order of preference for the sources. Source 1 and 3 are lowest as 1 is very small and 3 very big. We should in the future add the extensions to 'logo_names', but for now not for source 4 as older version then will have problems. I anyhow just added a check on the extension, adding one if absent. Source 0 and 1 have '.gif' 2 and 6 '.jpg' and 3, 4, 5, 7, 8, 10 and 11 have '.png', 9 is mixed. Source 0, 1, 2 and 4 don't come with a grab through --configure. The others do. If I have such a list of sources in declining preference I can select the best on running configure. Next to quality, also scalability is relevant. I don't know exactly about the three formats, gif is bitmap and jpg vector? png I don't know, but vector is better scalable. Also some sources don't update.

[hikavdh]

So except for icons from the empty_channels only source 0, 1, 2 and 4 need to be in logo_names! That's probably why I set it to only use source 4. Originally they were always added to tvgids.nl as source 1 is from there but not present in the channel list and source 2 originally came with teveblad.be. So only source 4 icons were relevant. At present it's a bit arbitrary which source wins in supplying the icon.

[hikavdh]

I added "logo_source_preference" to sourcematching.json. Just add the sources in declining order. I'll add code to use the first available icon in that order.

[kyl416]

That text file is kind of outdated and doesn't have anything added after 2012, you can see all the images here: http://212.142.41.211/ChannelLogos/02/

[hikavdh]

Weird, I grabed that file when earlier this year they hadn't locked access. Only just before I got access denied! In the last half year I only could get an icon if I knew the name? So don't expect access to the listing to stay.

[hikavdh]

Anyway they are the same icons as from source 5 on http://212.142.41.211/ChannelLogos/01/ there is a different size.

[hikavdh]

Maybe it opened because I'm on their network and tried accessing it earlier??? In the past I wasn't.

[hikavdh]

I added source 99 as the first source. leave that one in place, so that any self added icons stay in place!

[hikavdh]

I updated the tag with the new iconselection code. I marked out what I already had for grabbing so it still only knows source 10 on configure. I'll remove the channels from empty_channels as this now should be managed by the inactive tag!

[hikavdh]

Oh, and I saw you cleared out the no longer existing channels in empty_channels for tvgids.tv. That's OK, but keep in mind that that list is also used to clear out those ids from old configurations. They won't get fatal errors, but they will get 'url not found' errors on those! I was planning on some procedure to do this say 3 months after disappearance. There are also some on the other sources, but do not clear tv5 from nieuwsblad.be It keeps now and then reappearing??

[hikavdh]

https://github.com/tvgrabbers/tvgrabnlpy/releases/tag/alfa-2.2.8-p20160102 With the basic (start, stop, title) vrt.be fetch. I tested with één and it looks promising. There are some items missing like the night loop and for instance 'clips' at 16:30 today. But they are filled up by the others. Timings are up to the minute, so ...

[hikavdh]

The seconds precision is for the running day up till after midnight. For één till just after 1 when the night loop starts. I guess that the missing programs are those that are just fill-ins. So I guess they do update accurately for the running day and leave the rest proximately correct. I should test when they update. But for me,... I always run a fast update at 1700. I have my first recording on Belgium on Monday, but I'll check tonight on the accuracy.

[hikavdh]

I found the night loop, it's there as a group that repeats until the next item. Also corrected some things. I'm going to run it to night with vrt as prime_source https://github.com/tvgrabbers/tvgrabnlpy/releases/tag/alfa-2.2.8-p20160102

[kyl416]

Can you figure out why 1-cbeebies is appearing under Nederlands Overig instead of Engels? The channel_grouping shows it under 3, but the final output of the configure puts it in 7

# Cbeebies;7;1-cbeebies;;cbeebies;;;;;;;;;;4;bbc_cbeebies.png

Also I might look into cleaning up the other "Overig", many of those channels should probably be under Nederlands Overig. The pan-European multilingual channels like Euronews should probably go under Internationaal while English channels like MTV Live and VH1 Classic should probably go under Engels.

Nick Hits and MTV Brand New are Dutch originals (the last remaining digital music nets from TMF) so they should probably go under Nederlands Overig.

[kyl416]

Should we add the disable_ttvdb flag to the radio stations from the VRT source like we do for the other radio stations?

[hikavdh]

Remove the line before running configure. I do not reset the group on configure as people might have changed it. This with exception for 0, 8, 9, 10, 11 and 99 as I had to renumber them one/ two months ago. Feel free, but you might want to leave 'dutch' not specific Flemish or Dutch in overige. Anyhow the main idea is to not let any group grow to big and to keep it somehow logical. If needed an extra group can be created, let me know and I see if code change is needed first. It originated from the grouping on tvgids.tv and teveblad.be and I think later on humo.be. I don't think MTV and the like should go under English as it is not British or American, but international. And YES, if I haven't already place the Flemish radio in the exception I'll do that. I was thinking somewhat earlier that I forgot to update that after the above mentioned group renumbering.

[hikavdh]

I have 6, 8, 11 and 12 in the exception list for ttvdb. I'll add 13. So all regionals and all radio. But normally radio programs don't have subtitles, so they are already excepted. They need to have a subtitle, it must be a series and the season number must be absent.

[kyl416]

BBC First is actually a channel created for the Netherlands where everything is subtitled and the promos/ads are in Dutch, should it still be under Engels?

Also while CBS Reality is English, it's the pan-European feed that's different than the one they get in the UK.

[hikavdh]

Maybe we should create a group 'Europees' next to 'Internationaal' ? But then we should move the radio groups up. Say 21, 22, 29. We can then either put that on 11 or in between 5 and 6, moving those up. Say:

        "0":"Actieve Zenders",
        "1":"Nederlands",
        "2":"Vlaams",
        "3":"Engels",
        "4":"Duits",
        "5":"Frans",
        "9": "Europees"
        "10":"Internationaal",
        "15":"Nederlands Regionaal",
        "16":"Vlaams Regionaal",
        "21":"Nederlands Overig",
        "22":"Vlaams Overig",
        "31":"Radio Nederlands",
        "32":"Radio Vlaams",
        "39":"Radio Overig",
        "99":"Overig"},

Leaving space for future extension. But maybe I better create a group sort list, so we don't need to renumber!

[hikavdh]

I've added a 'group_order' list to sourcematching. I'll update the code to use it.

[kyl416]

I don't think a Europees is needed right now, that might create some confusion on what belongs where. Maybe we can create a seperate entry for Arab and Turk so International won't be too cluttered.

Is all the coding safe for previous versions outside of the group_order, so for users still on the previous version any new groups would just appear at the bottom?

[hikavdh]

I know that your not European ;-), but in a lot of ways Americans are honorary Europeans! ;-) But There are only a few in International that would qualify for European. Most are Turkish, Asian or Arab: (CCTV News, MBC, Mediaset Italia, Rai Uno, Show TV, TV E and TVM Europe)

I think the code is save. The 'channel_grouping' table is relative new, I think I created it with sourcematching for 2.2.6. It anyhow could only create problems on running configure. Further only the actual grouping in the configuration file is used.

[hikavdh]

Not Show TV it's in the Turkish/Asian package on Ziggo. TV E is Spannish

[hikavdh]

I added a tag to the updated code. (ttvdb exception and group sorting) The last was already in essence in place as with sourcematching the dict get disordered and I had a list of ordered key values in place. It checks on missing keys in the order list. I might also later add a reverse check.

[hikavdh]

I'm now signing off! I'll let you know if my night run does anything weird. The discrepancies tommorrow are little between vrt and the other.

[kyl416]

Is it safe for me to create new groupings for Turks, Hindi, Arabische and Erotiek?

    "channel_groups":{
        "0":"Actieve Zenders",
        "1":"Nederlands",
        "2":"Vlaams",
        "3":"Engels",
        "4":"Duits",
        "5":"Frans",
        "6":"Nederlands Regionaal",
        "7":"Nederlands Overig",
        "8":"Vlaams Regionaal",
        "9":"Vlaams Overig",
        "10":"Internationaal",
        "11":"Radio Nederlands",
        "12":"Radio Vlaams",
        "13":"Radio Overig",
        "14":"Turks",
        "15":"Hindi",
        "16":"Arabische",
        "17":"Erotiek",
        "99":"Overig"},

I'm thinking of using this order, the new languages between internationaal and radio, and have Erotiek before Overig:

    "group_order":[0,1,2,3,4,5,6,7,8,9,10,14,15,16,11,12,13,17,99], 

Maybe we can also bump Nederlands and Vlaams up to be before Engels (and have NL/BE Overig appear before NL/BE Regionaal):

    "group_order":[0,1,2,7,9,6,8,3,4,5,10,14,15,16,11,12,13,17,99], 

I think for now Italian should remain part of Internationaal since there's only two channels Mediaset Italia and Rai Uno.

Non-UK/US English channels like Al Jazeera English, Extreme Sports Channel, RT, CCTV News, Fashion TV, VH1 Classic, etc would be in Internationaal. Along with multilingual channels like 2M, Mezzo and Euronews

Overig would be the channels that don't quite fit anywhere else like Zender van de Maand

[kyl416]

I updated a bunch of logos and "demoted" nieuwsblad because several of their logos were blank or outdated and the logos they do have are very tiny.

[hikavdh]

Sorry if I was blunt. But the thought is not about countries, but about languages and culture. European includes US/Canada but not Turkey/Asia/Arabia. So if you find a group 'Europees' a problem,make it 'Amerikaans/europees' or 'Western internatiomal'. What I mean to say is that our customers are interested in American channels but not in Asian. You have to look at the users not at yourself.. So exclude the possible interesting channel from international in stead of the possible uninteresting channels.

[hikavdh]

I think overig in any case should follow any specifics. That is the thought about the distinction. Anyway I wanted you to take charge of the not code things, as I have I lot on my board and I think you capable.

[hikavdh]

To make thing clear! I do want you to take charge of all those merge and other things you know much more about it. That's why I brought you in. I want to focus on the coding. But at the same time I want you to see that the American perspective is different from the European. And our users are European To us Italian and Spanish is like Frensh, Turkish, Asian and Arabian is not. European is like US and Canada combined., where every country is a state!.

[hikavdh]

To us Italy or Spain is closer then Mexico to you, while Turkey and others are way farther. Think Culture in proximity, language is relatively irrelevant. We have to many!

[hikavdh]

Oh and giving porno their own group gives them to much prominence. Please leave them in the crowd. But it is your choice!

[hikavdh]

They order for icons is the first in logo_source_preference that in order gives an icon in logo_nanes or through the fetch. In that order. So a mentioning in logo_names precedes an icon delivered by the same source

[hikavdh]

I saw you didn't push jet. But do, but try to keep the European perspective. Italy Spain comes way before Mexico and the other EU countries follow that, before Mexico.. After Mexico comes first Turkey and Maroko almost equal and then way farther Arabia an Asia.

[hikavdh]

But to put it long and shot. I would love it if you for the near future take charge of sourcemerging! I only ask that you tell when you update,

[hikavdh]

To say it strait. Do you wan't to take responsibility for the data in the sourcematching.json file? So I can focus on coding. Do you need any more instruments then I already have provided through the 'write_info_files' option?

[kyl416]

Yeah, I can take responsibility. I don't think I need anymore instruments.

[hikavdh]

Great!!!! \

[hikavdh]

Ask If you need , but I think I told you most of the inner workings

[hikavdh]

I won't from now on do anythimg to sourcematching.json without prior consultation with yuou

[hikavdh]

Keep in mind The latency of people upgrading. I'll work on vrt in the coming days. I'll notify you when it is time to go stady. You may post the announcement. You master Dutch, to upgrade the WIKI at need?

[hikavdh]

Anyhow. I did create an English entry to the WIKI. Do you feel yourself capable of translating my text to English? I'll do it when I find time, But I haven't found in the last half year.The sidebar is not possible, but the rest? I know you can read Dutch.

[kyl416]

I don't know if you implemented it yet, but VRT has genres that you might need to decode into a table: http://services.vrt.be/doc/epg/standardgenres.html http://services.vrt.be/epg/standardgenres

I should be able to translate the wiki for you

[hikavdh]

That is one of the next complicated steps in implementation. I'm currently in the process of getting their feel. And that's one of the parts. I'm at present working on code to inject their variables to me and then to interpreted it.. That's how I found their night grouping. The basis is simple, the rest takes weeks or months.

Yes please!