search

linuxmint / hypnotix

An M3U IPTV Player
1.26k stars 169 forks source link

[Bug] Character encoding error in the name of the German IPTV channel 'MDR Thüringen' #112

Closed mendel5 closed 3 years ago

mendel5 commented 3 years ago

Today I noticed a character encoding error in the name of the German IPTV channel 'MDR Thüringen'. The German letter Ü in the second word is displayed incorrectly. The Ü is also called U-umlaut, see https://en.wikipedia.org/wiki/%C3%9C.

'MDR Thüringen' is the public service television broadcaster for the German state Thuringia, see https://en.wikipedia.org/wiki/Mitteldeutscher_Rundfunk .

The channel can be found here:

Here is a screenshot of the character encoding error: hypnotix_character_encoding

mendel5 commented 3 years ago

Here are some more errors:

hypnotix_character_encoding2 Should be allgäu.tv Kempten, see https://de.wikipedia.org/wiki/Allg%C3%A4u.tv

hypnotix_character_encoding3 Should be:

mendel5 commented 3 years ago

I noticed there is another issue about the same topic here: https://github.com/linuxmint/hypnotix/issues/101 . Maybe you @Free-IPTV can have a look at this? Thanks!

Free-IPTV commented 3 years ago

I noticed there is another issue about the same topic here: #101 . Maybe you @Free-IPTV can have a look at this? Thanks!

Hi @mendel5,

The problem has long been resolved, actually. Sometimes there were problems with umlauts (ä, ü and so on) but then all m3u's were saved in UTF-8 format, then that was done. Could it be that you still use my old m3u's? All m3u's in my repo are now saved in UTF-8 format.

mendel5 commented 3 years ago

Could it be that you still use my old m3u's?

I don't think so. To be sure I removed and reinstalled hypnotix with these commands:

$ sudo apt purge hypnotix
$ sudo apt-get install hypnotix

My hypnotix version is 1.5.

In my Providers settings there is one provider (Free-IPTV) with the URL pointing to https://raw.githubusercontent.com/Free-IPTV/Countries/master/ZZ_PLAYLIST_ALL_TV.m3u. I removed this provider entry and added it again so that the m3u file is getting updated.

Unfortunately that did not solve the issue. The names of some German IPTV broadcasters are still displayed with encoding errors.

Free-IPTV commented 3 years ago

Unfortunately that did not solve the issue. The names of some German IPTV broadcasters are still displayed with encoding errors.

Hmm, okay, problem is not solved. I don't know which character sets Hypnotix supports. I use Kodi myself, so the ZZ_PLAYLIST_ALL_TV.m3u is created with the Add-ON IPTV-Merge. As I wrote, we used to have the problem with special characters, but since I have been saving the m3u's in UTF-8 format, there have been no more problems.

Of course this has to be solved somehow, how does Hypnotix deal with character codes, is UTF-8 supported?

mendel5 commented 3 years ago

Hmm, okay, problem is not solved. I don't know which character sets Hypnotix supports. how does Hypnotix deal with character codes, is UTF-8 supported?

I'm quite sure that Hypnotix supports UTF-8. When I open the file ZZ_PLAYLIST_ALL_TV.m3u in Linux Mint's xed text editor, I get this error: screenshot_playlist-all

However when I open the file DE01_GERMANY.m3u where MDR Thüringen is included there is no encoding error: screenshot_playlist-germany

So this issue seems to be about the specific file ZZ_PLAYLIST_ALL_TV.m3u and not about Hypnotix.

mendel5 commented 3 years ago

I just did another test to check whether the issue arises from the file or from Hypnotix.

In Hypnotix' Providers overview I deleted the entry with the URL pointing to https://raw.githubusercontent.com/Free-IPTV/Countries/master/ZZ_PLAYLIST_ALL_TV.m3u and created a new provider entry with the URL pointing to https://raw.githubusercontent.com/Free-IPTV/Countries/master/DE01_GERMANY.m3u.

When using the second file DE01_GERMANY.m3u the IPTV broadcaster MDR Thüringen is displayed correctly (without character encoding error).

So by now I'm sure this issue has something to do with the file ZZ_PLAYLIST_ALL_TV.m3u.

Free-IPTV commented 3 years ago

Okay, when you open the Playlist here, there all characters okay:

https://raw.githubusercontent.com/Free-IPTV/Countries/master/ZZ_PLAYLIST_ALL_TV.m3u

In other apps, the problem with the playlist no longer exists since all files (including the playlist) are stored in UTF-8.

The only thing I can do now, I swap all letters (special characters), as an example Ü for ue and so on. But that takes time and is not even done in a few minutes. I could start doing that tomorrow morning. I just don't understand it, since the playlist is also in UTF-8 format and nobody has reported a problem with character codes since then.

mendel5 commented 3 years ago

The only thing I can do now, I swap all letters (special characters), as an example Ü for ue and so on.

No, please don't do that. I will try to fix this and get back to you.

Free-IPTV commented 3 years ago

The only thing I can do now, I swap all letters (special characters), as an example Ü for ue and so on.

No, please don't do that. I will try to fix this and get back to you.

Tomorrow Morning i create with the newest IPTV-Merge Addon and Kodi 19 a new Playlist and we test it, okay? I has a second TV-Box with 905x3 to test some things in the new Kodi 19. ;-)

mendel5 commented 3 years ago

Tomorrow Morning i create with the newest IPTV-Merge Addon and Kodi 19 a new Playlist and we test it, okay?

Sure, we can do that.

What is the origin of the individual files (not the ZZ_PLAYLIST_ALL_TV.m3u)? In https://github.com/Free-IPTV/Countries/blob/master/README.md you write:

To automatically create the individual lists to one list, i use the ADD-On IPTV-Merge from Matt Huisman.

But where do the individual lists come from in the first place?

Free-IPTV commented 3 years ago

Tomorrow Morning i create with the newest IPTV-Merge Addon and Kodi 19 a new Playlist and we test it, okay?

Sure, we can do that.

What is the origin of the individual files (not the ZZ_PLAYLIST_ALL_TV.m3u)? In https://github.com/Free-IPTV/Countries/blob/master/README.md you write:

To automatically create the individual lists to one list, i use the ADD-On IPTV-Merge from Matt Huisman.

But where do the individual lists come from in the first place?

I have all the individual lists on my PC. If there are updates from me, I upload the new file to Github. In the case of updates from others, I also save the modified m3u on my PC. Check out the playlist here, it is now created with the latest IPTV merge under Kodi 18. Rename it to .m3u ;-)

ZZ_PLAYLIST_ALL_TV.txt

mendel5 commented 3 years ago

I think I found the character encoding error. It's located in the file DE03_GERMANY_Local.m3u.

You can reproduce the process on your PC:

Free-IPTV commented 3 years ago

I think I found the character encoding error. It's located in the file DE03_GERMANY_Local.m3u.

Okay, i upload this file again to github tomorrow morning, when i makes the next updates. I use Windows, edit the files with Notepad++ which supports UTF-8. I has no Linux... sorry. ;-) Only Libreelec with Kodi on my Raspi4 ;-)

I'll get back to you tomorrow, okay?

Have a nice evening. :-)

mendel5 commented 3 years ago

Yes, thank you.

I added a comment here: https://github.com/Free-IPTV/Countries/commit/6a6a05ab9c974f01a4565160746c7a0253f087cc#r47965647.

Direct link to the line: https://github.com/Free-IPTV/Countries/blob/49ab8324f2a941e685279bb4c739a3705bc71ac8/DE03_GERMANY_Local.m3u#L6 (see end of line)

Here is the output of $ isutf8 --verbose *:

$ isutf8 --verbose *
DE03_GERMANY_Local.m3u: line 6, char 189, byte 578: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
6E 67 22 2C 61 6C 6C 67 E4 75 2E 74 76 20 4B 65  | ng",allg.u.tv Ke
                        ^^^^^                    |         ^^

ZZ_PLAYLIST_ALL_TV.m3u: line 400, char 189, byte 86973: After a first byte between E1 and EC, expecting the 2nd byte between 80 and BF.
6E 67 22 2C 61 6C 6C 67 E4 75 2E 74 76 20 4B 65  | ng",allg.u.tv Ke
                        ^^^^^                    |         ^^

You only need to fix the character encoding in the first file. Then you can merge all individual lists to the big one and the fix should be included as well.

Free-IPTV commented 3 years ago

Hi @mendel5,

I hope that all characters are now displayed properly. Thanks for your great search for help.

By the way, Iran joined last week, Pakistan joined today. We now have over 80 m3u's with over 3600 streams ... brilliant. :-)

mendel5 commented 3 years ago

I hope that all characters are now displayed properly.

Yes, that seems to be the case.

Thanks for your great search for help.

Thanks for your quick response!

I will close this issue now.