Please make this wonderfull editor support TCL Google TV.

[tomxhachijuroku]

Please, here is my .tar channel list, for example: Channel_List_V8-R51MT05-LF1V460_20221223.zip

[Double-A-92]

The data itself is stored in simple SQLite databases... BUT there is a CRC check, so it's probably not possible to edit it. Unless somebody can guess how that CRC is calculated exactly.

[PredatH0r]

I used Hex Editor Neo to auto-calculate all sorts of checksums of DtvData.db and it appears that its CRC16-CCITT can be found in cloneCRC.bin at offset 2 (in little-endian byte order). The CRC16-CCITT of satellite.db is stored at offset 4. Everything else in the cloneCRC.bin has value 0.

So that particular problem is solved, the more severe is to find the time. I must get some business work done first and hope to do some work on ChanSort later this week.

[PredatH0r]

@tomxhachijuroku What is the exact directory and name of the file that the TV writes to the USB stick? Does it write a .tar file directly into the root folder or does it create a folder "Channel_List_V8-R51MT05-LF1V460_20221223" and inside a file named "Channel_List_V8-R51MT05-LF1V460_20221223.tar" or does it create a .zip file that contains the folder and the .tar?

[PredatH0r]

@tomxhachijuroku Can you please give this version a try at let me know if it works? https://github.com/PredatH0r/ChanSort/releases/tag/v2023-01-03

[matepor464]

I successfully edited my channel list with 2023-01-04 version, but can't import it into the TV. While comparing the edited and original list in SQL lite 3 browser, i found that all the channel names went from "BLOB" (original) into their actual names, eg. "HBO HD" (edited). Maybe this is preventing a successful import? If You find some time, please look into this issue - the official TCL channel editor has been down for some time now.

And to answer Your question from 3 days ago - TV writes the channel list to USB stick root directory as .tar file. The .tar file contains "database" folder with cloneCRC.bin file and another folder, "userdata", which in turn contains DtvData.db and satellite.db files

[PredatH0r]

Thanks for the feedback, I'm looking into it right now

[PredatH0r]

When I open your DtvData.db file with SQLiteStudio 3.2.1 it shows that in PrograminfoTbl table the ServiceName column has (default) data type VARCHAR(64). I'm reading it as as string and stripping the trailing spaces. I'm writing it also as a string without any further manipulation.

SQLite unfortunately can be very weird, allowing individual rows/fields have a different data type as the column's definition. And Microsoft's .NET System.Data.Sqlite library has its own set of quirks to deal with this.

EDIT: Where did you see that the name is a BLOB? nvm, just found out using "select typeof(ServiceName) ..." that the individual values are indeed BLOBs and not VARCHAR(64).

[matepor464]

I see that You already found the BLOBs, but I will post screenshots anyway, in case it helps :) I am using DB Browser for SQL Lite. "original.png" shows the original channel list. There are a lot more values displayed in binary for Channel 1. After opening it with ChanSort and simply saving it, there are less values, and the name is shown (TVN HD), as seen on "edited.png"

I also want to mention that I know nothing about SQL lite, so my findings may be of little use to You.

[matepor464]

Forgot to attach files. First is original, second - edited

[PredatH0r]

https://github.com/PredatH0r/ChanSort/releases/tag/v2023-01-06 has now a new .zip that writes the names back as BLOB with length 64. Let's hope that works :)

[matepor464]

Thank You, I will try it out and let You know if everything works fine :)

[matepor464]

Unfortunately it still does not work, but I have no further ideas as to why :( All the data in PrograminfoTbl table seem identical to the original file. Maybe the issue is in a different table or in cloneCRC file? The edited file (only opening and saving) has 1 less kB in size, so this might indicate that some changes are made even just by saving, but it is only a wild guess. Still, thank You for a quick response and an attempt at fixing the issue :)

[PredatH0r]

In this new build here I removed the "REINDEX" statement that I usually execute on SQLite databases because it can help recover broken databases (which some TVs export). The file size however is still 1KB less compared to the original file.

I'm not updating any other tables than PrograminfoTbl. I do update the cloneCRC.bin file with the new checksum of DtvData.db. Reopening the saved file then verifies the CRC correctly.

Another possible cause is the library used to create the .tar. From what I've read there are several different flavors of .tar and the TV might only work with a specific one. Opening the original one with 7zip and File / Properties it shows "Characteristics: GNU ASCII" The saved one shows only "ASCII". I have to try and find out if there is any way for me to control what TAR-dialect is used by the library.

Do you have a way to untar and re-tar the file modified with ChanSort and see if the TV can process it then?

[matepor464]

I will try to re-tar it, but i have yet to find a way to do it. I used winRar to untar it and check .db files, but winRar can't tar them again

[matepor464]

Just one question - how do You view the .tar file properties? I can't seem to find any info about GNU-ASCII/ASCII characteristics on Windows 11

[PredatH0r]

I opened the .tar with 7-Zip and clicked on "File / Properties" menu within 7-Zip.

When using "Add to Archive" 7-Zip also allows to select between GNU and POSIX mode for the TAR archive format:

[matepor464]

I extracted the original list with winRar, then used these commands to re-tar it:

The TV accepted the .tar file, so I will be trying to edit the extracted files with ChanSort and then making a .tar file with cvf command

[matepor464]

Didn't work with edited files :/ when I extracted the original files and then re-tarred them immediately it worked fine

[matepor464]

One thing is strange - both unedited and edited .tar files are POSIX ASCII after I used cvf command, but only unedited file got accepted by the TV.

[PredatH0r]

Maybe the problem isn't related to the .tar archiving after all. I'll upload a new build that doesn't "REINDEX" the database and doesn't change the names or anything other than the ProgNum values. I hope that narrows down the cause.

[PredatH0r]

New build at https://chansort.com/ChanSort_2023-01-06.zip

[matepor464]

So this is weird - simply copying the .tar file in Windows makes ot unable to be imported into the TV. I dont't know if You can do anything about that, maybe it is meant to be edited on another OS ;P

[PredatH0r]

How did you perform the earlier test with un-taring and then re-taring? Was that under Windows or Linux and directly on the USB or copy to PC first and then back? Also, did you use a FAT32 or NTFS formatted stick? TVs very often have issues with NTFS sticks or sticks >16GB.

It seems unlikely that simply copying back+forth causes the issue. But I'm at a point where I'm not ruling out anything anymore :) So your last test was export from TV, copy to Windows and then straight back to USB and TV?

[matepor464]

Last test was TV -> USB -> PC, then copying, and back to USB -> TV. When re-taring, I copied the file to PC, extracted with winrar, then re-tarred with commands, copied to USB and then to TV USB stick is FAT32. I will try to make some tests again later and try to see what works and what doesn't. The other thing is that TCL firmwares are famously bad, so it may be my TV's issue :)

[PredatH0r]

I just created a new build using Win10's built-in "tar" command instead of the "SharpCompress" code library. The "SharpCompress" library produced a .tar that just showed "ASCII" in 7-Zip. The original is "GNU ASCII". And the new build with Win10's tar shows "POSIX ASCII".

The original GNU version includes file permissions, user, group and group-id, while the others don't. Maybe the TV needs that data for the import. But then again, the untar-retar should not have worked either.

I just uploaded my new Win10-tar based build here: https://chansort.com/ChanSort_2023-01-06_1925.zip EDIT: fixed link

[matepor464]

I no longer know what's going on. Just now I copied the original file 2 times, extracted them and then re-tarred it with 7-zip and cvf command on my PC - all 3 files could be imported into the TV. Then I tried to use ChanSort on all of them and also on extracted original files which I then re-tarred as mentioned above - they couldn't be imported :( What is weird is that simply copying the file sometimes makes it unable to be imported. That was the first thing I tried - the copy of the original .tar couldn't be imported But the again - the files which I re-tarred were also copies and they could be imported. I will try to ask TCL support about exporting channel lists and also send them the files - maybe they can tell what's wrong with the edited lists

[tomxhachijuroku]

I just created a new build using Win10's built-in "tar" command instead of the "SharpCompress" code library. The "SharpCompress" library produced a .tar that just showed "ASCII" in 7-Zip. The original is "GNU ASCII". And the new build with Win10's tar shows "POSIX ASCII".

The original GNU version includes file permissions, user, group and group-id, while the others don't. Maybe the TV needs that data for the import. But then again, the untar-retar should not have worked either.

I just uploaded my new Win10-tar based build here: https://chansort.com/ChanSort_2023-01-06_1925.zip EDIT: fixed link

Thank You Everyone!!! It works fine with my TCL 43P638 Google TV, editing and importing 100%. That's a big help since my TV provider likes to add and delete channels, all the time without notice and sorting over 120 HD channels with the remote was taking hours (They deleted all over 60 analog channels so it's less work but still), thank you again and Happy new year!!!

[PredatH0r]

oh wow, thanks for the feedback! I am currently working on my own implementation to create "tar" archives because non of the windows implementations preserve file permissions and ownership and I thought that might have been the cause of matepor464's problems. But good to know there is a was to get it actually working.

It would be great if you could give my final code another test once it is finished (probably tomorrow). I'll keep you posted.

[PredatH0r]

latest build is now on https://chansort.com/ChanSort_2023-01-08.zip

It also allows changing names and set "hidden" and "favorites" flags, which was disabled in the 2023-01-06_1925 build.

[matepor464]

I just created a new build using Win10's built-in "tar" command instead of the "SharpCompress" code library. The "SharpCompress" library produced a .tar that just showed "ASCII" in 7-Zip. The original is "GNU ASCII". And the new build with Win10's tar shows "POSIX ASCII". The original GNU version includes file permissions, user, group and group-id, while the others don't. Maybe the TV needs that data for the import. But then again, the untar-retar should not have worked either. I just uploaded my new Win10-tar based build here: https://chansort.com/ChanSort_2023-01-06_1925.zip EDIT: fixed link

Thank You Everyone!!! It works fine with my TCL 43P638 Google TV, editing and importing 100%. That's a big help since my TV provider likes to add and delete channels, all the time without notice and sorting over 120 HD channels with the remote was taking hours (They deleted all over 60 analog channels so it's less work but still), thank you again and Happy new year!!!

Could You describe Your exporting and importing process step by step? I have a 65C635 Google TV, so it should also work for me, but my TV doesn't accept the edited files. Which version of ChanSort did You use? Did You edit the .tar file directly, or did You extract it first?

[matepor464]

I'm still trying to make it work somehow, tomxhachijuroku could import the file so it's highly possible that there is an issue on my end 😄. Thank you for trying to help me. Anyway, I found a bug unrelated to my importing problems - when I use the empty list option while opening the .tar file, add some channels and want the rest deleted during saving I get this error: (build 2021-01-08) Adding the unsorted channels at the end of the list works fine, but I think that deleting them may be beneficial for the TV. I don't know exactly how EPG works in TVs, but maybe with less channels it could load faster?

[PredatH0r]

Deleting is now fixed here: https://github.com/PredatH0r/ChanSort/releases/tag/v2023-01-10

Have you tried resetting your TV to factory defaults yet? Some other TVs (e.g. LG WebOS 6) only import correctly on a freshly reset TV without any channels in its memory. Reset is also sometimes needed when there was a firmware-update (maybe automatically) without re-scanning the channels later. Since most TV manufacturers don't offer any editing software, the export/import function is mostly intended to fully clone one TV onto another identical one without having to scan channels there. Typical use-case for that are hotels with many identical TVs.

[matepor464]

Haven't tried that yet, but definitely will, thanks for the advice. I did recently update the firmware, so maybe that's the problem

[PredatH0r]

There is a "DatabaseInfoTbl" table inside DtvData.db which holds a u8Version value. Maybe the new firmware is now expecting a different value and rejects a list that was exported with an older firmware and version number in it.

[Gp0n5]

Hello, i have 55c728. edited and trying to import TV but it has error when importing. Using 2023-01-10 version.

[Gp0n5]

There is a "DatabaseInfoTbl" table inside DtvData.db which holds a u8Version value. Maybe the new firmware is now expecting a different value and rejects a list that was exported with an older firmware and version number in it.

I compared v470 and v474 exported files. Table values are same.

[PredatH0r]

Thanks for sharing your findings.

I compared files from a 43P638 and a 65C635 that were sent to me. While the user with the 43P638 said that the upload worked, two users with a 65C635 reported that it didn't work for them. The databases of both models have identical structure and value 45 int DatabaseInfoTbl.u8Version. I even used a hex editor to compare the modified and original DtvData.db. Except from the channel number and a few changed bytes in internal Sqlite data, they were identical. And the same with the .tar file after writing my own Tar library which keeps everything identical except checksum and the file size information. It almost looks like that there is something wrong in the TV's firmware.

Did you use the TCL web-based editor before? I wonder if that editor worked for all models, or if they pulled the plug because it only worked for some models.

[Gp0n5]

Yes, I used it before. But link not works for a while. Because of that I am looking for another solution :) I am also trying to export import with 7zip and will edit on sqlite than will inform you.

[PredatH0r]

Do you by any chance have a pair of original file + file edited with the online editor saved somewhere? Comparing the two should give me a clue what the online editor actually changed and what ChanSort is making differently.

Theoretically it's also possible that they broke the import function with a firmware update and then took the site offline. But I'm not quite ready to give up just yet.

[PredatH0r]

Have you tried making a minimal change with Chansort, like just swapping program 1 and 2 and upload that file again? The more complex the changes (like deleting a channel or setting favorites), the more likely it is that the result doesn't meet the TVs expectations.

I uploaded a new build, similar to the one that succeeded for one user, in which only the ProgNum column is updated and nothing else. Maybe you could give that build another try and swap 2 programs: https://chansort.com/ChanSort_2023-01-11.zip

Thanks, Horst

[Gp0n5]

Tried new version and still same. I changed only 1 channel program number. If you want I can share with you exported and changed tar files.

[Gp0n5]

Channel_List_V8-T615T03-LF1V474_2023111.zip

[PredatH0r]

That would be great. The more test files I have, the better. Could you try to copy the original file from the USB stick to your PC, rename the file on the USB stick and then copy the file back from PC to USB? It sounds weird, but one user reported that the TV sometimes rejected "unmodified" files just after copying them. I sounds unlikely and something else might have been going on there too. But it's worth testing this out. If that really causes the file to be rejected, it would mean we are at a dead end.

[Gp0n5]

That would be great. The more test files I have, the better. Could you try to copy the original file from the USB stick to your PC, rename the file on the USB stick and then copy the file back from PC to USB? It sounds weird, but one user reported that the TV sometimes rejected "unmodified" files just after copying them. I sounds unlikely and something else might have been going on there too. But it's worth testing this out. If that really causes the file to be rejected, it would mean we are at a dead end.

Copied to PC, renamed, copied to USB and imported successfully.

[PredatH0r]

I think I know now what's going on. My analysis was based on the file provided by tomxhachijuroku from his P638. That firmware uses CRC16-CCITT for the checksum. Your TV seems to use a different checksum algorithm. That's why the checksum written by ChanSort is rejected by your model. I'll try to find out what kind of checksum your TV calculates. Hopefully I can replicate that.

[PredatH0r]

One last favor: could you export a list from the TV, save the file on the PC. Then move 1 channel with the TVs menu and export that list too. That should make it easier to find out how the checksum is calculated. Unfortunately it isn't any of the standard candidates.

[Gp0n5]

oisorginalcischanged.zip Channel_List_V8-T615T03-LF1V474_2023111o = Original Channel_List_V8-T615T03-LF1V474_2023111c = Moved 1 to 3, 3 to 1 from TV.

[PredatH0r]

Perfect! That did the trick. I was able to crack the CRC mystery with the help of your files. Looks like the CRC is only calculated over a maximum length of 307200 (0x4B000) bytes, ignoring everything that comes after that. With this limit in place, all TCL files I received so far check out fine.

New build is available at https://chansort.com/ChanSort_2023-01-12.zip It will now show an error message when opening a list if the cloneCRC.bin contains a checksum that the one calculated by ChanSort.

[kutyux60]

With this version, you can successfully restore the edited channel list: ChanSort_2023-01-12 Tested on 43C725 V8-R51MT05-LF1V488 firmware

Thank you very much!

[matepor464]

With this version, you can successfully restore the edited channel list: ChanSort_2023-01-12 Tested on 43C725 V8-R51MT05-LF1V488 firmware

Thank you very much!

So this should also work for me, C635 is the same platform 😁 Will test it after work. The only problem with 2023-01-12 is that deleting channels is disabled, but it is only a minor inconvenience