Box character drawn at the end of line when rendering subtitles in W64 build

[shag00]

Using any release from 1/3/20 or later results in subtitles being muxed incorrectly. Every line of every subtitle has a box drawn at the end as shown in the attachment Using the W64 version.

How something like this can go unnoticed for such a long period of time can only be attributed to your prolonged refusal to actually release a release version of this software. Do you not understand that the vast majority of people are not interested in daily builds, they want well used and stable releases that meet their needs. Version 2.6.12 (which I understand is the last stable release) from 2014 is not that. This only came to my attention because I fresh installed my OS.

I have not tested UHDs yet.

[Randrianasulu]

just a wild guess: may be this box represent undisplayable (with current font) character?

[Randrianasulu]

after little digging I found this commit, but it looks a bit invasive for (temporary, for testing) reverting...

https://github.com/justdan96/tsMuxer/commit/9670fd6b19599efb44b66f5c70dc43d112119bd1

@shag00 can you try and check srt files themselves - are they having unix or windows line endings? Also, installed fonts might play a role, as well as active codepage (on older Windows?)

[shag00]

Wild of the mark is what you mean. The same raw files muxed with MKVToolnix or TSMuxer release earlier than 29/2/20 work perfectly. The srt file used for the subtitle has been checked and has no issues. In any case, even assuming you are correct it would not affect every line of every srt file. As a final check to my reply I just downloaded a TV show from the internet which works fine, I demuxed it and remuxed the streams and the problem of the box is still there.

[shag00]

@Randrianasulu not sure what you are asking or how to check. All my subtitles are produced by a program called subtitle edit which I have been using since about 2014 without issue. As I can shown the muxing with MKVToolnix works fine and pin point when TSMuxer started having problems is it not resonable to assume that TSMuxer is the problem? I download 2 versions of the same program issued 1 day apart and 1 works and 1 does not on the same system, that's not a line ending issue.

[shag00]

Further investigation reveals further problems with the 1/3/20 release, which were subsequently corrected at some point, there is no sound when the streams are muxed as .ts though .m2ts is not affected.

[Randrianasulu]

by line endings I mean

diff between windiws and unix on how to signal end of line

[shag00]

OK, I understand, how to check?

[Randrianasulu]

it might be line ending issue in tsmuxer... at least from symptoms it looked like one.

yeah, some bugs fixed in new version - old version (w/o bug) not as useful as one hoped.. anyway, thanks for testing (i replied via email but apparently this email not yet arrived here..)

[Randrianasulu]

@shag00 this answer suggest notepad++ (I am on linux, so midnight commander shows those windows line endings by default)

[shag00]

Tell me how to find end of line that you want

[Randrianasulu]

i just pressed f4 (edit) and if there big ^m at every line end - its windows)

[shag00]

seems it's linux then

[Randrianasulu]

@shag00 are you usung WSL2 on Win10, or this is different ubuntu install? (just asking because hopefully compiling cli version of tsmuxer on linux should be a bit easier than on user-oriented Windows.. if you have time you can try to build latest tsmuxer on ubuntu and see if problem exist there too..)

[Randrianasulu]

also, you can try to manipulate line endings with dos2unix and unix2dos cli utilites..

[shag00]

I have a dual boot system. I have no idea what WSL2 is. On Kate in ubuntu I can change line endings between windows mac and linux though.

[Randrianasulu]

@shag00, WSL2 is new way of running Linux commands (or even whole distro!) on Windows 10

Microsoft says

[shag00]

tested this by changing line ends to windows, ending rectangle still shows up

[Randrianasulu]

( then it was not it... sorry.

[shag00]

I hate to say i told you so but the evidence is pretty overwhelming. If the problem was with the input srt then it would also be apparent when muxed with earlier versions of TSMuxer or MKVtoolnix. The whole problem arises because there is no release version available for most people to use. Everyone testing their own daily version does not work because there are not enough users, even windows with hundreds of millions of users and no doubt a very large pool of testers gets their releases wrong.

Can you reproduce the same results I am getting?

I have been at this game for almost a decade and do it daily so I will eventually notice.

[Randrianasulu]

if you can put small srt/video par and meta for running tsmuxer I can try on Termux and Slackware...

right now release might be not very good idea, see other recent issues and 'ts buffer management problem'. May be beta? And then series of RCs... (without major changes)

just my (non-dev) opinion..

[shag00]

What do you call small, 1TB, 100MB or something else?

It is very hard for me to take your opinion seriously when the last stable release was released in 2014. It's now 2022. By what logic or means do you justify to yourself that that is reasonable? I don't mean to give you a hard time it's just that I am very surprised you would say that.

[Randrianasulu]

100 mb should be enough? (just enough for seeing few subtitle lines)

I mean some recent bugs on Mac/arm... are you prefer release with known bugs? (this may hapoen, but then it should be stated in release notes)

tsmuxer was only 'recently' open sourced (2019) so a lot of time between 2014 and 2019 no one touched it..

[Randrianasulu]

issue 579 for example..

[Randrianasulu]

bug/issue about releasing: https://github.com/justdan96/tsMuxer/issues/502

[shag00]

My retort would be that the release of 29/2/20 appears to be stable and works. It could have been released as a stable build and most people who use TSMuxer would be using it and be happy. A stable release does not mean a perfect release, especially in relation to new hardware (Apple ARM). What is does mean is that proven hardware (X86 processors) and software (Windows 10) can do what the software is designed to do which in essence is to demux and mux raw audio/visual streams fro/to a number of containers, mainly .ts and .m2ts.

Apple has about 12% of the PC market and M1 ARM processors would account for maybe 10% of that which accounts for three tenths of two fifths of **** all and for this no stable release is released? Yes, you are quite correct, known bugs should be advised via the release notes. What, for example, that means is that I can download Ubuntu 22.04 and use it knowing that I cannot dual boot without a work-around but pretty much everything else works and in many cases better than they did in the previous release.

[shag00]

I cut a file down to 5 minutes but it's still 189MB, any good?

[Randrianasulu]

yes, i'll try to download it

[shag00]

where do you want me to send it?

[Randrianasulu]

you can't put it on google drive or microsoft drive? (like other examples in issues)?

you can send link to me directly but then other devs here will not see it....

[justdan96]

@shag00 please be aware we have a code of conduct and we are a team of volunteers, your issue won't be looked at any quicker or more thoroughly by swearing.

Releases is something we have been looking at but as I mentioned we are all volunteers and for me in particular I haven't had much free time at all recently. Please be patient with us.

As for how the regression was missed we rely on the community to test releases and report issues, we don't have any sort of automated testing infrastructure and I'm not sure how we would go about creating some. Any suggestions on that are very welcome.

Can you confirm your choice of font for us please?

[shag00]

@Randrianasulu can you give me a clean e-mail address to use to send you the link.

[Randrianasulu]

@shag00 randrianasulu@gmail.com

[shag00]

@justdan96

congratulations on your code of conduct. If you look more closely I am not swearing, I am using an old saying which contains what some people deem a swear word. I am most certainly not swearing at @Randrianasulu ! There is no need to swear at him because he has done nothing wrong to me or mine. I am as pleased as punch you guys decided to take up this project and all kudos and thanks to you. What I am talking about is the UX.

The rest of your comment falls on deaf ears, we are after all talking about 8 years. When you make these comments do you stop to think about what the listener is thinking or hearing? Even if you have only been involved since say 2019, do you think it makes a difference? I am not suggesting you are lazy or anything of sort. It appears to me you are waiting for a perfect product (which is a bit of a pipe dream) which causes you to delay and delay which ends up causing you even more problems, as is the case here, where a very old error has been uncovered because the bulk of your users are not using dailies and won't upgrade until there is a stable release or they run into some type of problem. It is not as if a stable release has not been requested.

[shag00]

@Randrianasulu link sent

[shag00]

@justdan96 You might consider that I have spent the last 3 hours predominately on this issue and have sought advice from @Randrianasulu to provide the information he was after. This is not something a malcontent would do, is it? I have a working TSMuxer, I don't need you to do anything to allow me to carry on my merry way, no doubt, very much the same position you are in. My motives are similar to yours, to provide a better product.

[shag00]

sorry, the font i am using is ariel

[Randrianasulu]

@shag00 I looked at provided file with mpv and ffplay (2.8.15! so quite old) and mediainfo.. it seems to be PGS subtitle (as intended after tsMuxer?) and.. it looks correct? At least I can't see those little squares you showed in screenshot...

[justdan96]

@shag00 perhaps try using Noto Sans or another Unicode font and see if the same squares appear? I don't have a Windows machine to test on but it's possible a Unicode character is being added to the output which can't be handled by your chosen font. Subtitles are handled differently on Windows and Unix, which may explain why others can't see the same.

[shag00]

@Randrianasulu sorry I have sent a replacement file with srt subtitles

You need to demux and then remux the file. To demux you may need MKVToolnix.

[shag00]

Here is my attempt to use Nato Sans. As you can see it is produces little rectangles in the editing software, Subtitle Edit. In each case the little rectangles are used in place of musical notes, "♪".

Then using this srt file, saved in Noto Sans via the subtitle editor, produces the same result, rectangles at the end of every line.

[jcdr428]

@shag00 can you please post the .srt (or part of it) so that I can see what tsMuxer replaces the music symbol with.

[Randrianasulu]

I used this meta file (tsmuxer do not read eac3 sound from mkv for some reason - might be user error on my side! Ah, sure, wrong type of audio! I construct those in mcedit not in tsmuxer's GUI)

guest@slax:~$ cat /mnt/sdb2/tmp/bd.meta
MUXOPT --hdmv-descriptors --auto-chapters=1
V_MPEG4/ISO/AVC, /mnt/sdb2/tmp/test_file_001.mkv, track=1, contSPS
A_LPCM, /mnt/sdb2/tmp/test_file_001.mkv, track=2
S_TEXT/UTF8, /mnt/sdb2/tmp/test_file_001.mkv, track=3 font-size=64 default=force, video-width=1920, video-height=1080, fps=23.976, font-name="/usr/lib/cin/plugins/fonts/arial.ttf"

guest@slax:~$

and I still can't see squares? But if they appear instead of some (unicode) characters - this point at font/locale/encoding interplay.. I think?

[shag00]

srt.zip

[shag00]

@jcdr428 re the music symbol, I suspect we may have crossed wires here, the music symbol issue relates to Subtitle Edit which is a completely different issue to the TSMuxer issue. Even though Subtitle Edit can not display a music symbol using Noto Sans TSMuxer still recognises it and displays it correctly.

The issue is that TSMuxer is appending something to the end of each line contained in an .srt file. It may be the carriage return that is the issue, I don't know. The whole test was merely to see if unicode font was an issue, which we now know is not the case.

Quite clearly, whatever the problem is is contained in whatever commit(s) was/were added after 29/2/20 to create the release of 1/3/20. I still don't know if anyone else can reproduce the same error? For me it effects every .srt file, I have been downloading from various sources to test and always the same result.

[shag00]

I used this meta file (tsmuxer do not read eac3 sound from mkv for some reason - might be user error on my side! Ah, sure, wrong type of audio! I construct those in mcedit not in tsmuxer's GUI)

guest@slax:~$ cat /mnt/sdb2/tmp/bd.meta
MUXOPT --hdmv-descriptors --auto-chapters=1
V_MPEG4/ISO/AVC, /mnt/sdb2/tmp/test_file_001.mkv, track=1, contSPS
A_LPCM, /mnt/sdb2/tmp/test_file_001.mkv, track=2
S_TEXT/UTF8, /mnt/sdb2/tmp/test_file_001.mkv, track=3 font-size=64 default=force, video-width=1920, video-height=1080, fps=23.976, font-name="/usr/lib/cin/plugins/fonts/arial.ttf"

guest@slax:~$

and I still can't see squares? But if they appear instead of some (unicode) characters - this point at font/locale/encoding interplay.. I think?

Just mux without sound

[Randrianasulu]

@shag00 it works on Slackware with provided srt (even music sign show up!)

I already fixed my meta file for using AC3 sound

edit 2:

MUXOPT --hdmv-descriptors --auto-chapters=1
V_MPEG4/ISO/AVC, /mnt/sdb2/tmp/test_file_001.mkv, track=1, contSPS
A_AC3, /mnt/sdb2/tmp/test_file_001.mkv, track=2
S_TEXT/UTF8, /mnt/sdb2/tmp/Outlander.ENG.srt, track=3, font-size=56 default=force, video-width=1920, video-height=1080, fps=23.976, font-name="/usr/lib/cin/plugins/fonts/arial.ttf"

[shag00]

@Randrianasulu S_TEXT/UTF8, /mnt/sdb2/tmp/Outlander.ENG.srt, track=3, font-size=56 default=force, video-width=1920, video-height=1080, fps=23.976, font-name="/usr/lib/cin/plugins/fonts/arial.ttf"

This is telling me you have not muxed the file with TSMuxwer?

[Randrianasulu]

@shag00 I do not have any other software to turn srt into pgs...

May be I butched this line, still tsmuxer worked and ffplay too...

what caught your eye?

[shag00]

@Randrianasulu

You have totally confused me with, "I do not have any other software to turn srt into pgs...". That is part of what TSMuxer does, turn .srt files into .pgs files. .srt files are a valid input to TSMuxer and have been for almost 20 years. You do not need any other software.

To confirm what we are doing, we are combining an H264 video file with a .ac3 audio file and an .srt subtitle file with TSMuxer. Another format of sound file can be used without affecting the finished product. Once those files have been created no other software is used.

If you are using some other program to create a .pgs file from the .srt file then you likely won't have a succeeding rectangle in your text display because the error is in TSMuxer converting the .srt file to a .pgs file.

What caught me eye? - .m2ts containers do not support srt streams.

[Randrianasulu]

What caught me eye? - .m2ts containers do not support srt streams.

well, this is how tsmuxer works for me - it takes few streams (stand alone or muxed into mkv, or mix of two methods) and while just muxing audio and video it also turn srt into pgs stream, to mux into final ts.

edit:

guest@slax:/mnt/sdb2/tmp$ ./bd.sh
++ dirname ./bd.sh
+ sdir=.
++ cd .
++ pwd
+ dir=/mnt/sdb2/tmp
+ PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R7/bin:/usr/games:/usr/lib/java/bin:/bin:/opt/kde3/bin:.:/root/cinelerra/cinelerra-5.1/bin
+ mkdir -p /mnt/sdb2/tmp/udfs
++ tail -1
++ sed -e 's/[  ].*//'
++ du -cb /mnt/sdb2/tmp/bd.m2ts
+ sz=88
+ blks=4096
+ rm -f /mnt/sdb2/tmp/bd.udfs
+ '[' -f /mnt/sdb2/tmp/bd.meta ']'
+ tsmuxer /mnt/sdb2/tmp/bd.meta /mnt/sdb2/tmp/bd.iso
tsMuxeR version git-41dfd71. github.com/justdan96/tsMuxer
Decoding H264 stream (track 1): Profile: High@4.0  Resolution: 1920:1080p  Frame rate: 23.976
H.264 muxing fps is not set. Get fps from stream. Value: 23.976
B-pyramid level 1 detected. Shift DTS to 2 frames
0.3% complete
H264 bitstream changed: insert nal unit delimiters
Decoding E-AC3 (DD+) stream (track 2): Bitrate: 640Kbps (core 0Kbps) Sample Rate: 48KHz Channels: 5.1
Decoding PGS stream (track 3):  Resolution: 1920:1080  Frame rate: 23.976
H264 bitstream changed: insert SPS/PPS units
96.7% complete
Processed 7200 video frames
100.0% complete
Flushing write buffer
Mux successful complete
Muxing time: 7 sec
+ mv /mnt/sdb2/tmp/bd.iso /mnt/sdb2/tmp/bd.udfs
+ echo To burn bluray, load writable media and run:
To burn bluray, load writable media and run:
+ echo for WORM: growisofs -dvd-compat -Z /dev/bd=/mnt/sdb2/tmp/bd.udfs
for WORM: growisofs -dvd-compat -Z /dev/bd=/mnt/sdb2/tmp/bd.udfs
+ echo for RW: dd if=/mnt/sdb2/tmp/bd.udfs of=/dev/bd bs=2048000
for RW: dd if=/mnt/sdb2/tmp/bd.udfs of=/dev/bd bs=2048000
+ kill 29975
���������