Sauce Comments Do Not Respect Carriage Return / Cuts off Comments

[christianvozar]

Not sure if the spec even allows for it but a quick example of the disparity I'm talking about.

https://16colo.rs/pack/thesauna01/ak67-terminal-colloquy.ans

Open the file in Moebius and the comments are:

 a million suns?
And what will you do in the rain of ash?

I'll draw the blinds and pull down the sash,
And hide from the sight of so many noons.
But how will it be when the blinding flash

Disturbs your body's close-knit mesh
Bringing to light your lovely bones?
What will you wear in the rain of ash?

I will go bare without my flesh,
My vertebrae will click like stones.
Ah. But where will you dance when the blinding flash

Settles the city in a holy hush?
I will dance alone among the ruins.
Ah. And what will you say to the rain of ash?

I will be charming. My subtle speech
Will weave close turns and counter-turns-
No. What will you say to the rain of ash?
Nothing, after the blinding flash

Charles Martin "Terminal Colloquy”

Now I know that Moebius is chopping off some of the beginning of the poem, that's on me as I clearly went over the allowed space for comments.

But the comments on 16c render as:

a million suns? And what will you do in the rain of ash? I'll draw the blinds and pull down the sash, And hide from the sight of so many noons. But how will it be when the blinding flash Disturbs your body's close-knit mesh Bringing to light your lovely bones? What will you wear in the rain of ash? I will go bare without my flesh,

Much gets cut off and line ending isn't respected. Either 16c is just adhering to the standard or Moebius but I'd like to see congruency between the two so I know what things will look like when they release on 16c.

[bart-d]

the problem is the SAUCE spec is not very clear about this, which leads to interpretations. The spec mentions up to 255 lines of comments of 64 characters each. In no way it is mentioned that when displaying comments they should be seperated by newlines. But then again, one could interpreteit "comment line" as being newline separated. You can read for yourself here: http://www.acid.org/info/sauce/sauce.htm

I remember my last suggestion to get this cleared up is contacting the SAUCE author and ask him what his intentions were with regards to displaying comments.

[christianvozar]

Does anyone we know have a way to contact Tazmaniac? I’d be happy to reach out but wouldn’t know where to even begin.

On Jul 1, 2020, at 3:42 PM, bart-d notifications@github.com wrote:

 the problem is the SAUCE spec is not very clear about this, which leads to interpretations. The spec mentions up to 255 lines of comments of 64 characters each. In no way it is mentioned that when displaying comments they should be seperated by newlines. But then again, one could interpreteit "comment line" as being newline separated. You can read for yourself here: http://www.acid.org/info/sauce/sauce.htm

I remember my last suggestion to get this cleared up is contacting the SAUCE author and ask him what his intentions were with regards to displaying comments.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[oreubens]

I would have thought that comment LINES would have been obvious enough :-) Apparently it is not. So yes, the intent has always been to display these as individual lines, and not to be consider as a single big block of continuous text. If that were the intent, it would have been defined differently. Requiring implementers to write an entire text-parsing and word-wrapping engine would have been a bit of a big burden on the developers. (we're talking DOS development afterall)

There should be no control codes in here (Ascii 0-31), but it is not technically prohibited, if present they should be rendered as explicit codes and not treated as formatting. So a CR/LF, if present, technically should be rendered as visible characters (or a pseudo character if the font doesn't support it as a displayable glyph)

[bart-d]

Thanks @oreubens! This appears to be a combination of several issues:

1. Moebius stores comment lines rounded down. This may skip the first comment line from being read. The amount of comment lines in this SAUCE record is 12 while it should be 13. The maximum amount of comment lines is 255, so you're nowehere near the limit. This is the reason why the first line is missing from the output in both Moebius and on 16c. This is clearly a bug in Moebius: https://github.com/blocktronics/moebius/blob/master/app/libtextmode/textmode.js#L86

2. The SAUCE comment for this item contains newline characters. If the specification and the clarification of the author would be followed, you would see double linebreaks in the output. Note 3 of the SAUCE Layout describes allowed characters in Character types (such as comment lines). It's not clear if newlines are allowed here, but since they are used to format, I would guess they are not. In this case Moebius should not add newlines characters and assume other SAUCE parsers will insert linebreaks at the end of the comment line. Quite likely my own misinterpretation has caused the implementation in Moebius to be incorrect.

3. And this also leads to the next issue, 16c's SAUCE parser is using fgets() to read the comment lines, which reads 64 bytes as instructed OR when a newline is read. This causes the reading the be stopped prematurely. When i force the parser to read 26 lines (in stead of 13), it gets read in full. This is a bug in 16c's SAUCE parser.

4. 16c is not inserting (HTML) linebreaks after each comment line when they are being displayed as clarified by Tasmaniac. Note that linebreaks in comment lines are effectively in the output but ignored in HTML. This needs to be fixed on 16c but the question remains if the newlines in the comments itself should also be displayed or not.

imho to fix everything: the rounding issue in Moebius should be fixed, Moebius should skip newline characters in comment lines + space pad comment lines. 16c must add HTML linebreaks after each commentline. Additionally 16c's parser should not stop reading after a newline just to avoid issues.

[bart-d]

16c parser adjusted to continue when newline encountered + newlines converted to HTML breaks when being displayed (https://github.com/16colo-rs/16colo.rs/commit/0d1ab697f0ab4576ea424f74e30f544bd8cf773c) which fixes the issue except the first line is still omitted.

in case newlines were added (which should be to be entirely according to spec), the output would be for the example file:

a million suns?
And what will you do in the rain of ash?

I'll
draw the blinds and pull down the sash,
And hide from the sight
of so many noons.
But how will it be when the blinding flash

Di
sturbs your body's close-knit mesh
Bringing to light your lovely
bones?
What will you wear in the rain of ash?

I will go bare w
ithout my flesh,
My vertebrae will click like stones.
Ah. But wh
ere will you dance when the blinding flash

Settles the city in
a holy hush?
I will dance alone among the ruins.
Ah. And what wi
ll you say to the rain of ash?

I will be charming. My subtle sp
eech
Will weave close turns and counter-turns-
No. What will you
say to the rain of ash?
Nothing, after the blinding flash

Char
les Martin "Terminal Colloquy”

[oreubens]

Hi,

Note that the "intent" of Sauce was to help DOS based viewers in the limited features DOS text had to offer. We now have gotten used to graphical user interfaces that handle text considerably differently than was normal on DOS. It was also written in with Turbo Pascal in mind, which at the time SAUCE was developed was by far the most popular language in use by programmers in the art/demo scene. For this reason, the spec is laid out as "records". And SAUCE should be read assuming the entire file is a binary file. So the sauce data should be read/written from the file using non-formatting file-read/write functions so fgets()... bad idea...

If you have any remembrance of TP... You had 2 kinds of files, text files where TP did the splitting into lines and you only had ReadLine() and WriteLine() and you had to read from the start to the end, there was no positioning. And you had binary files, where you had to do define records (structures) yourself and where you couldn't use the formatted ReadLine/WriteLine.

As for displaying... Again, consider the DOS era. You have a space-padded set of comment lines, and the way to display them would be to just hard-output all 64 characters in succession, regardless of what those characters are. So while "technically" SAUCE does not prohibit the inclusion of tab (0x09), carriage return (0x0d) and linefeed (0x10) if they were in there, a viewer would output those character codes literally. If I recall correctly, 0x09 was a small (centered) circle, 0x10 was the inverse of that, and 0x0d was a (single) musical note. In the first 31 characters there were also the card symbols (hearts, diamonds, clubs, spades), a couple arrows, and some smiley faces. So any of those in SAUCE would be displayed as those symbols.

Given the restrictions of DOS and DOS text... Sauce was speced out with that in mind... Make it easy which would make adoption of the spec by others easier... A SAUCE reader and writer could be written in a couple hundred lines of TP code (I even published those).

Translated to what to do now for display on a web page... I would more or less expect: Read a comment line(s). Right trim/remove spaces. Convert special characters (there are unicode glyphs for all those old control symbols), or you may need to replace them with some other "safe" symbol. HTML-ify the rest. (you'll need to replace &, <, >) add a
at the end ideally... you probably also want to display this with a monospace font, since I've seen a fair amount of sauce comments where they tried some form of formatting or even drawing (ascii logos were popular).

On Sat, 10 Oct 2020 at 20:55, bart-d notifications@github.com wrote:

16c parser adjusted to continue when newline encountered + newlines converted to HTML breaks when being displayed (16colo-rs/16colo.rs@0d1ab69 https://github.com/16colo-rs/16colo.rs/commit/0d1ab697f0ab4576ea424f74e30f544bd8cf773c) which fixes the issue except the first line is still omitted.

in case newlines were added (which should be to be entirely according to spec), the output would be for the example file:

a million suns?

And what will you do in the rain of ash?

I'll

draw the blinds and pull down the sash,

And hide from the sight

of so many noons.

But how will it be when the blinding flash

Di

sturbs your body's close-knit mesh

Bringing to light your lovely

bones?

What will you wear in the rain of ash?

I will go bare w

ithout my flesh,

My vertebrae will click like stones.

Ah. But wh

ere will you dance when the blinding flash

Settles the city in

a holy hush?

I will dance alone among the ruins.

Ah. And what wi

ll you say to the rain of ash?

I will be charming. My subtle sp

eech

Will weave close turns and counter-turns-

No. What will you

say to the rain of ash?

Nothing, after the blinding flash

Char

les Martin "Terminal Colloquy”

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/16colo-rs/16c/issues/67#issuecomment-706595509, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARFP645OOHSHT4I3EMRLEBTSKCUYZANCNFSM4ON53BTQ .

[bart-d]

So the sauce data should be read/written from the file using non-formatting file-read/write functions so fgets()... bad idea...

indeed, that part has been fixed already, fgets() was replaced by fread(). it no longer stops at newline characters now.

So while "technically" SAUCE does not prohibit the inclusion of tab (0x09), carriage return (0x0d) and linefeed (0x10) if they were in there, a viewer would output those character codes literally.

Well, before 'modern editors' (=before PabloDraw) I don't think it was even possible to add a carreiage return or linefeed in SAUCE comments. When I press enter in Spoon II (https://16colo.rs/pack/acdu0896/SPOON.EXE), this just moves to the new line and when the file is saved, there is not CR or LF in the comment lines, they are just space padded. So I guess this would also be the desired behavior in Moebius: enter goes to the next line but newline character are stripped off and comment lines are space padded.

If I recall correctly, 0x09 was a small (centered) circle, 0x10 was the inverse of that, and 0x0d was a (single) musical note. In the first 31 characters there were also the card symbols (hearts, diamonds, clubs, spades), a couple arrows, and some smiley faces. So any of those in SAUCE would be displayed as those symbols. Given the restrictions of DOS and DOS text... Sauce was speced out with that in mind...

Indeed the visible control characters (https://www.ascii-codes.com/) are sometimes still a pain today, especially for TABS and newlines as conversion tools such as ansilove would display them as they would have in DOS/ANSI.SYS while Pablodraw just displays the visibile character.

Convert special characters (there are unicode glyphs for all those old control symbols), or you may need to replace them with some other "safe" symbol. HTML-ify the rest. (you'll need to replace &, <, >) add a
at the end ideally... you probably also want to display this with a monospace font

Those should display just fine currently on 16c as we use the ansilove's BlockZone font which is monospace and contains the visible control characters. I do think however that newlines that ended up in the comment lines should be ignored, leaving just the newline/
after every comment line in place, to avoid the issues as you can see above with double linebreaks.

[bart-d]

test builds with propsosed changes in Moebius available here: https://github.com/blocktronics/moebius/issues/165#issuecomment-712267296