Closed mgeisler closed 3 months ago
@moaminsharifi, the underlying problem here is that I thought we could normalize a translation like
msgid "foo bar"
msgstr "FOO BAR"
by running mdbook-xgettext
on both of "foo bar"
and "FOO BAR"
and then matching up the results pairwise. However, this breaks down when either field contain more/less parts on purpose.
We can throw in as much custom logic as we want in the normalization tool: I consider it a bit of free space where we clean up our past problems :slightly_smiling_face: However, it would be nice if we could find a clean way to keep the inline HTML intact in general.
@dyoo, you've run into similar problems with the parsing of HTML comments. Do you have an idea of how we can stop splitting out the inline HTML?
This is related to #97 as well: when we extract messages from a Markdown string, we get multiple messages if the Markdown contains HTML. So
foo <span>bar</span>"
becomes foo
and bar
. This causes the normalization problem above.
Since HTML is very rare in my experience, it would probably better to not split it out like that. So maybe we should just let Text
and Html
nodes become part of the same Group
.
@kdarkhan, did you want to look into this too? I think it's basically a duplicate of #97 so fixing one will probably fix both :smile:
With the pulldown-cmark upgrade the span tags are detected as InlineHtml
. This allows them to be grouped into the same translation group while other block level Html
tags are still handled as before.
The change to InlineHtml
also affects these tags:
<!-- mdbook-xgettext:skip -->
- this can be detected and handled as before<script>alert('there');</script>
- this needs further discussion how tags like these should be handled.There are two ways this could go:
a) Handle all InlineHtml
tags (that are not mdbook directives) in the same group (this would then not split the <script>
tags)
b) Allowlist or Denylist for tags that should be kept in the same group as text.
With the statement above that HTML is rare in these translations I would go for solution a). Only if Html tags are already used in a meaningful way in other places, maybe b) with an allowlist for e.g. span (or other "stylistic" tags) would be better to keep everything compatible. From my perspective it would be a explicit decision to support (some) Html in the Markdown, and I'm not sure how useful this feature would be at this moment.
Any thoughts on this @mgeisler?
Hi @michael-kerscher,
Thanks for looking at this!
With the statement above that HTML is rare in these translations I would go for solution a). Only if Html tags are already used in a meaningful way in other places, maybe b) with an allowlist for e.g. span (or other "stylistic" tags) would be better to keep everything compatible. From my perspective it would be a explicit decision to support (some) Html in the Markdown, and I'm not sure how useful this feature would be at this moment.
I would also lean towards solution a). The less opinionated we have to be the better in my opinion :smile: The amount of HTML allowed in the Markdown is ultimately a decision for the people who use the tooling here, so I don't think we need to guard against JavaScript tags or similar.
<!-- mdbook-xgettext:skip -->
- this can be detected and handled as before
Great! Just for my own curiosity, does a HTML comment like this in its own paragraph show up as InlineHtml
or as Html
now?
A related question would be what happens to Markdown wrapped in a large div
element? Like this:
<div class="warning">
Beware of the dog!
</div>
I haven't tested it, but I expect us to extract "Beware of the dog!"
today. It would be a shame if we suddenly end up extracting the whole thing as a self-contained unit.
Great! Just for my own curiosity, does a HTML comment like this in its own paragraph show up as InlineHtml or as Html now?
A HTML comment in its own paragraph is still showing up as HtmlBlock so there is no change: see this link https://spec.commonmark.org/dingus/?text=%3C!--%20mdbook-xgettext:skip%20--%3E%0AHi%20from%20%3Cspan%20%3ERust%3C/span%3E
But if it is used inline it is detected as an Inline Block.
To give a bigger example:
<!-- mdbook-xgettext:skip -->
This comment is a Html Block and this text would be skipped
This is printed and the comment is an Inline Block <!-- mdbook-xgettext:skip --> this would be skipped
This is not skipped
The pull request has further details and examples on this.
This is from https://github.com/google/comprehensive-rust/issues/1471: running
mdbook-i18n-normalize
on a translated message that contain HTML gives the wrong output.I can reproduce this with
This tests normalizing a catalog looking like this:
The test fails with
which tells us that the normalized catalog contain two messages:
In other words, the original message was split into two messages.
In case only the translation contain HTML, the normalization will end up with a different number of messages: 1 for the
msgid
field, and 2 for themsgstr
field. This is seen as an error, so a fallback kicks in: the normalized message is marked fuzzy and we accumulate the "left-over" messages into the final message.This behavior can be seen in this unit test in
normalize.rs
:This is what happened in https://github.com/google/comprehensive-rust/issues/1471 where we transform
into