Parser Inconsistency for Several Math Expressions

[xiaodong-hu]

Hi, after several days trial of mitex, I found several annoying inconsistencies, even for standard math notation:

1. In math, we use square bracket [ and parenthesis ( to represent the close and open set. But when I try to convert my notes including something like $[0,1,\cdots, N_s)$ blablabla, I got serious parse error (collapse to one line)

   It seems because the parser found some unclosed/unpaired bracket. If so, that is really stupid to me (we should be free to use any symbol in math mode).

2. The vertical line in latex is somewhat parse to wrong symbol, for example, Bloch states$|\boldsymbol k\rangle$ is wrongly parsed by mitex: I got extra spaces between k and the vertical line. But if I try to make it converting from $\ket{\boldsymbol k}$ by explicitly telling mitex what the composite symbol is (here is the ket state), then the conversion is OK.

In latex at least $|\boldsymbol k\rangle$ and $\ket{\boldsymbol k}$ looks the same, without knowing the meaning for the composite symbol |...\rangle, and there are indeed some situation when $\bra{...}$ or $\ket{...}$ or $\braket{...}$ cannot handle, that is why most people still control the size of vertical line | and the size of angle bracket \langle and \rangle by hand.

[OrangeX4]

1. I tried

#import "@preview/mitex:0.2.1": *
#mitext(`$[0,1,\cdots, N_s)$`)
#mitex(`[0,1,\cdots, N_s)`)

and found no parse error, can you provide more details?

2. duplicated with https://github.com/mitex-rs/mitex/issues/49

[xiaodong-hu]

Nope, try this: (I just quote the relevant paragraph from my notes, please take close look on several last several words in item/paragraph 1)

#import "@preview/mitex:0.2.1": *
#set math.equation(numbering: "(1)", supplement: [#text(blue)[*Eq.*]])

#mitext(`
    So what is the natural way to fix the phase ambiguity here? Here is the logic:

    1. Recall that the momentum get quantized totally from the periodicity of the real-space. So for a real-space lattice of size $[N_1,N_2]$, so must be the reciprocal lattice: $[N_1,N_2]$. Namely with the label $\boldsymbol k\equiv\boldsymbol k_{(m_1,m_2)}$, we know that $m_1\in[0,1,\cdots N_1)$ and $m_2\in[0,1,\cdots,N_2)$.

    2. Now we must also specify $N_1$ different eigenvalues for the first slot (and $N_2$ different eigenvalues for the second slot). So it's natural for us to **equally distribute** $2\pi$ for each slot.

    3. Becuase there can be extra global phase twists for Haldane-Rezayi wave functions from $t(\boldsymbol L_\alpha)\psi_{\text{HR}}(z)\equiv t(a_i)^{N_i}\psi_{\text{HR}}(z)=e^{i\phi_\alpha}\psi_{\text{HR}}(z)$, i.e., we can also choose to **equally distribute** such phase twists for each Bloch state.
`)

Although here is markdown syntax, it should display as normal paragraphs in latex. However, in mitex, after the paragraph of item 1, everything after that is crushed (collapse in one line of infinite length). And if I change the open domain to close domain like we know that $m_1\in[0,1,\cdots N_1]$ and $m_2\in[0,1,\cdots,N_2]$, then there is no problem.

[OrangeX4]

it is strange, minimal reproducible example:

#import "@preview/mitex:0.2.1": *
#mitext(`
$[)$

test
`)

maybe @Myriad-Dreamin could also have a look.

[Myriad-Dreamin]

First, we feel appreciated that you take time to report these issues. I bring some response as follow.

1. In math, we use square bracket, ... simply because the parser found some unclosed/unpaired bracket. That is really stupid...

mitex is well tested by fixtures extracted from oiwiki, which always use mimath (math mode), while you are using mitext (text mode), which is not well tested.

Considering that, the introduction in README.md are misleading, since it wrongly tells mitex in text mode is also well tested. The introduction was correct as I wrote it when we only implemented math mode. However, once after introducing text mode, there should be quite misleading.

Therefore, I think we should keep a warning for that until we have tested a large set of papers on text mode. But I also feel a bit offended that you emphasize that mitex has a stupid parser.

2. ...

It is probably a bug in converter.

[Enivex]

If the parser or tokenizer dost contains such stupid paired brackets check, and cannot be fixed by escape ANY POSSIBLE symbols (because in math we are free to choose symbols), then I should say mitex is far to be used by the math/theoretical physics community...

Dude, chill. This is a very new third party plugin for a typesetting system still in beta. The maintainers do this out of passion for the project, and owe you nothing.

Issues won't get fixed sooner just because you act like this in your bug reports. If anything, it might have the opposite effect.

[xiaodong-hu]

Hi all, apologize for being raging on my issue yesterday.

I got emotional like this because I spent several days converting my notes from markdown to typst, giving my enthusiasm and confidence to typst and mitex package. I understand that there can be minor issues during the process (like the vertical line reported here), but realizing there seems to be "paired brackets checker" within expression $...$ in mitext really makes me clownish, indicating a full waste of time for myself. That's why I got mad yesterday, and used the word "stupid" to describe the parser...

mitex is well tested by fixtures extracted from oiwiki, which always use mimath (math mode), while you are using mitext (text mode), which is not well tested.

I see, thanks for the clarification.

Anyway, respect for all your work and appreciated. I apologize for my tone and will edit the issue to description only.

[Myriad-Dreamin]

Hi all, apologize for being raging on my issue yesterday.

Thank you for your understanding.

I have surveyed the behavior of overleaf, pandoc and several other tex parsers. Then I make a PR to fix the first point of this issue. Luckily, it is easy to fixed. The "bracket/paren group parser" was used as part of argument parsing (e.g. [3] in \sqrt[3]{4}, or (w,h) in \dashbox{x}(w,h)[p]{text}). Now, we can fix the issue by removing these "group parser", because their functionality are replaced by the "command argument parser".

---

I got emotional like this because I spent several days converting my notes from markdown to typst

oh, so you can basically use mimath instead of mitext. I haven't checked your converter but I guess you're lazy and put some entirely markdown code in mitext. That's the reason that you also said:

Although here is markdown syntax, it should display as normal paragraphs in latex.

I think you may be interested in the following tool, they have converted a 2500-page markdown document into equivalent typst one with a huge number of equations.

https://github.com/OI-wiki/OI-Wiki-export/actions/runs/7945608285

Among the equations, there are cases containing the unbalanced braces:

[xiaodong-hu]

Thanks for your efforts on fixing. It's great to hear to be resolvable.

Yes I am trying to move from markdown to typst writing my academic notes (along with overleaf), and I already wrote a typst template similar to physical review journals, as a preparation.

Now I indeed just simply throw every jupiter cell (markdown) to mitext and fix the rendering results one by one, because the number of math symbols is huge. It is kind of boring but OK to me. This is not only because I am lazy (a script can be helpful to wrap every single math with #mimath(...), but there are also plenty of equation, align etc environment), but also a test for future workflow:

In fact, the only reason I choose typst (and previously markdown) rather than latex to write notes is its instant interactivity, giving the negligible compilation time. This is attractive to me because during the research time the notes can be frequently changed, and full derivations require heavy math (where markdown can be slow). But here is another crucial point: the syntax should be the same if I am wrapping up the final paper --- the simplest way is to copy the source code from my notes to my latex draft.

So clearly the suitable workaround is what I am doing now --- wrap a large part of latex with #mitext(...) --- then I can take advantage of both typst's speed and latex's interactivity with my collaborators and publishers : )