Rewritten core TypoScript parser

[lolli42]

I'm creating this issue to leave some information about current TYPO3 core development that happened in the last month for version 12 and raise awareness in this area. I guess it's fine to close the issue again, but I hope it's ok to outline a bit.

TYPO3 v12 comes with a rewritten TypoScript parser, deprecating the old approach.

First, I want to give a huge "Thank You!" to Martin for this repository: I know this parser is used by various people to parse TypoScript a-like syntax as a standalone solution for various use cases. It also inspired the new core solution and gave me another source to look up details. As such, the new core parser follows similar principles: A tokenizer, a solution to parse "@import" statements into a tree, an AST builder, together with tree traversers and visitors.

While the new parser is currently fully marked @internal in the core, with only the consuming code (TypoScript, pageTsConfig, userTsConfig) being public API, the core parser could still be used to achieve similar things this repository provides, in case people don't hesitate to access internally marked core classes. The parser "library" is part of the "core" extension, it can be composer-required directly, to then use the classes. The core extension has no direct dependency to other core extensions like "backend" and similar, so requiring "core" still loads various dependencies, but does not end up in a fully working TYPO3. Eventually, when dust settles, it would be possible to extract the TypoScript parser to a standalone library as well. Details like DB queries are already relatively well contained and are only a semi-hard dependency at the moment, possibly with further extraction.

A couple of details have not yet been implement in the core approach - for instance a "PrettyPrinter" is missing. This may be added later though, improving the tstemplate extension with features.

The new parser also streamlines some TypoScript syntax details. For instance using a 'constant' within an identifier is no longer allowed: foo.{$bar}.baz = something. This will apply to all TYPO3 projects with v12 and above.

I'm writing this since I see various options to continue with this repository. I'm fine with all of them and I'm absolutely not saying this approach is obsolete in any way:

• Keep everything as is and continue with the codebase. That's of course totally fine!
• Retire this repository and point people to the core solution, maybe creating some real examples along the way.
• Join forces on the new parser to add eventually missing features this repository provides like a pretty printer and maybe more.
• Turn this repository into a subtree-split of the new library, eventually enriched with additional visitors for a low-dependency library.
• Create a new library containing the new parser and join forces on it, extracting the parser from the core.

Martin, please feel free to contact me (eg. via TYPO3 slack) in case you're worried and interested, I'm open for everything. Feel free to continue as is if that fits your needs and wishes, too: Your work on this parser is great and I totally don't want to destroy anything of it.

[martin-helmich]

Hey Christian,

thanks for reaching out. I'll respond here instead of Slack, so that others may read along and chime in. Apart from that, it might also be easier to have a quick chat about some things, so I might also give you a poke on Slack shortly. 🙂

This project originally started as a programming excercise many moons ago; It was fun to build, but I don't have any emotional attachment to it and only maintain it out of necessity for the typo3-typoscript-lint^1 project.

Actually, keeping the parser in sync with the "actual" implementation in the TYPO3 core has always been a hassle that I would not mind getting rid of. So, replacing this library with an "official" parser that offers the same capabilities does -- in general -- seem favourable to me (with some caveats being listed below).

Keeping this in mind:

The parser "library" is part of the "core" extension, it can be composer-required directly, to then use the classes.

THAT, on the other hand, does not sound particularly appealing to me. I would prefer not depending on any internal APIs that might break without any notice. Also, the typo3-typoscript-lint project is often composer-(dev)-required into many TYPO3 projects, so it might be a potential issue if the linter requires its own version of the TYPO3 core as a dependency. Although, the PHAR installation is the recommended mode of installation anyway, so that second argument might be a non-issue.

A couple of details have not yet been implement in the core approach - for instance a "PrettyPrinter" is missing.

That doesn't sound too bad for me. Once could argue that printing code is actually a separate concern from parsing it, in any case. So, maintaining a separate library with printers and other "add-on" features, might actually be preferable.

Some additional caveats that I could think of stem from the usage of this parser library for the Typoscript Linter:

• The Linter requires access to some low-level data structures of the parser (like the token stream, for example) that some other parser library might decide not to export as part of its public API.
• For usage in a Linter, it might be preferable to have a more lenient parsing mode; the actual TYPO3 parser might reject some things as syntactically invalid (which is fine), but a linter might want to continue despite any (recoverable) parsing errors.

---

So, regarding to the suggested alternatives:

I would prefer not depending on the TYPO3 core itself. I would prefer having a separate library without any (unrelated) dependencies and with a stable API contract that I could either depend on directly in typo3-typoscript-lint, or to shim in a new version of the typo3-typoscript-parser library.

TL;DR: My make-a-wish solution would consist of us joining forces and creating a unified standalone library that both the TYPO3 core and the linter could require as a dependency. My time to work on this project has been limited in the recent past, but I'd be happy to join forces and help as my spare time allows.

Maybe we can schedule a quick call on Slack to hash out some more details. I'll leave this issue open (and pin it) for a while, in case anyone else wants to chime in.

[sabbelasichon]

@lolli42 @martin-helmich Thanks for all your work and starting this discussion here. As we are creating a new tool to modify also TypoScript files to ease the upgrade of TYPO3 (https://github.com/andreaswolf/fractor), i would also like to see a dedicated composer package for the TypoScript part instead of relying on the core package itself. What has to be done to move forward? I would be happy to help and join forces.

[lolli42]

Hey.

Thanks for feedback so far.

I understand having the core parser as standalone library would be a good thing to have. Simon Schaufelberger pinged me on this recently as well. I essentially answered this. Note: Opinions below!

• The core parser is currently bound to core at a few places, and uses various API provided by ext:core (starting with various helper stuff from utilities).
• The core parser is currently internal and actively receives the one or the other change (like recently the support for null coalesce constants).
• The core parser needs to stay within the core monorepo since extracting this as 3rd party library in a different repo is a hindrance for core development. We recently merged ext:styleguide into core to resolve the hassle with cross-patches in this area. Unfortunately, the monorepo has currently no support for true libraries (as-in: not typo3 extensions), which is why testing-framework has not been merged as a library, yet. This is on our list, but requires a series of infrastructure changes that have not been done, yet. In the end, we'd first merge testing-framework, to then see if turning the TS parser into a library is feasible in a second step. I'm all in for that, but we're not there, yet.
• Turning the core TS parser into a library requires various patches to further separate stuff. The factory classes for instance are typo3-application classes, and not parser classes. The dependency to the sys_template records on the outer facing classes may need adaption as well.
• I actually see no real issue with requiring typo3/cms-core to receive the current version of the parser: typo3/cms-core is rather library-like, and does not "do" much in itself. If the only problem is that such a requirement gives you more code than you actually want, then I'd still pick that as a (temporary) drawback. I personally would try to work in this direction, since you receive the advantage of having a 'current' parser at hand at any point in time.
• If above is out of option for you, I suggest creating a meta-repo that can extract the parser from core using some scripting that can be re-run. You can of course very well maintain a different parser, but you'll be faced with the situation that it needs to keep up with core and it's continuously increasing feature set: The new parser is still "young" and we did by far not implement everything possible that is on our radar. Also the code base is much more flexible now, which allows us to implement features more easily. For a third party consumer like a ts-parser solution for rector, I'd favor the core implementation as well, even if details are still subject to change (we will for instance change the visitors again to make them stateless).

[sabbelasichon]

@lolli42 Thanks for your detail explanation. I am not against requiring typo3/cms-core in Fractor by no means. The only concern for now is that we would like to support a great range of TYPO3 versions in the same branch. That is not easily doable if we would require typo3/cms-core as far as i understand. Let us have a direct conversation so we can see what we can achieve in the current situation. Thank you very much for your passion and work.