How to make a custom tokenizer?

[hgoona]

Hi! This package works great but I'm having trouble figuring out how to customise the rendering when a special character is detected, eg. If "%%%" is detected in the markdown string, I would like to swap that part of the string with a custom svelte component. 🤞Is that possible?

By chance is there an example? I was unable to find more info online about this especially with this package..

I'm on Sveltkit 1.22, with Typescript and Tailwind CSS.

[ssssota]

Thank you for your question!

If we want to use custom syntax, we can make own remark plugin. I made an example, please check it: https://stackblitz.com/edit/svelte-exmarkdown-custom-syntax?file=src%2FApp.svelte

[hgoona]

@ssssota Thanks so much! How can i modify this to include the strings that follow the token, and terminate it in my custom component?

eg. md string: "normal text string %%% special text that is inserted into the custom md component /%%% << this ends the special stuff and returns to normal markdown behaviour"

So special text that is inserted into the custom md component is used in the custom component. Is this how it would be done??

[hgoona]

Hi @ssssota! Any ideas on this scenario? I saw this video do exactly this, but the package they used has deprecated dependencies with SvelteKit so I had trouble getting that to work.. youtube video: https://www.youtube.com/watch?v=ZiEROAqobwM&t=833s

[ssssota]

Sorry for the late reply. I'll check it soon.

[ssssota]

I have updated an example: https://stackblitz.com/edit/svelte-exmarkdown-custom-syntax?file=src%2Flib%2Fcustom-syntax-plugin%2Findex.ts

[hgoona]

I have updated an example: https://stackblitz.com/edit/svelte-exmarkdown-custom-syntax?file=src%2Flib%2Fcustom-syntax-plugin%2Findex.ts

I just took a look! Amazing!! Is there a way for it to maintain the markdown rendering inside the custom component as well?

Say the input is this: """# Hello svelte-exmarkdown

normal text string %%% special text that is inserted into the custom md component /%%% << this ends the special stuff and returns to normal markdown behaviour

More text and %%% 2nd special text with bold stuff /%%%` """

the bold in the 2nd input seems to interrupt the custom tag - if that makes sense?

[ssssota]

I've tried, but still can't do it. I'll try another technique.

[hgoona]

Oh really? :(

I found a repl that manages to achieve it with Markdoc package and allows types: https://svelte.dev/repl/39fdc4d5e7af4c9ea0712aef6204ebba?version=3.48.0 ^Don't know if that might be helpful?? I think your package works slightly different..

It does however add a weird addition character to the end of code blocks rather than AFTER the code block is ended🤔I don't know how to fix that in Markdoc..

[ssssota]

@hgoona I did it! And I have updated an example. Please check it. https://stackblitz.com/edit/svelte-exmarkdown-custom-syntax?file=src%2Flib%2Fcustom-syntax-plugin%2Findex.ts

[hgoona]

@ssssota Woohoo!!! Awesome!! Thank you so much 💪🏾💪🏾💪🏾 Was there a tutorial or something that explained how you got the solution? I noticed you used rehype??

[ssssota]

Sorry, I didn't follow any specific tutorial or documentation. As you noticed, I customized it by creating a rehype plugin based!

https://github.com/ssssota/svelte-exmarkdown/blob/c9411bec33b0ee27d8293c2c1e507fa276c9aac6/docs/02-plugin.md As the diagram in this document shows, Remark plugin edits Markdown ASTs, Rehype plugin edits HTML ASTs.

Initially, I thought I would use the Markdown AST (mdast) because we needed to edit the Markdown AST to add syntax. However, I decided to use the rehype plugin because I thought that editing the HTML ASTs (hast) would be sufficient.

[hgoona]

Noted! I'll close this one - thanks so much @ssssota 💪🏾