pablo-abc / svelte-markdown

Markdown parser to svelte components
MIT License
360 stars 50 forks source link

Question: Extending marked #17

Open retani opened 3 years ago

retani commented 3 years ago

Is it possible to extend markdown with custom tokens?

marked has the marked.use() method for this (see https://marked.js.org/using_pro#use) but I cound't find it exposed. Or maybe somehow through the options object (see https://marked.js.org/using_pro#extensions)?

And if not, how would one go about implementing this into svelte-markdown?

pablo-abc commented 3 years ago

So, in the current implementation of this package we are only importing the Lexer from marked. Everything else gets tree-shaked out of the bundle. Skimming the docs it seems there's no way to extend it like this from just the Lexer. The options prop in <SvelteMarkdown /> is exactly the same one as Marked's to which you can pass a "tokenizer" or "renderer" property which might already do what you want? If it's not possible like that then I'd be open to add that option somehow.

retani commented 3 years ago

I've tried, but there are a few problems.

  1. Extending tokenizer and renderer using marked.use(options) works, but when passing the options object directly marked(src, options), it seems that the tokenizer and renderer functions get overwritten instead of being merged (this.tokenizer.space is not a function)
  2. I am not sure if it is also necessary to modify the Lexer, because I am not skilled enough in this area. I tried to do it with just using a tokenizer and a renderer, but it didn't work out well. Could be just a lack of knowledge.

BTW what I am trying to do it to implement footnotes. [fn] footnote [/fn] should turn into <sup title="footnote">[1] footnote</sup>

https://github.com/retani/marked-footnotes

image

pablo-abc commented 3 years ago

Thanks for the repo! I'll be tinkering with it. From what I see you're using Marked directly. I'll get back to you as soon as possible.

retani commented 3 years ago

Thanks for checking it out. I was using marked directly to see if I can extend it using just the options. The strangest bit is that it doesn't work without marked.use() at all.

I have found a workaround for my specific problem, but would be good to being able to extend it and knowing how. I realized that extending markdown is something for true regex experts.

pablo-abc commented 3 years ago

Sorry for the delay. I'll look into this. That is really weird. I've been debating on whether I should migrate this to remark instead of marked due to its plugins (and because React Markdown uses remark and I'm trying to recreate a similar API), but sadly I have not even had time to experiment with it.

retani commented 3 years ago

My experience with marked hasn't been great. remark's plugins are quite nice. That's basically all I know :)

raulfdm commented 3 years ago

Hey guys.. sorry to jump in in this conversation but I have a question regarding remark: Will that work client-side?

I'm asking that because I'm diving deep into a remote mdx/shortcodes solution to run client-side and it seems because unified rely on a lot of node apis and package, it won't work exactly like how marked works right now. 🤔

pablo-abc commented 3 years ago

Svelte Markdown is inspired by React Markdown which uses remark behind the scenes and works client side. The reason I used marked was that I wanted a working version ASAP and remark was giving me some issues with rollup.

raulfdm commented 3 years ago

Hmm... gotcha.

Maybe I have to read more about how React Markdown does that... Are you open for contributions?

I'm trying to find a good solution to fetch "mdx" content from a CMS and render truly Svelte components without rely on {@html}.

pablo-abc commented 3 years ago

@raulfdm I am definitely open to contributions. The only issue I see is that with mdx style syntax, Svelte components do need a build step. If you wanted to fetch it in the browser and process it there you'd need to bundle the Svelte compiler with your app which takes away most benefits of Svelte. In this scenario I'd usually just recommend mdsvex.

raulfdm commented 3 years ago

The problem I'm trying to solve is this one:

workflow

But I've encountered a lot of issues which I realized that's because of Vite SSR:

mdsvex is indeed a really good candidate but it's truly focused on local files .svx

Unless I walk through ALL posts via node script and generate those files, It won't work. Even importing the compile method, it would throw the same SSR error.

genericFJS commented 2 years ago

@retani @pablo-abc I've found a solution which enables the use of extensions (but costs an import of marked). See #38 .

genericFJS commented 2 years ago

You can pass extensions to SvelteMarkdown components indirectly by first passing your extensions to marked.use() and then reusing the marked.defaults as the options-parameter for the component. If you want to set other options in addition to the extensions, you may use marked.setOptions() and use the marked.defaults afterwards (or just manipulate the marked.defaults object yourself).

Example: File App.svelte

<script>
  import SvelteMarkdown from "svelte-markdown";
  import LatexMarkdown from "./LatexMarkdown.svelte";

  const latexTokenizerExtension = {
    name: "latex",
    level: "inline",
    start(src) {
      return src.match(/\[\[/)?.index;
    },
    tokenizer(src, tokens) {
      const rule = /^\[\[latex\s*(?:color="(.*)")?\]\]/;
      const match = rule.exec(src);
      if (match) {
        console.log(match[1]);
        return {
          type: "latex",
          raw: match[0],
          text: match[0],
          color: match[1],
        };
      }
    },
  };

  marked.use({ extensions: [latexTokenizerExtension] });
  // marked.setOptions(…)
  const options = marked.defaults;

  const renderers = {
    latex: LatexMarkdown,
  };

  const source = `
  # This is a header

This is a paragraph.

This is red [[latex color="red"]] text.

* This is a list
* With two items
  1. And a sublist
  2. That is ordered
    * With another
    * Sublist inside

| And this is | A table |
|-------------|---------|
| With two    | columns |`;
</script>

<SvelteMarkdown {source} {options} {renderers} />

Imported file LatexMarkdown.svelte

<script>
  export let color;
</script>

<span style={"color: " + color}>LaTeX</span>
AshfordN commented 1 year ago

Has this been tested? I'm trying to do something similar using marked-katex-extension, but I noticed you're passing the extension to marked.use(). Where is this defined? It appears that marked is undefined in your example as well.

AshfordN commented 1 year ago

So far I have this:

<script>
  import { marked } from 'marked';
  import katex from 'marked-katex-extension';
  import Markdown from 'svelte-markdown';

  marked.use(katex({ throwOnError: true }));
  const options = marked.defaults;

  let source = `# Markdown-Latex Example

Inline: $\\frac{4}{3}$

Block: 
$$c = \\pm\\sqrt{a^2 + b^2}$$`;
</script>

<svelte:head>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.2/dist/katex.min.css" integrity="sha384-bYdxxUwYipFNohQlHt0bjN/LCpueqWz13HufFEV1SUatKs1cm4L6fFgCi1jT643X" crossorigin="anonymous" />
</svelte:head>

<Markdown {source} {options} />

But the LaTeX portions are not displayed, as shown below: Screenshot from 2023-04-08 10-17-52

I'm not sure what I'm missing

AshfordN commented 1 year ago

Sorry for the back-to-back posting, but the following code seems to work:

File: App.svelte

<script>
  import { marked } from 'marked';
  import katex from 'marked-katex-extension';
  import Markdown from 'svelte-markdown';
  import KatexRenderer from './KatexRenderer.svelte';

  marked.use(katex({ throwOnError: true }));
  const options = marked.defaults;
  const renderers = { blockKatex: KatexRenderer, inlineKatex: KatexRenderer };

  let source = `# Markdown-Latex Example

Inline: $\\frac{4}{3}$

Block: 
$$c = \\pm\\sqrt{a^2 + b^2}$$`;
</script>

<svelte:head>
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.2/dist/katex.min.css" integrity="sha384-bYdxxUwYipFNohQlHt0bjN/LCpueqWz13HufFEV1SUatKs1cm4L6fFgCi1jT643X" crossorigin="anonymous" />
</svelte:head>

<Markdown {source} {options} {renderers} />

File: KatexRenderer.svelte

<script>
  import Katex from 'svelte-katex';

  export let text;
</script>

<Katex>{text}</Katex>

Something about this feels a bit inefficient though, so if anyone has any ideas for improvement, I'd appreciate the feedback.