Support for remote MDX actually exists now (and plugin is not part of Algolia search)

[hoijnet]

Preliminary Checks

• [X] This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
• [X] This issue is not a question, feature request, RFC, or anything other than a bug report. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Summary

I created a new plugin, gatsby-mdx-remote, which is published onto npmjs, and also has additional users it works for. The documentation on gatsbyjs.com and npmjs for gatsby-plugin-mdx probably should be updated to provide a link once the Algolia plugin search is resolved.

This is the specific text in the gatsby-plugin-mdx text that is inaccurate:

Loading MDX from other sources as the filesystem is not supported. If you have a need for that, please comment in the GitHub Discussion

Also the refresh of the Gatsby plugin list does not list the new plugin after about one week, unsure if there is an error, as my other added plugin, @dfrnt/gatsby-plugin-graphql-nodes is also not listed yet (generic plugin to source nodes from any graphql endpoint, and is very usable with the previous mdx plugin).

Steps to Resolve this Issue

1. Update Algolia search repo to include the two new plugins
2. Update gatsby-plugin-mdx, that there is community support to create mdx from remote sources, and add a link to the community plugin for arbitrary nodes - remote mdx support: gatsby-mdx-remote

[LekoArts]

Hi, thanks for the issue!

1. We don't have control over this, we use https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby-source-npm-package-search which uses https://github.com/algolia/npm-search. It seems to have long indexing times: https://github.com/algolia/npm-search/issues/1112
2. I'm vary of adding a recommendation without us actually having a look at it. I don't want to recommend something that will be a footgun over the long run or which uses anti-patterns.

[hoijnet]

Thanks Lennart, looks like a broad problem. And thanks for the reference, then I know it's on their end!

Of course! I have tried to follow the latest standards and authoring it with Typescript and making it as focused as possible (which is why I split the problems into two parts). Let me know if there is anything I can do with it to better adapt it to community needs, it's my first two plugins to the Gatsby ecosystem.

[LekoArts]

Do you have an example site I can just clone and run to test the plugin out?

[thiagorossener]

Hi again @hoijnet, thanks for working hard on that plugin. It's extremely necessary since gatsby-plugin-mdx supports only files now.

I have the exact same scenario than @rsaarloos here. We were using <MDXRenderer> from gatsby-plugin-mdx v3 to render MDX content from Contentful and it's no longer possible.

I'd like to ask you a question. Does your plugin work with Gatsby Cloud incremental builds?

I've been having a hard time to make Contentful + MDX work with that. I was able to grab MDX content from Contentful, create .mdx files and use them like the documentation recommends:

component: `${postTemplate}?__contentFilePath=/path/to/my-generated-file-from-contentful.mdx`

However, when previewing content in Contentful, the preview works for the first run and fails with a run out of memory error due the incremental builds feature. It's frustrating since I already spent over 3 entire days trying to fix it.

This is the schema for incremental builds:

@LekoArts if you know how to make Gatsby Cloud previews work with Contentful + MDX, it'd be super helpful and we'd much appreciate it.

You can see how Gatsby Cloud previews work for the first run and fails for the second one here.

What we do is create a File node using crateFileFromNodeBuffer from gatsby-source-filesystem. We do that in onCreateNode after gatsby-source-contentful. So, for each text/markdown type from Contentful, we create a File node pointing to a .mdx file.

You can see in the Gatsby Cloud logs something like that in the first run (the one that works):

18:52:47 PM:
info Creating node file... 2dce3d24-ea28-53e7-93be-bd04ed152cbe

18:52:47 PM:
info Creating node file... f301e253-32ee-508c-abf7-557b3b709d06

18:52:47 PM:
info Creating node file... ff739ae5-25cf-56ea-93ae-15e988f2147e

18:52:47 PM:
info Creating node file... 5c943e8f-bdf1-5a04-bdc0-bd704924da22

18:52:47 PM:
info Creating node file... ca5e4db5-0563-5cee-b81d-4662a8133863

18:52:47 PM:
info Creating node file... a19153b1-efdf-5c43-88ad-38d8d96e596b

18:52:47 PM:
info Creating node file... 76faabd8-1df5-5473-9a6c-ebced6ca384f

18:52:47 PM:
info Creating node file... 8c14d93e-6838-5677-9d20-0c25a64801c6

And a log like that for the second run. Note that the File node is created only for the Contentful blog node that got updated. I don't understand why it's running out of memory:

18:57:33 PM:
success Contentful: Fetch data - 0.392s

18:57:33 PM:
info Contentful: 1 updated entries

18:57:33 PM:
info Contentful: 0 new entries

18:57:33 PM:
info Contentful: 0 deleted entries

18:57:33 PM:
info Contentful: 531 cached entries

18:57:33 PM:
info Contentful: 0 updated assets

18:57:33 PM:
info Contentful: 0 new assets

18:57:33 PM:
info Contentful: 0 deleted assets

18:57:33 PM:
info Contentful: 1215 cached assets

18:57:33 PM:
verbose 295.166095809 Building Contentful reference map

18:57:33 PM:
verbose 295.214527969 Resolving Contentful references

18:57:33 PM:
info Creating 1 Contentful Blog nodes

18:57:33 PM:
info Creating node file... 2dce3d24-ea28-53e7-93be-bd04ed152cbe

18:57:33 PM:
success Contentful: Create nodes - 0.044s

For now, we decided to drop MDX completely and are converting all Contentful blog posts (~120 posts) to common markdown.

[hoijnet]

Do you have an example site I can just clone and run to test the plugin out?

I have set aside some time in the coming days to provide a demo repo to clone and help people get started!

I'll add a link here and a ping once updated 🎉

[hoijnet]

I have the exact same scenario than @rsaarloos here. We were using <MDXRenderer> from gatsby-plugin-mdx v3 to render MDX content from Contentful and it's no longer possible.

I'd like to ask you a question. Does your plugin work with Gatsby Cloud incremental builds?

I have not tried as I don't have any incremental source plugin to try with yet. As I use the cache I think the standard mechanisms should work, but it's outside what I have tested and worked with in the past.

Thanks a lot for clarifying the flow. Could be a good first step after the demo repo is released. Bit unsure how to get started. Feel free to massage your post into an enhancement request to help track it.

I believe I will need more help to test incremental parts as things stand now.

[hoijnet]

What we do is create a File node using crateFileFromNodeBuffer from gatsby-source-filesystem. We do that in onCreateNode after gatsby-source-contentful. So, for each text/markdown type from Contentful, we create a File node pointing to a .mdx file.

It sounds like this is much of what the plugin is automating, could be worth to collab and fix together 😄 The plugin source should (IMHO) be fairly straightforward. As long as the cache cleans up downstream nodes, normal semantics should apply I hope.

[hoijnet]

Do you have an example site I can just clone and run to test the plugin out?

@LekoArts Yes, now I have! Thanks for looking into it! It was easier than I thought to create a starter repo for it, and it creates a single node programmatically to keep things super simple:

• gatsby-starter-mdx-remote-hello-world

Or just go with below. README.md is updated too for others to follow the instructions and get started.

gatsby new my-mdx-remote-hello-world-starter https://github.com/dfrnt-com/gatsby-starter-mdx-remote-hello-world

Kept it as simple as possible. Do you think it makes submit it as a separate starter, considering its bare-boned-ness? I'm thinking of adding a graphql source too and gauging whether to keep it all together in one repo, or have two separate ones, leaving this as-is to just show how to get started with the remote MDX part?

Also added a single <YoutubeEmbed/> for good measure as it shows how to embed a local file. I believe most would included a component library, but for a full example it's probably better to include it.

[jejebecarte]

@LekoArts Out of curiosity, how does https://www.gatsbyjs.com/docs pull its content? Given its hosted in the monorepo, which is not the same location as the source for the website, as far as I'm aware.

I know gatsby-source-git exists, but it's not maintained and has some pretty critical bugs. I'm just looking to pull MDX content from a GitHub repository...

[LekoArts]

We use https://github.com/stevetweeddale/gatsby-source-git

[jejebecarte]

Interesting. I've tried using but got a series of unhelpful errors, unfortunately

[hoijnet]

The new plugins are still not updated on the GatsbyJS website. This feels strange!

[hoijnet]

Or just go with below. README.md is updated too for others to follow the instructions and get started.

gatsby new my-mdx-remote-hello-world-starter https://github.com/dfrnt-com/gatsby-starter-mdx-remote-hello-world

@LekoArts, did get a chance to get it up and running? Anything I can help with?

[LekoArts]

No, unfortunately not. I hope to get to it this week with my colleagues

[LekoArts]

So I've given the code and the repository a look & try. Yeah, it kind of works, but I don't think this will scale and work correctly all the time. That it doesn't work properly with Incremental Builds is already a sign of it that the data tracking of those nodes is not set up.

I think MDX from remote sources should be totally separate from gatsby-plugin-mdx, so do not try to somehow make things work with gatsby-plugin-mdx and create a transformer plugin: https://www.gatsbyjs.com/docs/how-to/plugins-and-themes/creating-a-transformer-plugin/

I think much inspiration can be taken from https://github.com/hashicorp/next-mdx-remote

[hoijnet]

Thanks @LekoArts for the review! It's a first plugin attempt. Your pointers are super helpful to take the next steps with it. Your point about transformer plugins is unclear. It sounds like transformer plugins could be a good idea to align to? Your working make it sounds like it is not a good idea?

As for the linked library I have reviewed it and it uses eval to render on the client side with MDXRemote which can be dangerous from XSS perspective.

I think much inspiration can be taken from https://github.com/hashicorp/next-mdx-remote

We operate in a strict-dynamic CSP environment and I avoid eval as much as I can. Their note on security is clear:

This [next-mdx-remote] library evaluates a string of JavaScript on the client side, which is how it MDXRemotes the MDX content. Evaluating a string into javascript can be a dangerous practice if not done carefully, as it can enable XSS attacks.

Fixing the issues of incremental builds in the plugin would be helpful I think. If aligning with a transformer plugin makes sense I think I will continue the exploration and learn some more, but incremental builds is so far new territory for me. Hopefully I can get some help testing from others that are interested in getting this working ☀️

@thiagorossener was the incremental build issues with the gatsby-mdx-remote plugin or your own code? Sounds like it was in your own code and you were asking if this plugin would help you?

I believe the parent/child relationships are setup (hopefully correctly too), if there is a good test suite @LekoArts tontest with Inciuld run it through them and test!

So key question is whether there are significant roadblocks to foresee doing MDX with the plugin, using a transformer pattern?

[LekoArts]

Your point about transformer plugins is unclear. It sounds like transformer plugins could be a good idea to align to?

Yes, for your use case there should be a gatsby-transformer-mdx that does the work equivalent to https://github.com/hashicorp/next-mdx-remote

The current gatsby-mdx-remote is doing a lot of work to make gatsby-plugin-mdx happy but it should be totally separate from it.

it uses eval to render on the client side with MDXRemote which can be dangerous from XSS perspective.

But you'll only use this at build time from a source that you can (hopefully) trust. So my first gut feeling would be that in context of Gatsby it's not more dangerous than all other existing patterns from remote sources.

So key question is whether there are significant roadblocks to foresee doing MDX with the plugin, using a transformer pattern?

So yeah, to give an answer here:

• We don't feel comfortable recommending the plugin as-is in the README because while technically it works, it's far from an ideal experience
• There should be a package in the transformer pattern (https://www.gatsbyjs.com/docs/how-to/plugins-and-themes/creating-a-transformer-plugin/) to handle the functionality
• We don't have the bandwidth to author such a transformer but happy to help in an consulting position to author it

One wouldn't be able to just take https://github.com/gatsbyjs/gatsby/tree/master/e2e-tests/mdx and use it for the transformer, but most tests can be adapted/slightly changed. This is the E2E test suite we use to test gatsby-plugin-mdx.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[hoijnet]

Seems it's still broken at the source from the comments there.

[hoijnet]

it uses eval to render on the client side with MDXRemote which can be dangerous from XSS perspective.

But you'll only use this at build time from a source that you can (hopefully) trust. So my first gut feeling would be that in context of Gatsby it's not more dangerous than all other existing patterns from remote sources.

I agree, this makes a lot of sense. I'll reiterate and see where I take it from here! Thanks a lot for the detailed comments. Get a bit overwhelmed with other things for a while.

[github-actions[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here. If we missed this issue or if you want to keep it open, please reply here. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[github-actions[bot]]

Hey again!

It’s been 60 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it. Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to comment on this issue or create a new one if you need anything else. As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community! 💪💜