How to render remote markdown using content layer API

[johnmw]

📚 Subject area/topic

markdown, content layer api

📋 Suggested page

https://docs.astro.build/en/reference/configuration-reference/#experimentalcontentlayer

📋 General description or bullet points (if proposing new content)

I think that with the new content layer API, rendering remote markdown (from a server or api) is going to be a common use case.

The Astro docs currently says "Astro does not include built-in support for remote Markdown outside of experimental content collections!"

However I was unable to find any further information or examples about rendering remote (server) markdown.

So firstly, I was wondering if there already is an good example that I have missed?

If not, I have got some rough code (shown below) that renders markdown using Astros built in functions that seems to work and might help someone. But I would really like someone to review it and let me know things that can be improved.

Example Loader that renders remote markdown:

import { defineCollection, z } from 'astro:content';
import type { Loader } from "astro/loaders";

function markdownTestLoader({}):Loader {
  return {
    name: "markdown-test-loader",
    load: async ({ config, store, parseData, generateDigest, entryTypes }) => {

      store.clear();

      // Would normally load this JSON data from API. 
      // The "content" field has the markdown I want to render
      const dummyAPIData = {
        id: "home",
        title: "Test Markdown Data",
        content: "## Markdown to Render \nTesting markdown rendering.\n- List item 1\n- List item 2"
      }

      const id = dummyAPIData.id;
      const data = await parseData({ id, data:dummyAPIData });
      const body = dummyAPIData.content; // markdown to render next
      const digest = generateDigest(data);

      // Render markdown to HTML using built in Astro markdown render function
      let rendered;
      const markdownEntryType = entryTypes.get(".md");
      if( markdownEntryType?.getRenderFunction ) {
        const render = await markdownEntryType.getRenderFunction(config);
        rendered = await render?.({
          id,
          data,
          body,
          digest
        });
      }

      store.set({ id, data, rendered, digest });
    }
  }
}

// Define collection with loader

const markdownTestCollection = defineCollection({
  loader: markdownTestLoader({}),
  schema: z.object({
    id: z.string(),
    title: z.string(),
    content: z.string()
  })
});

export const collections = {
  'markdownTestCollection': markdownTestCollection  
};

The markdown HTML can be displayed on the page using the "render" function and built in "Content" component:

---
import { getCollection, render } from 'astro:content';

export async function getStaticPaths() {
  const pages = await getCollection("markdownTestCollection");
  return pages.map((page) => ({    
    params: { 
      slug: page.id },
    props: { 
      page 
    },
  }));
}

const { page } = Astro.props;
const { Content, headings } = await render(page);
---
<h1>{page.data.title}</h1>
<Content />

As I said, I'm sure there are things above that could be done better.

🖥️ Reproduction of code samples in StackBlitz

No response

[Fryuni]

We don't add guides and recipes for experimental APIs, only the refence and examples in the description there. So it won't be added on the docs for v4.

The feature is stabilized on v5, so we can document it there. The docs for the LoaderContext is missing the entryTypes property which enables rendering the different content types. We can add documentation for that and how to use it on the guide for writing a loader.

[sarah11918]

Thanks for this @johnmw ! Just showing off working code is already very helpful to others, and I appreciate your care in wanting to further make sure that the code is something recommended for others to copy.

Fryuni is right that any docs for this would have to go on our v5 beta docs. I will also ask @ascorbic to take a look to ensure that we end up with something that we'd recommend as a "happy path" as a general model!

If so, then I also agree this is for the Loader API page, in one of two ways:

• an example of a loader that fetches remote Markdown (the first code snippet only. I think the other code examples are pretty standard usage once you have the loader configured properly? There's nothing there that seems unique to having built that particular loader?)

or

• a link from this page to a full recipe set of instructions that includes everything above, but uses a real data fetch from a real API. This should be something that someone can follow along and have something working in their project at the end.

[ascorbic]

I agree! Adding it to the Loader API page would work best.

[trespaul]

I just want to add that there is a single sentence in the docs that — other than the "outside of experimental content collections" bit — made me go on an hours-long wild goose chase, that being

Once queried, you can render Markdown and MDX entries to HTML using the render() function property. Calling this function gives you access to rendered HTML content, including both a <Content /> component and a list of all rendered headings.

at https://5-0-0-beta.docs.astro.build/en/guides/content-collections/#rendering-body-content

The docs need to specify that the user needs to build that functionality themselves. I'd make a PR but I don't think I know enough about Astro to cover all the bases.

Also, this would be a nice built-in function to have anyway, since, like @johnmw said, rendering markdown is probably one of the common use cases.

Update: I think v5 doesn't have entryTypes, so I cobbled together the following (with reference to this):

import { defineCollection } from 'astro:content';
import type { Loader, LoaderContext } from 'astro/loaders';
import { createMarkdownProcessor } from '@astrojs/markdown-remark';
import { getData } from 'wherever';

type Collection = 'pages' | 'updates';

function myLoader(options: { collection: Collection }): Loader {
  return {
    name: 'myLoader',
    load: async (context: LoaderContext) => {
      const data = await getData();
      const processor = await createMarkdownProcessor(context.config.markdown);
      data.forEach( async item => {
        const id = item.slug; // I'm using slug as id
        const data = await context.parseData({ id, data: item });
        const digest = context.generateDigest(data);
        const rendered = await processor.render(item.content ?? '');
        context.store.set({
          id,
          data,
          digest,
          rendered: {
            html: rendered.code,
          },
        });
      });
    },
  }
};

function makeCollection(name: Collection) {
  return defineCollection({
    loader: myLoader({ collection: name }),
    // schema: z.object...
  });
}

export const collections = {
  pages: makeCollection('pages'),
  updates: makeCollection('updates'),
};