MDX support

[nathanjhood]

Chore

Describe the chore

Time to bring in the content.

I want to use NextJs MDX support to create a '[projects]' slug. Then we can easily import the README.md directly from the repo's in question and just use the current live main branch's write-up as the page to render.

If that presents any issues, it's not really any harm to just copy/paste the README.md contents as needed. The only gripe here is not having synchronicity in the case that I update those project repos (a likely and intended scenario).

I don't really want to drag the additional responsibility of updating in two places, probably in differing syntaxes/formatting.

It also presents an opportunity to enrich the associated project repos by upgrading from standard .md to .mdx (maybe?).

A further consideration is to keep the sitemap generation dynamically synced to the page count, again without the responsibility of doing so manually. That is fine; the current sitemap generation scheme needs to be deprecated anyway in favour of NextJs MetadataRoutes (new ticket?).

Additional context

Add any other context or screenshots that help clarify the task.

[nathanjhood]

Re-opening because there is more to be done...

[nathanjhood]

Ok great, we have the Biquads project page in place as a successful first run.

Here are some snags that happened:

• Not currently sure why the styles are broken (page width?). Probably need to import 'globals.css' into the layout. Wondering if I can combine the MDX layout support with the projects page layout... how to nest the files to not break anything but apply the MDX layout downstream from the 'projects' page... hmm.

• The markdown that I copy/pasted from the biquads project repo refused to bundle due to formatting errors in the text. This is definitely broken at the source - my C++ was far beyond my markdown at the of creating that particular project!

• I intend to remote-fetch all 'projects' markdown content so that I can have the pages be automatically in sync with the project(s) state(s). This currently won't work of course as there are formatting issues at the source. I also need to investigate the security of this.

• Somehow my nextjs config was missing...? Fixed.

• EDIT: Will need to change the project links to point to the new pages, as currently they are inaccessible in web app mode. EDIT 2: maybe the nav menu 'projects' option needs a drop-down? Would need to be mobile-friendly...

[nathanjhood]

Hmm... these mdx pages should probably be dynamically rendered using a [slug]...

[nathanjhood]

Aghhh... I did the write up for CModule in ReStructured Text, not markdown.... That will need a refactor.

[nathanjhood]

What kind of change does this PR introduce?

Added the following sub-pages to the 'projects' route using MDX:

• biquads
• orfanidisbiquads
• ubento
• cxxwin
• nonlinearfilters
• msys2-toolchain
• bilineareq
• audioplugin-svf

Mostly just placeholders as there is an architectural refactor beginning to arise - these should probably be dynamic blog entries using [slug], and furthermore the CSS modules aren't really being used for anything right now (might do the image link handling here...).

[nathanjhood]

Two more issues here:

• Currently the layout doesn't appear to be functioning well when running as a web app (lots of cropping...)
• the project links open in new windows, in the browser (including when running as an app)

[nathanjhood]

Ok, so the CSS;

The .container and .content divs are sizing up the textual content on all the native NextJs pages as they should.

However, when it comes to the MDX pages, these divs are failing to contain the MDX component(s)...

One of those "grrr why" type of screenshots to capture ('print screen' only captures the active window, 'snippet tool' deactivates the on-hover feature in dev tools that highlights the affected area... grrr)

[nathanjhood]

Latest:

I refactored the way this all works, such that the pages are now written as Next-style pages in typescript, which import a component. Inside the component is the markdown handling. This resolves the issue with CSS that was previously not pulling through to the MDX layers.

I am still a bit nervous about remote-fetching the content, even from my own repo's which all have TLS and branch protections. Although I am sure it will be fine to do so, the current state of the markdown in each target repo is wildly inconsistent; many of the pages refuse to render due to not passing the MDX bundler requirements because of formatting errors.

Remote-fetching broken markdown and re-writing it at source while de-bugging it within the page is just not the way to do this.

I am currently just passing the markdown in as a big long string (groan!!!) defined as a const inside of the function, meaning the content is totally locked in to the build at the moment and can't be interfered with.

What I want to figure out is how to import Markdown from './biquads.mdx' locally, and pass that as <MDXRemote content={Markdown} />. I can then re-format all the content using hot-reload to make easy work of it. Once passing, the formatted content can be migrated back into the source repos and then fetched.

[nathanjhood]

Latest:

I refactored the way this all works, such that the pages are now written as Next-style pages in typescript, which import a component. Inside the component is the markdown handling. This resolves the issue with CSS that was previously not pulling through to the MDX layers.

I am still a bit nervous about remote-fetching the content, even from my own repo's which all have TLS and branch protections. Although I am sure it will be fine to do so, the current state of the markdown in each target repo is wildly inconsistent; many of the pages refuse to render due to not passing the MDX bundler requirements because of formatting errors.

Remote-fetching broken markdown and re-writing it at source while de-bugging it within the page is just not the way to do this.

I am currently just passing the markdown in as a big long string (groan!!!) defined as a const inside of the function, meaning the content is totally locked in to the build at the moment and can't be interfered with.

More on all of this.

So, I found at least one more reason why my markdown was/is not getting correctly styled (max width seems fixed at some size larger than the screen)... it's a dead simple one.

Some of my markdown content contains lots of whitespace:

{

    X = input sample;                                                             // audio input stream...

    _b0 = 1;                                                                       // initialize coefficients to safe numbers...
    _b1 = 0;
    _b2 = 0;
    _a0 = 1;
    _a1 = 0;
    _a2 = 0;

    a0 = (1 / a0_);                                                                 // calculate new coefficients from parameters...

    a1 = (-1 * (a1_ * a0));
    a2 = (-1 * (a2_ * a0));

    b0 = (b0_ * a0);
    b1 = (b1_ * a0);
    b2 = (b2_ * a0);

    Y = ((X * b0) + (X * b1) + (X * b2) + (X * a1) + (X * a2));                     // apply coefficients to input sample...

    output sample = Y;                                                              // audio output stream...

}

I tested the following markdown fix on the dev server and it resolved the issue:

{

    // audio input stream...
    X = input sample;

    // initialize coefficients to safe numbers...
    _b0 = 1;
    _b1 = 0;
    _b2 = 0;
    _a0 = 1;
    _a1 = 0;
    _a2 = 0;

    // calculate new coefficients from parameters...
    a0 = (1 / a0_);

    a1 = (-1 * (a1_ * a0));
    a2 = (-1 * (a2_ * a0));

    b0 = (b0_ * a0);
    b1 = (b1_ * a0);
    b2 = (b2_ * a0);

    // apply coefficients to input sample...
    Y = ((X * b0) + (X * b1) + (X * b2) + (X * a1) + (X * a2));

    // audio output stream...
    output sample = Y;

}

I'll probably open a new feature per-project-page and work through them each one at a time. This allows some chance to reflect on the content and probably enhance it.

What I want to figure out is how to import Markdown from './biquads.mdx' locally, and pass that as <MDXRemote content={Markdown} />. I can then re-format all the content using hot-reload to make easy work of it. Once passing, the formatted content can be migrated back into the source repos and then fetched.

This actually worked: import * as markdown from './biquads.mdx' passed to`. It actually exposed the markdown type definition in use and some helpful stuff from intellisense:

(alias) module "*.mdx" import markdown An MDX file which exports a JSX component.

The default export of MDX files is a function which takes props and returns a JSX element. MDX files can export other identifiers from within the MDX file as well, either authored manually or automatically through plugins

It’s currently not possible for the other exports to be typed automatically. You can type them yourself with a TypeScript script which augments *.mdx modules. A script file is a file which doesn’t use top-level ESM syntax, but ESM syntax is allowed inside the declared module.

This is typically useful for exports created by plugins. For example:

 // mdx-custom.d.ts
declare module '*.mdx' {
import { Frontmatter } from 'my-frontmatter-types';

export const frontmatter: Frontmatter;
export const title: string;
}

The previous example added types to all .mdx files. To define types for a specific MDX file, create a file with the same name but postfixed with .d.ts next to the MDX file.

For example, given the following MDX file my-component.mdx:

export const message = 'world';

# Hello {message}

Create the following file named my-component.mdx.d.ts in the same directory:

export { default } from '*.mdx';

export const message: string;

Note that this overwrites the declare module '*.mdx' { … } types from earlier, which is why you also need to define the default export. You can also define your own default export type to narrow the accepted prop types of this specific file.

It should now be possible to import both the MDX component and the exported constant message.

[nathanjhood]

MDX fetching is all working nicely, it's being called from my own TLS-certified repos so it should ought to be quite secure.

One pretty major issue though...

If and when I do update the source project repos, there is scope to inadvertently and unknowingly break the webpage by doing things like breaking the formatting in the markdown file which is being fetched.

This would currently yield undefined behaviour. I also might not notice it if my attention is focused on the source project and I'm not checking the webpage...

I need to wrap the fetch function inside a try-catch and have it return a basic 404 message in the event of any issue.

I also think that in doing so, it would be perhaps wiser to move all the mdx-fetching routine into a global utility function that we import in the page files where/when needed (i.e., whenever we need to use MDX on the page content).

The try-catch and 404 message can all be handled in this utility function. The 404 itself can be as simple as returning a markdown-flavoured string:

if (error) {
    return `
# 404

The requested content could not be found, or is unavailable at this time.
`
}

It might be a good idea to move this to a 'pre-build' step in the npm package.json.

Thinking in more advanced terms, we could create a cache folder and store the latest successful mdx-fetch results in there, and fallback to these cached instances in case of error/failure...

[nathanjhood]

Thinking of the fetch routine as a build step, perhaps even a pre-build step, I think it might be wise to...

• First attempt to fetch the content from the secure remote repo
• Try-catch the result, returning a 404 (or whatever) if failed
• If fetch was successful, pass the result through the whole NextJs MDX parser
• Try-catch the result....
• If the fetched MDX is successfully parsed, write it to the cache
• If the fetched MDX fails, reject it and fall back to the last good cache entry
• Use the cached content to populate the page

Drawbacks:

• prevents truly dynamic fetching
• preventing truly dynamic fetching is kind of the point...

Several angles to consider.

At the very least, though, we should be try-catching our fetch result and returning a reasonable and well-defined error handler if failed.

[nathanjhood]

(sigh) ...we could create a Github API webhook to notify our webpage whenever there is an update to one of the project pages... triggering a fetch-try-catch-render scenario.

Honestly it probably is a step in the right direction, despite seeming to be overkill. We're dynamically fetching remote content to be executed on the server here... it really needs to be bulletproof.

[nathanjhood]

In either case, I believe this whole topic and the projects pages functionality will be greatly enhanced by usage of dynamic routing as a means to making a formalized/generic 'blog' style feel for the fetched results (no matter how or what we fetch).

Opened a chore on #433

[nathanjhood]

Beginning of error handling in place using a simple try/catch, in this example I've used an intentional typo in the fetched link to cause the failure:

[nathanjhood]

I have intentions of wrapping this into a component (on #433) but meanwhile it's kind of critical to have something in place, so I'm going to roll this out across all MDX pages in it's current form with the (technical debt, cough cough) intention to refactor into a 'handler' component soon thereafter.

[nathanjhood]

Ok, try/catching is in place for the 'fetch()' part at least.

What this doesn't cover is a case where the content is fetched successfully, but fails the parsing step (where the MDX is converted into HTML by the NextJS plugin).

Rather than address this on another per-page implementation, with lots of silly code repetition, instead I'll work that into our new "markdown fetch, validate, and style up" component, in due course.

[nathanjhood]

Argh, it happened... the MDX failed to parse and caused an error:

[Error: [next-mdx-remote] error compiling MDX:
Expected a closing tag for `<project_name>` (101:18-101:32) before the end of `paragraph`

   99 | * the "./src" directory, where 'source' files are kept, and,
  100 |
> 101 | * the "./include/<project_name>" directory, where 'header' files are kept
      |   ^
  102 |
  103 | Both specified relative to the project's root folder.

More information: https://mdxjs.com/docs/troubleshooting-mdx]

[nathanjhood]

Well, it's kind of fixed...

I think it must be those code blocks that cause this CSS breakage...

[nathanjhood]

Great.

• Dynamic fetch via a utility component running on the server
• Error handling for fetch failure
• Error handling for MDX parse failure
• Refactored to place the project pages into an article element that matches the other pages
• New component for GH repo cards
• 'Like' and 'Back' buttons running on client side with suspense boundary

Productive. Time to move this into all the projects pages.

[nathanjhood]

Almost done but there are some blockers, namely certain pages with issues:

• cmodules styles is ignoring the width setting. Probably because of the codeblocks. Maybe I'll move some lengthy comments out of the codeblocks and into regular markdown paragraphs...
• MSYS2-Toolchain haven't really attempted to fetch it yet due to unknown ability to pass MDX parsing
• Ditto for UBento, but that one is going to be a complete nightmare no matter what...

I suppose it might be useful to test our mdx tests on the last two. The first one should be painless enough.

EDIT: otherwise, it's really looking good now! :D

[nathanjhood]

Ok so Nextjs extends fetch to incorporate data caching natively which is really quite cool.

In the event of a successful fetch, the data is written to cache at Vercel Project level. If fetching fails, the behaviour is supposed to be a fall-back to the last good cache entry, according to the docs (and, wonderfully, this is precisely what I was wishing for out loud just a few posts ago).

There is also an in-built route for fetch response failures to call the in-built error handler - which I haven't really got any knowledge of yet; I've just wired it up per the docs for now.

In the current implementation, the data fetching is happening dynamically and re-validating every 10 mins (I think). This is proven by making small edits to the source project README.md's and observing the changes on the webpage, without deploying a new build.

I've added a server action to regenerate the items marked with the 'collection' tag, and added this tag to our mdxFetch function. I also added some server-side console-logging, for the time being, so that this behaviour can be observed and verified "in the wild".

I was a little mistaken about how the MDX custom components are supposed to work.

MDX remote's docs suggest that it is literally a case of creating a custom React component that returns the MDXRemote component, but with your custom settings hardwired into it's guts. This component is then to be imported and used instead of MDXRemote.

Unfortunately MDXRemote (or Next MDX, I don't know which) does not generate any kind of tab index for the outputted HTML. Not a problem, I've just added a tab index of '0' to the heading elements as custom MDX components and it works great.

I've also disregarded that MDX Remote has a serialize function, which typically is passed to the MDX Remote component's source input after doing it's work on the fetched content. I will look into this some more as time permits.

It will probably be wise to component-ize the MDX Remote customization to include:

• Fetch using try/catch and cache re-validation
• Serialization (I believe it's a necessary step for the above)
• Tab-indexed HTML heading components
• Desireable CSS classes where needed (such as adding doxygen's 'fragment' class to all 'pre' elements)

I've been trying to use MDXRemote in things like a new try/catch statement to test that the fetched content parses successfully (and return a sensible error in case of failure, instead of just completely breaking the server connection with undefined behaviour), so far with no luck. I'm a little surprised actually that I can't seem to find this functionality natively within the libs themselves, considering things like the data caching going on already. I will keep digging in the docs because this is a slight showstopper in some potential scenarios currently.

[nathanjhood]

Funny thing: The MDX Remote docs insist that for the majority of use cases, where a simple portfolio page is being created (such as this), the overall most common issue is over-complexity and that users should stick to HTML and CSS, not least because of breaking dependencies in packages over time.

Ironically, all of the above is precisely my original mission objective; I had set out to build a solid HTML5/CSS5/Vanilla Js page with every fine detail under precise control.

Coming from C++ world, I was especially keen to avoid time-bombing my page with dependencies where a little education and testing should do the trick...

Unfortunately, that method led to a lot more vulnerability handling than I am equipped to deal with. I migrated to this framework because I needed to defer some critical work to bodies more able and knowledgeable than myself.

So, bring on the fetch caching and validation...!

[nathanjhood]

Cool, the fetching is showing up in the server logs now. I may have to roll this back due to GDPR, not sure, but there is no user/IP/session information otherwise co-located with the fetch logging.

I'm not seeing evidence of data caching - rather, I'm seeing evidence that this isn't happening. Probably will investigate that whole scenario in a new issue/discussion.

Now it's only UBento that needs formatting so I can fetch it properly.

Probably time to call this thread a job well done - moving to discussions page for future discoveries.