Epic: Migrate from Gatsby to NextJS

Gatsby is working fine for now, but as a long-term goal, we should move to NextJS since it has become more mature. We shouldn't migrate until we have finished implementing most of our desired features.

After spending considerable time over the past week on this, I want to offer my (very long) thoughts:

Content Organization

Having any form of a content layer to abstract the filesystem is critical given how we're storing our content on the filesystem instead of a database and API.

For Gatsby, this is done through GraphQL and its gatsby-source-filesystem plugin, as well as other plugins to transform files into useful data that can be queried. Since the filesystem plugin considers all files, any file can be transformed, including MDX files, images, PDFs, etc. The filesystem plugin also makes all files accessible on the live site. These plugins are officially maintained by the Gatsby development team.

For NextJS, there is no official support for a filesystem content layer. @next/mdx only allows creating pages and routes with MDX files stored in ./src/app or ./src/pages (it doesn't even support frontmatter). However, there is a fantastic third-party library called Contentlayer that's in beta. It allows MDX files to be transformed into content types by building them into accessible JSON objects, similar to what GraphQL does for Gatsby (but without custom queries). This solves the majority of my concerns with processing MDX files, but not for other file types, such as images, slides/PDFs, or miscellaneous files (like additional assets provided as part of a meeting, ZIP archives, and whatnot). Contentlayer has been looking into image support.

For example, we keep our slides in ./content/meetings/XXXX-XX-XX/slides.pdf, but that won't actually be accessible on the live site because any file that needs to be made statically available has to either be:

Bundled with the rest of the site in the output distribution directory (this is what gatsby-source-filesystem does)
Stored in ./public/ to eventually be included in the output distribution directory

For option 1, I couldn't really think of much other than trying to set up webpack or esbuild, but I'll be honest, that goes over my head.

For option 2, I first tried copying the files from ./content/ into ./public/. This commit demonstrates that working for images, thanks to these two articles: 1 and 2. But uh it's very jank and does not handle edge cases which should be considered when working with file manipulation 😬.

An alternative option that I just came up with (and would probably be what we would do if we continue pursuing NextJS) would be to just rename ./content/ to ./public/. This would leave all of the index.mdx and any other files that we only need to build the site hanging around, but it's really the only idea I have that could work. I still consider this as a disadvantage to gatsby-source-filesystem since that plugin automatically uses hashed paths so caching won't be a problem with a CDN.

Deployment

Since Gatsby is a static site generator, it doesn't matter where we deploy since any web server application can do the job. Netlify, GitHub Pages, a Raspberry Pi running Apache, sure. One of my highest motivations for a static site is so that we can deploy on Cloudflare Pages - which offers unlimited requests/bandwidth, a CDN, and very good analytics so we don't have to use third-party trackers (all for free!)

With NextJS, we can either run the site on Node.js/Edge runtime OR create a static export of the site. If we go with Node.js/Edge, we'll have limits in terms of server-side execution and bandwidth and probably will have to purchase. If we go with a static export, there will be understandably less features compared to server-side rendering. However, the feature set we have is far less than what Gatsby can provide because NextJS wasn't made for static site generation!

For example, if we want image optimization and processing, we'll have to implement all that functionality ourselves, whereas with Gatsby, I don't even need to think about it. Or something as simple as generating a _redirects file for Cloudflare Pages/Netlify to use for server-side redirects. Gatsby supports generating redirects dynamically on build (e.g. a /slides redirect for each meeting)! NextJS doesn't - you have to define all the redirects in their config file. Again, NextJS can do all of that when it's using a runtime, but not with a statically exported site.

Future Considerations

Right now, we don't need server-side rendering or logic since all we are doing is displaying static content. I'll quickly go over some features that we are looking to implement to determine if we would ever need anything like that:

CTFd integration - We don't actually need any server-side functionality for the website since all of CTFd's features are accessible through its API. In that case, we can simply implement requests to CTFd on the client-side web bundle. This also eliminates security risks related to our website managing user sessions to CTFd.
Content Management System - We've already proven that we can manage our content on a "serverless" model, but if we really wanted to, we can create a CMS that is part of the website to update it. But then, we wouldn't be using Git-based/filesystem content.
Discord archive site - Originally, our plans for this were to run it as a separate server, but we could add it to the main site. I still think that an approach to how we have CTFd available as an API is better.

Conclusion

I really wanted to migrate to NextJS, but there have been too many issues that I'm not sure how to resolve (which have already been solved in Gatsby). I also do not see major benefits that would result in us migrating to NextJS other than possibly more stability (although I haven't had any issues with Gatsby) and a better attitude towards the website from the helper team lmao. But I am curious to hear more thoughts to determine where we go from here.

For NextJS, there is no official support for a filesystem content layer. @next/mdx only allows creating pages and routes with MDX files stored in ./src/app or ./src/pages (it doesn't even support frontmatter).

This is simply not true. I searched "nextjs static blog", and the first result is an article by the next.js team using gray-matter (a frontmatter parser) with files stored in the top level /posts directory.

I also do not see major benefits that would result in us migrating to NextJS other than possibly more stability (although I haven't had any issues with Gatsby) and a better attitude towards the website from the helper team lmao.

The main benefit is easy-to-find documentation and tooling. I don't know if I'm just a complete idiot, but for example adding the images in Gatsby took me 2-3 hours, and doing something similar in next.js would have taking 15 minutes. While trying to add it in Gatsby, I ran into outdated repos, forks of official (unmaintained) plugins so that items could work, and a lack of good documentation if you didn't have the exact usecase that the Gatsby homepage describes.

I have no issues with using Gatsby, because at the end of the day I don't mind using graphql and markdown. I just worry that Gatsby is dying. See this state of javascript post under "retention". People do not want to continue using Gatsby. This lack of an ecosystem in the future leads to an overall slower development process. I think if we write it in Gatsby, in 2-3 years, it will already need a rewrite. That might be OK.

This is simply not true. I searched "nextjs static blog", and the first result is an article by the next.js team using gray-matter (a frontmatter parser) with files stored in the top level /posts directory.

I was referencing this article in my first few attempts at implementing content type parsing from the filesystem (you can see gray-matter added in package.json). This implementation only works for files stored directly in the "posts" folder and it doesn't work recursively. It does not handle "index.md" files. It does not handle the actual MD content. It does not generate types or ensure type checking. In our specific use case, we sometimes specify a slides or image field in our front matter with a relative path to the slide or image file - we would have to extend the code to somehow create a relation between the markdown content and this relative file, plus implement error handling for whether linked files exist or not. Everytime we call this function, it will read every file again - we could memoize it, but that would probably break next dev from properly updating.

This is what I meant by Next.js not officially supporting a filesystem content layer - we have to write the layer ourselves and that feels unnecessary when there are already plugins that do exactly this in Gatsby (and Contentlayer did end up writing all of that themselves, so there really is no point dwelling on this further).

for example adding the images in Gatsby took me 2-3 hours, and doing something similar in next.js would have taking 15 minutes

I assume you are referring to storing images in the public folder, then using the image's absolute path on the built site within the markdown to render it? Realize that would force the end user to store their content in the content folder, and then they would have to store their images in the public folder. Then the public folder becomes a mess of assets, and eventually no one knows what assets corresponds to what content. Unless we decide to replicate the directory structure of content within public, but now the end user is managing two separate folders for something that could have just beeb one folder. I haven't been able to find a resource for anyone who has fixed this properly in Next.js, mainly just tutorials already using the public folder and those accepting defeat. (Except for the person above with the two articles explaining how they did it in a hacky way.)

And realize that the only thing you did to add image support to MDX for Gatsby was an additional 20 lines to the config file, most of which was JSON formatting. And if you wanted to do it the Next.js way, you could have done that too - Gatsby's version of the public folder is static.

All that being said, I was not aware of Gatsby's declining popularity. I do agree it would not be in our best interest for the framework we use to go bottom-up in the next few years. Which is a complete shame because after working with Next.js I completely understand why Gatsby does things the way it does. Still, my concerns for things like image optimization and site deployment remain unaddressed.

Then the public folder becomes a mess of assets, and eventually no one knows what assets corresponds to what content. Unless we decide to replicate the directory structure of content within public, but now the end user is managing two separate folders for something that could have just beeb one folder.

Yeah, this is probably true. I should refactor the sponsor images to be relative to the sponsor file. This makes a lot of sense. Potentially a lot of the trouble I had was not understanding this design.

I think it is unrealistic to migrate to Next.js unless we hit any major issues with Gatsby, and it looks like those have all been resolved. I do think that you could find a solution to all the concerns you have about image optimization and site deployment, but is it worth it? Then the migration process would be more about : "what is the benefit of next.js over Gatsby", and my main response would be "better long-term ecosystem support, and baseline ability from helpers to contribute to the site". But if we add most of the features we want this summer, the long-term ecosystem support isn't that critical, and so the main point for migration would be Is it significantly easier to do things the next.js way over the Gatsby way?

And in my previous projects, it seemed that answer is yes, but you reached a different conclusion which is equally valid -- next.js does not make things significantly easier for us. So TLDR; based on your research into our site needs and how next.js can meet them: in it's current state, Gatsby is more suitable for our static content-driven model than Next.js (is what I'm hearing)

And I think that soothes my head, so I will stop complaining about Gatsby :)

feel free to close if you agree, but also happy to discuss further.

Closed in favor of #190

sigpwny / websites