v2 Rewrite

[Js-Brecht]

This plugin's initial design has resulted in some compatibility issues that simply require too many workarounds. This is primarily due to the "proxy" method that was used to source the desired gatsby-* files.

This issue is to track the plan for v2, but will be implemented incrementally. OP will be updated with progress as it becomes available

Issues

• If gatsby-config.ts is in the root directory of the default site, it will cause it to be run twice; once by this plugin, and once by Gatsby. This results in node ownership issues, and of course the multiple execution. See #2
• The proxy method employed also results in additional node ownership issues because gatsby-node isn't being run as the default site. See #18
• A singleton pattern is used in various places to support the proxy method. This, as well as the proxy method itself, breaks Gatsby theme compatibility. See #20.
• To remain compatible with yarn2/pnpm, the plugins from gatsby-config have to be resolved to absolute paths. This breaks Gatsby's deduplication, as well as their internal plugin overrides. See #26, #27
• Various (potential) issues with Gatsby v3 due to the obfuscation of imports, and other possible problems. See #29
• The highly dynamic nature of the sourcing breaks static analysis, which means non-critical compatibility issues with the Gatsby dev process. Gatsby dev doesn't do static analysis/import tracing yet (I am looking into implementing in core), but the current plugin design would not work with it.

Solution

To fix these requires changing the majority of the design, so it is simpler to do a complete rewrite. The proxy method will be replaced by isolated endpoints:

pseudo-code: pattern subject to change

// /gatsby-config.js
const { useGatsbyConfig } = require("gatsby-plugin-ts-config");

module.exports = useGatsbyConfig("./.gatsby/config", {
  ... custom transpiler options ...
  ... usual plugin options ...
});

// For the purposes of static analysis
module.exports = useGatsbyConfig(() => require("./.gatsby/config"), {
  ... custom transpiler options ...
  ... usual plugin options ...
});

pseudo-code: pattern subject to change

// /gatsby-node.js
const { useGatsbyNode } = require("gatsby-plugin-ts-config");

module.exports = useGatsbyNode("./.gatsby/node", {
  ... custom transpiler options ...
  ... usual plugin options ...
});

// For the purposes of static analysis
module.exports = useGatsbyNode(() => require("./.gatsby/node"), {
  ... custom transpiler options ...
  ... usual plugin options ...
});

These "isolated endpoints" will be able to share options between them for each project/theme that uses the plugin. Only one will need to be configured, so you could have /gatsby-config.js, /.gatsby/config.ts, and /gatsby-node.ts; or you could simply use gatsby-node.js, and point it to a Typesecript version... no gatsby-config required.

Because gatsby-browser and gatsby-ssr already support Typescript (they are run through Webpack by Gatsby), and proxying them to a different location breaks themes (and possibly incremental builds), they will need to be set up manually by the end user.

Features

Some new features will be available in v2:

Builder Pattern

The idea here is to have a working POC for a new plugin design pattern that can be used with Gatsby. This would be opt-in

It would look something like this:

// plugin-types.ts
export type MyPluginOptions = {
  foo: string
}

// gatsby-node.ts
import type { GatsbyNodeBuilder } from "gatsby-plugin-ts-config";
import type { MyPluginOptions } from "./plugin-types";

type MyPluginBuilder = GatsbyNodeBuilder<MyPluginOptions>;

// sourceOptions are options passed around between endpoints by gatsby-plugin-ts-config
// includes import chain and property bag
const pluginBuilder: MyPluginBuilder = (builder, sourceOptions) => {
  builder.sourceNodes<MyNodeType>((actions, options) => {
    // I'm strongly typed
    actions.createNode({ ... })

    // I'm strongly typed
    console.log(options.foo);
  });
}

export default pluginBuilder;

// To modularize the logic, accept an array of builder functions
import { sourceNodes } from "./source-nodes";
import { createPages } from "./create-pages";
const builders = [
  sourceNodes,
  createPages,
];

export default builders;

// gatsby-config.ts
import type { GatsbyConfigBuilder, PluginDefs } from "gatsby-plugin-ts-config"; 
import type { SomePluginType } from "some-other-plugin";
import type { MyPluginOptions } from "./plugin-types";

type MyGatsbyConfig = GatsbyConfigBuilder<MyPluginOptions>;

const configBuilder: MyGatsbyConfig = (builder, sourceOptions) => {
  // Allow merging any properties into the config, loosely typed
  builder.merge({
    ...
  });

  // Add siteMetadata.  Optional generic type parameter
  builder.siteMetadata<RequiredSiteMetadata>({
    ...
  });

  // Add loosely typed plugin definitions
  // Also looks at/transpiles local plugins
  builder.addPlugins([
    ...
  ])

  // Add strongly typed plugin definitions
  builder.addPlugins<PluginDefs<SomePluginType | SomeOtherPluginType>>([
    ...
  ]);
}

export default configBuilder;

// Also accept an array (for modularization)

Definitions

• JIT: Just In Time
• Activated: A gatsby-* js file that uses the endpoint transpiler function provided by this plugin
• Active Pattern: An activated module that exports a function to be consumed/processed in order to generate the content for Gatsby's consumption
• Builder Pattern: See Builder Pattern above

Specifications

• [x] Babel or TS Node for JIT transpiling - #32
  • [x] Custom transpiler options merge - #32
• [x] All gatsby-* represent themselves (no proxy) - #32
• [x] Only one of gatsby-config or gatsby-node required to be activated - #32
  • Don't need gatsby-config activated in order to use Typescript gatsby-node
  • Don't need gatsby-node to be activated if gatsby-config is (can use /gatsby-node.ts)
  • Caveat: cannot use only gatsby-node and expect /gatsby-config.ts to be transpiled, since gatsby-node is run later than gatsby-config
• [x] Shared options between gatsby-config & gatsby-node - #32
  • One of:
  • [x] Merge options defined in gatsby-config with the ones defined in gatsby-node when it runs?
  • [ ] ~Replace options defined in gatsby-config with the ones defined in gatsby-node when it runs?~
• [x] Options/transpilers isolated per runtime (themes support) - #32
  • [x] Tied to call sites - #32
• [x] Callback import method (for static analysis support) - #32
• [x] Local plugin transpiling - #32
  • Fully processed property bag passed down from parent & cloned
• [x] Active pattern for activated modules - #32
  • Requires activated module
  • Active pattern accepts metadata passed between endpoints by this plugin
  • [x] Import chain - #32
  • [x] Property bag (mutable props defined by end user; isolated by project/theme) - #32
• [ ] Builder pattern for activated modules

Priorities

1. Isolated endpoints
   • Should support the same "Active module" pattern that v1 supports (default export function)
2. Plugin builder pattern

Estimated Timeframe (conservative)

• Isolated Endpoints: 05-23-2021 - #32
• Builder Pattern: 06-06-2021

[alvis]

@Js-Brecht just wanna leave a comment for my appreciation on your effort for the community. Thanks for creating this amazing plugin. 👍🏻♥️

[openscript]

@Js-Brecht If I can help you somehow, let me know. I'm also happy to do documentation or a starter library for Gatsby.

[Js-Brecht]

@alvis @openscript hey guys, thanks for the support. I really appreciate it.

@openscript Documentation is definitely going to be quite the chore, and writing docs is not really something I enjoy doing; it can also be challenging for me to make them clear/concise. If you were interested in picking that up, it would be pretty awesome. I think most of the v1 docs will need to be ripped out/replaced. There are some details in the behavior that would need to be covered in the docs that may not be immediately apparent from the code, so let me know and I can try to explain.

I have #32 up with the initial v2 rewrite. I haven't actually done any testing with it yet, so I do expect there to be bugs. Another thing I would appreciate is some pre-release test cases. If you're interested, then once I have the initial debugging complete I can ping you.

:pray:

[KyleAMathews]

Don't want to take away from your great work here but curious if you've looked into what it'd take to support directly adding Typescript support to the framework? It might be less work & would be a lot simpler to use & easier to maintain long-term!

[alvis]

I have been using Gatsby even prior 1.0 and using typescript with it is always been hacky, tricky and a lot of 😭😭😭

@Js-Brecht If we have @KyleAMathews's blessing, let's do something!

[Js-Brecht]

@KyleAMathews I have considered it. The simplest implementation would not be difficult, but there would need to be some decisions made regarding behavior. I never pursued it because I tend to do work with different ts transformers (and/or my own babel plugins), and a core implementation would probably conflict with that: the "simplest" method likely wouldn't support extensions... which is why I've been more invested in doing it here.

Not that I don't think it is worth doing though. I can work on an RFC tonight to get some feedback on the desired method for an initial implementation, and to at least discuss the possibilities for extending a default core transpiler.

[Js-Brecht]

RFC is here: https://github.com/gatsbyjs/gatsby/discussions/31587

/cc @KyleAMathews

[alvis]

Hi @Js-Brecht, I know we have a RFC in process and a third-party plugin may be ultimately dropped in favour to the native approach, but do you think we are close to get gatsby-config.ts working on gatsby v3 in the mean time?

[Js-Brecht]

@alvis yes, I still intend on completing this, especially since I will continue using it myself instead of the core implementation to start. There’s just a lot of things competing for my attention right now, including another major upgrade at work, and that one wins.

V2 is pretty much ready to go. I need to do some testing with it and update the docs, but that’s about it.

[alvis]

@Js-Brecht Is there anything we can help out?

[Js-Brecht]

I'm sorry guys... I've been completely swamped at work, and I just got back from vacation so now I'm playing catch up.

@alvis #32 just needs testing, and it needs new docs. If you were interested in helping out with either of those, drop any questions you have on the PR; it seems a more appropriate place.

[Js-Brecht]

Hey everybody, sorry for the delay. I've had so little time to spend over here, but I'm finally happy with the state of v2 and I think it's time to release it into the wild. v2.0.0 has been published.

I'm going to close this issue out now and handle any problem with v2 as they come; I'll need to work on the extra stuff I wanted to do in another iteration.

Next steps: Typescript support in the core...

[cameronbraid]

Great work I’ll be giving it a spin this week

[Beraliv]

Good job, thank you for the update!

[KyleAMathews]

Thanks @Js-Brecht !

[rramphal]

Amazing, @Js-Brecht — I'm trying it now. Thank you 🙌🏾 !!!

[Js-Brecht]

Please see the release notes for breaking changes

[cameronbraid]

Just wanted to report that I upgraded to v2 and its working great. Thanks heaps.

[Js-Brecht]

That's great to hear :raised_hands: :tada:

---

Bit of a progress report:

• I've got the implementation about 95% complete in the core; just need to write a few tests. Possibly done and ready for review this weekend, maybe next :crossed_fingers:

• I just merged a new feature tonight that includes a base level, always-on, transpiler; meaning, you can have Typescript in any of the plugins that you define for Gatsby (at least at the default-site level). It's not released yet, but will be soon.

• I am going to release a CLI for this project soon, so that no .js files are required. Essentially the same as what's going to happen with core support, but with the extra features/flexibility (that I still want). This is probably happening within the same time frame.

  Related: This will involve moving to a new npm package name, primarily because this isn't really a "Gatsby plugin" anymore; check out the discussion for more details.

If anybody has features they want included in a new version, post them on the discussion listed above.