Issue with gatsby@4

[cameronbraid]

Hi

Am tying an upgrade to gatsby v4, however the build is failing.

A change in gatsby 4 (https://github.com/gatsbyjs/gatsby/pull/33413/files) verifies the rendering engine to only allow node.js imports (and those in the .cache folder). However when using this plugin it tries to require the project's /package.json which breaks the build.

The gatsby build error is :

failed Validating Rendering Engines - 1.763s

 ERROR #98001  WEBPACK

Built Rendering Engines failed validation failed validation.

Please open an issue with a reproduction at https://github.com/gatsbyjs/gatsby/issues/new for more help

  Error: Generated engines use disallowed import "/Users/cameronbraid/workspace-new/drivenow-octane/package.json". Only allowed imports are to Node.js builtin modules   or 
engines internals.

  - child.ts:60 Function._module.default._load
    [gatsby-virtual-27f6b46480]/[gatsby]/src/utils/validate-engines/child.ts:60:11

  - loader:1005 Module.require
    node:internal/modules/cjs/loader:1005:19

  - helpers:102 require
    node:internal/modules/cjs/helpers:102:18

  - index.js:714801 Object.<anonymous>
    /Users/cameronbraid/workspace-new/drivenow-octane/.cache/query-engine/index.js:714801:19

the import that causes the issue in .cache/query-engine/index.js:714801:19 : the exports.pkgJson assignment below

/***/ }),
/* 3934 */
/***/ (function(__unused_webpack_module, exports, __webpack_require__) {

"use strict";

var __importDefault = (this && this.__importDefault) || function (mod) {
    return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", ({ value: true }));
exports.apiTypeKeys = exports.pluginName = exports.pkgJson = exports.thisRoot = void 0;
var path_1 = __importDefault(__webpack_require__(12));
exports.thisRoot = path_1.default.resolve(__dirname, "..", "..");
exports.pkgJson = require(path_1.default.join(exports.thisRoot, "package.json"));
exports.pluginName = exports.pkgJson.name;
exports.apiTypeKeys = ["config", "node"];
//# sourceMappingURL=constants.js.map

looking for what requires 3934 I found this : (see line with var constants_1 = __webpack_require__(3934);)

/***/ }),
/* 3974 */
/***/ (function(__unused_webpack_module, exports, __webpack_require__) {

"use strict";

var __importDefault = (this && this.__importDefault) || function (mod) {
    return (mod && mod.__esModule) ? mod : { "default": mod };
};
Object.defineProperty(exports, "__esModule", ({ value: true }));
exports.transpilePlugins = exports.resolvePlugin = void 0;
var path_1 = __importDefault(__webpack_require__(12));
var fs_tools_1 = __webpack_require__(3938);
var node_1 = __webpack_require__(3937);
var constants_1 = __webpack_require__(3934);
var project_1 = __webpack_require__(3975);
var resolvePlugin = function (relativeTo, pluginName, localOnly) {
    var scopedRequire = (0, node_1.createRequire)(relativeTo + "/:internal:");
    try {
        var pluginPath = path_1.default.dirname(scopedRequire.resolve(pluginName + "/package.json"));
        return localOnly ? void 0 : {
            path: pluginPath,
            pkgJson: require(pluginPath + "/package.json"),
        };
    }
    catch (err) {
        var pluginDir = path_1.default.resolve(relativeTo, "plugins", pluginName);
        var pkgJsonPath = path_1.default.join(pluginDir, "package.json");
        var pkgJson = (0, fs_tools_1.getFile)(pkgJsonPath);
        if (pkgJson && pkgJson.isFile()) {
            return {
                path: pluginDir,
                pkgJson: require(pkgJsonPath),
            };
        }
        return;
    }
};
exports.resolvePlugin = resolvePlugin;
var transpilePlugins = function (project, type, processApiModule, plugins) {
    if (plugins === void 0) { plugins = []; }
    plugins.forEach(function (plugin) {
        var pluginName = plugin.resolve;
        if (!pluginName)
            return;
        var projectRoot = project.projectRoot;
        var pluginDetails = (0, exports.resolvePlugin)(projectRoot, pluginName, type === "local-only");
        if (!pluginDetails)
            return; // We shouldn't transpile this plugin
        var pluginPath = pluginDetails.path, pkgJson = pluginDetails.pkgJson;
        project.linkPluginImports(pluginName);
        constants_1.apiTypeKeys.forEach(function (type) {
            var gatsbyModuleName = "./gatsby-" + type;
            var apiPath = (0, fs_tools_1.resolveFilePath)(pluginPath, gatsbyModuleName);
            if (!apiPath)
                return; // This `gatsby-*` file doesn't exist for this local plugin
            processApiModule({
                init: apiPath,
                project: project_1.Project.getProject({
                    apiType: type,
                    projectMeta: {
                        projectRoot: pluginPath,
                        projectName: pluginName,
                        pkgJson: pkgJson,
                    },
                    options: project.options,
                    propBag: project.propBag,
                }, true),
                unwrapApi: type === "node",
            });
        });
    });
};
exports.transpilePlugins = transpilePlugins;
//# sourceMappingURL=plugin-transpiler.js.map

Then based on the sourceMappingURL=plugin-transpiler.js.map hint, I looked in project for "plugin-transpiler.js" :

❯ grep -r plugin-transpiler .
Binary file ./.yarn/cache/gatsby-plugin-ts-config-npm-2.1.3-93aa5f6aab-e7003d3722.zip matches
./.cache/query-engine/index.js://# sourceMappingURL=plugin-transpiler.js.map
Binary file ./.cache/webpack/query-engine/index.pack matches
Binary file ./.cache/webpack/query-engine/0.pack matches

Which lead me to be https://github.com/Js-Brecht/gatsby-plugin-ts-config/blob/64e6a4b2db4e5c692e99db6399adbc6b7ad73045/src/lib/plugin-transpiler.ts

So, either this plugin needs to not require package.json or gatsby 4 needs to permit it be imported

[Js-Brecht]

Hi @cameronbraid... that is odd. I'll need to take a look at what they're doing that could cause that.

In the meantime, are you already using gatsby-ts? It's the next evolution of this plugin (since this isn't really a plugin anymore). Instead of running gatsby, you would run gatsby-ts as your CLI command. Maybe having it run before gatsby actually gets called will bypass the issue?

On the other hand, if they're maintaing some sort of virtual file system, it's possible that running it with gatsby-ts won't make any difference

[Js-Brecht]

@cameronbraid are you able to share the repository?

[cameronbraid]

Sorry, cant share this repo.

I will look at gatsby-ts

I will also see about setting up a standalone repro next week

[marcus13371337]

I'm having the same issue, I'm not using gatsby-ts

[marcus13371337]

Seems to be this line that's causing the issue: https://github.com/Js-Brecht/gatsby-plugin-ts-config/blob/481c6975deb1662c9a600e34dd0ba19dc8e8ca78/src/util/constants.ts#L6

[marcus13371337]

I think it has to do with the following note in the upgrade guide: https://www.gatsbyjs.com/docs/reference/release-notes/migrating-from-v3-to-v4/#bundling-external-files

Since there are new rendering options other than static nowadays, you are not allowed to import files as freely as before

[cameronbraid]

I've switched to gatsby-ts and now have a different issue. gatsby-ts develop works, however gatsby-ts build fails with at the same stage failed Validating Rendering Engines but with a different error

failed Validating Rendering Engines - 2.261s

 ERROR #98001  WEBPACK

Built Rendering Engines failed validation failed validation.

Please open an issue with a reproduction at https://github.com/gatsbyjs/gatsby/issues/new for more help

  Error: Module build failed (from ./.yarn/unplugged/gatsby-virtual-92ac5c178b/node_modules/gatsby/dist/schema/graphql-engine/webpack-remove-apis-loader.js):
  SyntaxError: unknown: 'Const declarations' require an initialization value. (17:25)

And the code in question from webpack-remove-apis-loader.js line 17 is the const result declaration below

module.exports = function loader(source) {
  const result = (0, _core.transformSync)(source, {
    babelrc: false,
    configFile: false,
    plugins: [[require.resolve(`../../utils/babel/babel-plugin-remove-api`), {
      apis: Object.keys(nodeApis).filter(api => !apisToKeep.has(api))
    }]]
  });
  return result === null || result === void 0 ? void 0 : result.code;
};

[cameronbraid]

Actually my above statement is incorrect, as I missed the code that was output just below the error - its line 17 in my gatsby-node.ts file

  > 17 | const onCreateBabelConfig : GatsbyNode["onCreateBabelConfig"] = ({ actions,  39m stage }) : void => {

[Js-Brecht]

@cameronbraid i believe that is probably the same problem as #57. I think Gatsby is doing some parsing & analysis on gatsby-node later on during the bootstrap, but it does not support Typescript.

I have yet to go and look at the new code where this is happening, but I’m pretty sure it’s going to be completely out of the control of this project. The best I could do is try to find some workaround, and potentially submit a pull request to add the typescript support in the core.

However, that support will probably wind up being included in the AOT Compilation RFC

[cameronbraid]

Ok, thanks. Unfortunately I cant use gatsby@alpha-ts-aot yet since I have a few local plugins

[Js-Brecht]

Not sure if @LekoArts is already aware of the issue here and in #57 when dealing with typescript gatsby-node modules, but I imagine that by redirecting its resolution to pre-preprocessed modules, the issue will have essentially resolved itself.

[Js-Brecht]

Ok, thanks. Unfortunately I cant use gatsby@alpha-ts-aot yet since I have a few local plugins

I just started looking through the diff on the AOT RFC. Maybe I can contribute some work to incorporate support for local plugins

[LekoArts]

Hi, thanks for the tag, I'm not aware of this issue or #57 -- I'd just encourage folks to try out the alpha and see if it works. If I understand it correctly people import stuff that has currently invalid syntax. From my (limited) testing Parcel happily compiles everything into JS that Gatsby understands so I'd assume it might just work.

[Js-Brecht]

If I understand it correctly people import stuff that has currently invalid syntax.

This plugin runs a JIT compiler for processing gatsby-node & gatsby-config, so your original config files remain in Typescript. Anything coming back later and reading them off the disk in order to process them will fail, unless the processing supports Typescript syntax.

Which is just a wordy way to say exactly what you said :laughing:

From my (limited) testing Parcel happily compiles everything into JS that Gatsby understands so I'd assume it might just work.

This is my assumption, too.

At this point, with the things that v4 is doing, trying to make this plugin work in tandem with Gatsby would likely be impossible without adding additional support to the core... so it seems to me that the current endeavor for first-class Typescript support is the best route to take.