Cookbook about sharing code between api and web side

[peterp]

Since we’re using yarn workspaces I would recommend creating a new folder called “packages/” in the root and create a shared npm package in that folder.

[chris-hailstorm]

@peterp Is it necessary to make the shared code an npm package? For a while now I've had a shared code folder at the same level as web and api (mine is uti for utilities) -- all I've had to do is add a path alias in my web/config/webpack.config.js like this:

module.exports = (config, { env }) => {
  config.resolve.alias['@uti'] = path.join(redwoodPaths.base, 'uti')

And then import the shared code by path like this:

import { enviro } from '@uti/netlify'

Webpack handles all the packaging. A lot easier than the npm packaging lifecycle.

[peterp]

@chris-hailstorm That's a great solution and probably a lot easier than the route I was suggesting. Another way to do this, for both sides, could be by specifying it as an alias in babel.config.js

One of the reasons why I like to share packages is because that's the natural way of sharing code in the node ecosystem and I try not to stray too far from that path, since you may want to share this package outside of your project eventually.

[chris-hailstorm]

@peterp I'd use the package approach if shared stuff is bigger / reusable, agree with you there.

And if it's smaller (one-off functions, regexes, constants) -- I'd probably not package it.

As usual, developer judgment is involved!

[noire-munich]

@chris-hailstorm thanks a lot for your solution, I stumbled upon this for days ><. I can get it to work on the web side no problem but not on the API, odd enough as my api webpack configuration is rather thin:

const { getPaths } = require('@redwoodjs/internal')
const path = require('path')

module.exports = (config) => {
  // Module working alias.
  config.resolve.alias['@shared'] = path.join(getPaths().base, 'shared/dist')

  return config
}

Did you have any issue on any of the sides? It's working well on the web side for me.

[krinoid]

@noire-munich I was able to make it work on the API side with changing root babel.config.js and using part of your config (thanks!):

const path = require('path')
const { getPaths } = require('@redwoodjs/internal')

module.exports = {
  presets: ['@redwoodjs/core/config/babel-preset'],
  plugins: [
    [
      require.resolve('babel-plugin-module-resolver'),
      {
        alias: {
          common: path.join(getPaths().base, 'common')
        },
      },
    ],
  ],
}

You'll need to run yarn add -D babel-plugin-module-resolver -W at the root level to install required babel plugin.

[noire-munich]

Thanks for the info, @krinoid ! I got it to work on both sides as well, but it crashed my deploys ><.

My shared package is as the project's root directory and is a typescript package, locally it runs just fine but on netlify it needs to be babelled to ./dist/*.js. It works if I commit those dist files but I'm feeling uncomfortable about this. Another way would be to modify the build command, which is cleaner.

I was wondering how you handled this, if you had to at all.

[peterp]

Hey @noire-munich,

You could modify the netlify build command to also build your package:

[build]
  command = "NODE_ENV=production yarn rw db up --no-db-client --auto-approve && yarn rw build && cd mypackage; yarn build"
  # ... rest of toml ...

I added && cd mypackage; yarn build

[noire-munich]

Hi @peterp ! Thanks! I ended up doing this indeed, actually I added a script to the root package.json and the build command calls it.

This is very convenient for prod build, is there some sort of equivalent to modify the dev command? It would let me have this extra package being watched along the sides of the application.

[peterp]

@noire-munich That's something that I want to add:

In the meantime you could do something like: https://github.com/redwoodjs/redwood/blob/b72db535c84a0656a28d1fa2dd2a9460d3bfd78f/packages/api/package.json#L41

[jangxyz]

I see that webpack/babel configurations above are for aliasing the path, but I have another problem.

I can't make the api side pick up exported modules or typescript files. Is there some other configurations I need to do?

I thought the babel.config.js file on the root containing the @redwoodjs/core/config/babel-preset would handle it, but I guess not.

---

What I mean (on 0.24.0):

1. Sharing CJS modules

sharing/sharing_js.js:

module.exports = { value: 'shared value'}

This works fine on both web and api (w/o path aliasing).

✅ web/src/pages/HomePage/HomePage.js:

import { value } from 'src/../../sharing/shared_js'

const HomePage = () => {
  return (
    <>
      <p>sharing value: {value}</p>
    </>
  )
}
export default HomePage

✅ api/src/functions/graphql.js:

import { value } from '../../../sharing/shared_js'
console.log('file: graphql.js', { value })

2. Sharing esm modules

sharing/sharing_export.js:

export const value2 = 'share through export'

This works only for the web side.

✅ web/src/pages/HomePage/HomePage.js:

import { value } from 'src/../../sharing/shared_js'
import { value2 } from 'src/../../sharing/shared_export'

const HomePage = () => {
  return (
    <>
      <p>sharing value: {value}</p>
      <p>sharing value2: {value2}</p>
    </>
  )
}
export default HomePage

❌ api/src/functions/graphql.js:

import { value } from '../../../sharing/shared_js'
import { value2 } from '../../../sharing/shared_export'
console.log('file: graphql.js', { value })
console.log('file: graphql.js', { value2 })

with error:

SyntaxError: Unexpected token 'export'

3. Sharing typescript files (with tsconfig.json file added)

sharing/shared_ts.ts:

export const value3 = 'shared typescript value'

Same thing happens on both web (✅ ) and api (❌ ), as 2.

[peterp]

@jangxyz you would have add your "sharing folder" to your paths.

So, the first thing I'm seeing is that you're using a relative import. I would modify the api/babel.config.js babel configuration to an add alias for "sharing":

const path = require('path')
const { getPaths } = require('@redwoodjs/internal')

module.exports = {
  presets: ['@redwoodjs/core/config/babel-preset'],
  plugins: [
    [
      require.resolve('babel-plugin-module-resolver'),
      {
        alias: {
          '~sharing': path.join(getPaths().base, 'sharing')
        },
      },
    ],
  ],
}

Then you would have to do the same thing for your api/{js,ts}config.json file:

{
  "compilerOptions": {
    "baseUrl": ".",
    "rootDir": "src",
    "noEmit": true,
    "esModuleInterop": true,
    "strictNullChecks": true,
    "module": "ESNext",
    "target": "ESNext",
    "allowJs": true,
    "moduleResolution": "Node",
    "jsx": "preserve",
    "paths": {
      "src/*": ["./src/*"],
      "~sharing/*": ["../sharing/*"]
    },
  },
  "include": ["src/**/*", "../.redwood/index.d.ts"]
}

[jangxyz]

@jangxyz you would have add your "sharing folder" to your paths.

Actually, that was the first thing I tried, following along the thread. However it did not make any difference with the result.

I figured babel config (and accompanying tsconfig) was for doing alias imports, and while I do like assigning module aliases, it has less to do with the SyntaxError problem it is facing.

I left the path configurations untouched on purpose because I thought that had nothing to do with the errors.

Here is a screenshot of (hopefully) presenting the problem I have, following your steps.

While not shown, both tsconfig.json and jsconfig.json files are updated as well.

※ here's a sample repository with the code above: https://github.com/jangxyz/redwoodjs-sharing

[peterp]

@jangxyz I think you should only have either jsconfig.json or tsconfig.json, but not both.

[jangxyz]

I don't think having both jsconfig.json and tsconfig.json is the key problem. AFAIK jsconfig is picked up by IDE and not the build process, and they are set up in both web/ and api/ sides anyway. Currently the only side that's facing the problem is the api side.

However, even I follow your suggestion the problem still remains without one of the configuration files. I've updated the sample repo too (tag:with-tsconfig-only).

Meanwhile, I find the other thread in the community raises a similar issue about transpiling from es6 modules:

I noticed it’s not being transpiled from es6 modules (and thus throwing errors). I see the passage about overriding webpack config for web, https://redwoodjs.com/guides/webpack-config.html#overriding-webpack-config, but what about api side, is typescript transpiling that?

But unfortunately that part seems to be ignored / resolved through the process ;( It could be just a regular babel, webpack configuration issue, only I am not that much skilled with them.

Reiterating the problem:

• Redwood fails to transpile es6 modules, either by JS or TS, when the code is outside the web and the api side.
• It only happens on the api side.

If you think this is a separate topic to the original subject, I'm okay on splitting it into another issue.

[thedavidprice]

Current Status:

Keep on v1-consideration list as a nice-to-have for v1. Would not be a breaking change in the future.

Short-term: create a doc or guide with options?

[jangxyz]

In case anyone is interested, here's a new discovery we have came up with: sharing typescript codes directly between api/ and web/, using babelrcRoots:

This option allows users to provide a list of other packages that should be considered "root" packages when considering whether to load .babelrc.json files.

The key is to assign the both current workspace and the other directory to babelrcRoots config in webpack.

Without this option set, importing api from the web works only for javascript files because yarn workspace has created a symlink in the node_modules directory for us. Importing a package under node_modules assumes loading already built javascript file instead of typescript. We want to tell to use the babelrc inside api/, but my understanding is that babel tries to avoid doing this in version 7. By setting this option it will load the babel configurations in the specified directory, and allows us to use typescript files directly.

So far we are using it to share custom type files from the api side.

Here's how we've managed our web/config/webpack.config.js file:

// web/config/webpack.config.js
module.exports = (config) => {

  // ...

  config.module.rules.forEach((rule) => {
    (rule.oneOf || []).forEach((oneOfRule) => {
      if (Array.isArray(oneOfRule.use)) {
        oneOfRule.use
          .filter((use) => use.loader === 'babel-loader')
          .forEach((use) => {
            applyBabelLoaderConfig(use.options); // found babel loader? apply code below
          });
      }
    });
  });
  function applyBabelLoaderConfig(options) {
    options.babelrcRoots = ['.', '../api'];
  }

  // ...

  return config;
};

[viperfx]

Just following this thread.

If I am understanding correctly, if the project uses TS we should use @jangxyz solution

But otherwise, with JS project @peterp or @krinoid works just fine for sharing utils and bits of code that can be used with both api and web side.

[peterp]

@viperfx I like to think of it the following:

1. If you want to share types, use paths in tsconfig.json
2. If you have to share code use babel alias and paths in tsconfig.json

[viperfx]

I setup the paths in jsconfig.json but the server is having issues finding the module when I import it on the api side.

Has anyone got this working? Also is esbuild enabled?

[pi0neerpat]

Here's my docs - compilation of the above with my own opinions :smile:

0. Make a package in /packages/ like super-cool-function

1. Install babel-plugin-module-resolver and update root babel.config.js:

yarn add -D babel-plugin-module-resolver -w

const path = require('path')
const { getPaths } = require('@redwoodjs/internal')

module.exports = {
  presets: ['@redwoodjs/core/config/babel-preset'],
  plugins: [
    [
      require.resolve('babel-plugin-module-resolver'),
      {
        alias: {
          '@common': path.join(getPaths().base, 'packages'),
        },
      },
    ],
  ],
}

2. Update web jsconfig.json

{
  "compilerOptions": {
    "paths": {
      "src/*": ["./src/*", "../.redwood/types/mirror/web/src/*"],
      "types/*": ["./types/*"],
      "@common/*": ["../packages/*"]
    },

3. Use in web like

import superCoolFunction from "@common/super-cool-function"

4. Use in api like

import superCoolFunction from "../../../packages/super-cool-function"

NOTE: For api I cannot get alias import working, even if I update the jsconfig.json

[thedavidprice]

@dac09 following up on my message in Discord — what do you think about leading the effort to turn this into proper documentation?

[thedavidprice]

Update

We discussed this at the Core Team meeting and would ideally like to have official support for this — currently two ideas:

1. Can we load ALL the babel config for a shared folder (not a package) ...or...
2. can we copy the files across to the respective folders as part of the build process (beautiful hack)

@dac09 is taking the lead on next steps

[pi0neerpat]

Update for those still struggling getting things to work on the API side. Once you complete the above https://github.com/redwoodjs/redwood/issues/531#issuecomment-958118622 theres a few more steps you can do to have a pretty decent setup. Unfortunately the api side won't transform your ESM code to commonjs, so you'll need to build each package. Bummer, I know, but hopefully there's a native redwood solution soon. Until then....The following steps should be done for each package you want to use on the API side.

First add .babelrc.js to your package:

module.exports = {
  extends: '../../babel.config.js',
  plugins: ['@babel/plugin-transform-modules-commonjs'],
}

Now add the build commands to your package, and update the entry point to use the built CommonJS files. Note that I've intentionally excluded a "module" entry point, since that causes issues. Also, while nodemon can conveniently re-build your package, it won't update on the API side until you trigger a re-build there as well.

{
  "name": "@treasure-chess/treasure-html",
  "version": "1.0.0",
  "main": "dist/index.js",
  "scripts": {
    "build": "babel src -d dist",
    "build:watch": "nodemon --watch src --ext \"js,ts,tsx\" --ignore dist --exec \"yarn build\""
  },
  ...
  "devDependencies": {
    "@babel/node": "^7.16.0",
    "@babel/preset-env": "^7.16.0",
    "nodemon": "^2.0.15"
  }
}

Now, since this package always must be built before it can be used, we need to specify this in our redwood build script. Update redwood.toml to automatically build each package:

[build]
  command = "NODE_ENV=production yarn rw db up --no-db-client --auto-approve && yarn rw build && cd packages/treasure-html; yarn build"

All redwood projects include the packages/* workspace by default, so you should be able to install this local package now in your API.

# in /api
yarn add @treasure-chess/treasure-html

If that doesn't work, just add it manually to package.json, making sure your versions match exactly, then run yarn.

Your API .babelrc.js can be left alone, and same goes for jsconfig.json

module.exports = {
  extends: '../babel.config.js',
}

Finally, you can use your local package in the API side like so:

// /api
import getCard from '@treasure-chess/treasure-html'

And you can continue to use the @common syntax that you've been enjoying on the WEB side. Its weird to have alternate imports for the same package, but once this issue is resolved, I imagine both sides will be using something like @common

// /web
import getCard from '@common/treasure-html'

Now you can share packages across WEB and API. Good luck!

[thedavidprice]

Next Steps (experimental):

• [ ] create a watch/copy to build web/shared and api/shared from ./shared

cc @dac09

[0x15F9]

Hello @dac09, what is the progress on this?

[dac09]

Hi @0x15F9 - no this isn't something we are likely to ship until we overhaul the build system, and potentially introduce bundling - there's a lot of edgecases that creep up! However if you follow this thread you can find some suggestions on how to set it up yourself (with limitations)

[0x15F9]

@dac09 I followed the above steps and got it to work. However, when I tried adding jest to the package, rw test would fail quoting

Validation Error:

Watch plugin jest-watch-typeahead/filename cannot be found. Make sure the watchPlugins configuration option points to an existing node module.

Running the test within the package passes without issue

[0x15F9]

As a work around, I am not installing jest in the package. I call a global installation of jest from the cli when I need to test the package.

[dphuang2]

Just dropping my experience here.

I have standalone TypeScript modules under packages/* and all I had to do was add babel.config.js with the following content:

module.exports = {
  // https://www.reddit.com/r/webpack/comments/jw516b/webpack_is_replacing_exports_with_webpack_exports/
  sourceType: 'unambiguous',
}

And I can import the modules under web/ and api/.

[majimaccho]

Hi. Thank you for your grate work on this framework. I'm currently working on this.

Is there any workaround about this issue for current version: 3.4.0? I'd like to share logic about date with web and api.

[will-ks]

I just recently got this working on 6.2.2, taking a slightly different approach than others in this thread.

I didn't want to mess with babel configs since I want to be sure not to break anything in future RW upgrades. My project is using TypeScript, so I decided to just compile all my shared packages so the api and web sides can import the compiled code instead of transpiling the TypeScript code.

The api side expects imported modules in commonjs format, while the web side expects ES modules, so I compile the shared code using both formats.

So the approach is:

1) For each package you want to share, your package.json needs these additions:

  "main": "dist/cjs/index.js",
  "module": "dist/mjs/index.js",
  "exports": {
    ".": {
      "import": "./dist/mjs/index.js",
      "require": "./dist/cjs/index.js"
    }
  },
  "scripts": {
    "build": "tsc --module esnext --outDir dist/mjs && tsc --module commonjs --outDir dist/cjs && echo '{ \"type\": \"commonjs\" }' > dist/cjs/package.json && echo '{ \"type\": \"module\" }' > dist/mjs/package.json"
  },

Also make sure your tsconfig.json is set up to emit compiled files, declarations and source maps. Eg:

  "compilerOptions": {
    "noEmit": false,
    "declaration": true,
    "sourceMap": true,
  }

Now when you run the yarn build script, you'll get a dist folder with both commonjs and ES module outputs.

2) Add a script to your root package.json to watch your packages for changes and rebuild them, eg:

  "scripts": {
    "build-packages": "yarn workspace foo build && yarn workspace bar build",
    "build-packages:watch": "nodemon --watch packages/foo/src --watch packages/bar/src --ext \"js,ts,tsx\" --exec \"yarn build-packages\"",
  }

3) Install concurrently at the root level and modify your dev script to concurrently run your build-packages:watch script along with rw dev:

  "scripts": {
    "dev": "concurrently \"rw dev\" \"yarn build-packages:watch\"",
  }

4) Now you can import your package in both the api and web sides just using the package name, eg if I have packages/foo I can just do

import x from 'foo'

Yarn already creates symlinks in node_modules to all your workspace packages, so they're just treated as standard dependencies.