Discussion: enable esbuild text loader for more additional file types

[erezrokah]

See https://github.com/floydspace/serverless-esbuild/issues/102 and https://esbuild.github.io/content-types/#text

Is enabling the text loader for additional files types something we would like to support?

Any other plugins/settings we would like to configure for our users?

[ehmicky]

This is a nice feature. However, this is non-standard and creates a stronger coupling to esbuild.

[erezrokah]

FYI, other bundlers have the same concept https://webpack.js.org/loaders/

[ehmicky]

The syntax is different though, and the set of formats not identical.

The concept might be as old as JavaScript bundling, e.g. RequireJS has it (maybe it created it?).

[eduardoboucas]

Any other plugins/settings we would like to configure for our users?

Worth noting that esbuild plugins carry a performance penalty, which can be pretty significant depending on the plugin. I think we should be conservative about plugins for this reason.

[spencewood]

I would appreciate the ability to include non-JavaScript/Json files in esbuild bundles. I currently have a function that uses sharp to generate images with custom fonts. It seems like a little bit of a hack, but in the function I can make use of fs to read the file and zisi faithfully bundles this up and everything works great with custom fonts. For example:

const currentDir = process.env.LAMBDA_TASK_ROOT || ".";

fs.readFileSync(path.join(currentDir, "fonts", "font.ttf"));

Others are doing things similar to this:

• https://answers.netlify.com/t/referencing-files-in-functions/30974
• https://answers.netlify.com/t/hosting-a-file-along-with-my-function/1527

With esbuild though, these files are not included, which is understandable given that the above seems like a workaround at best. Other attempts in getting these files into the bundle by simply requiring them of course yields error: No loader is configured for ".ttf".

It would be nice (critical if the old bundler is going away) to be able to accomplish this in some way, either through simply including raw, non-js files that are required by the function, some sort of build plugin that can include files in function bundles or opt-in configuration. For example:

[functions.image]
  node_bundler = "esbuild"
  include_files = "fonts/*"

[eduardoboucas]

Thanks for your input, @spencewood! We're currently working on adding this configuration option, with an API surface that looks very similar to the example you shared. We'll make sure this is in place before we deprecate the older bundler mechanism.

[canac]

+1 to this! I'd love to be able to load files as text so that I can import my GraphQL schema files. It would be awesome to just be able to do:

import schema from './schema.graphql';

[eduardoboucas]

It would be nice (critical if the old bundler is going away) to be able to accomplish this in some way, either through simply including raw, non-js files that are required by the function, some sort of build plugin that can include files in function bundles or opt-in configuration. For example:

[functions.image]
  node_bundler = "esbuild"
  include_files = "fonts/*"

We now have this feature available for testing while we finalize the interface and prepare the docs. You can include additional files with function deployments that are not statically imported using require/import statements.

If you'd like to try it out, you can use the new functions.included_files configuration property like so:

Example project:

files/
  one.json
  two.json
netlify/
  functions/
    hello.js
netlify.toml

netlify.toml:

[functions]
  included_files = ["files/*"]

netlify/functions/hello.js:

exports.handler = async function (event, context) {
  const { name } = event.queryStringParameters;
  const data = require(`../../files/${name}.json`);

  return {
    statusCode: 200,
    body: JSON.stringify(data),
  };
};

When accessing /.netlify/functions/hello?name=one, files/one.json will be dynamically loaded and returned in response body.

Items in the included_files array are globs, so you could include files based on the filename (e.g. files/prefix-*) or extension (e.g. files/*.json). You can exclude certain paths by prefixing the glob with a question mark, similarly to how .gitignore files work (e.g. !files/three.json).

Since this is an experimental feature, it's possible that some of the functionality might change before we make a public release. We'd love for you to it before then and hear your feedback so that we can deliver the best experience possible.

Thanks!

[spencewood]

Thanks for this! Based on a quick test, it does indeed appear to work as I would expect and my external assets are bundled correctly! I have a complicating factor with my current code and primary use-case for this however, because I'm also importing .tsx (which I'll submit an issue for separately).

[canac]

Quick question: What is the expected file structure when using included_files? My functions directory is "src/lambda/". This is the directory structure when running netlify dev. Notice the deep nesting and redundant src directories.

When I comment out the included_files line, it looks like this, which seems a lot more reasonable.

Is this a bug or a known change to the file structure?

[eduardoboucas]

@canac That is expected behaviour. One of the src directories comes from your project, whereas the other is introduced by zip-it-and-ship-it — they just happen to have the same name.

We're planning on changing this sometime soon, but there's nothing to worry about here.

[hector]

include_files does not solve the issue about text files not able to be loaded, right?

[mfanuzzi]

Just wanted to post that we're using this feature now and it's a lifesaver. In our case, we're storing templates for email notifications along with a function that sends those notifications. Makes loading them super easy:

  const notificationTemplate = fs.readFileSync(
    `./functions/assets/${formName}-notification.html`,
    'utf8',
  );

this is with a netlify.toml of:

[functions]
  included_files = ["functions/assets/*"]

Thanks for this great feature!