firebase.json 'functions.ignore' bloats package upload

[repentsinner]

Environment info

firebase-tools:

firebase --version
7.2.2

Platform: Both MacOS (10.14.6 (18G87)) & Windows 10 development environments.

Test case

MCVE: Firebase functions template + a 'functions.ignore' section in firebase.json

Steps to reproduce

Create a firebase project.

Run firebase init functions with default options, and add it to your new project:

firebase init functions

Uncomment the helloWorld sample function in index.js and deploy:

⋊> ~/D/f/functions firebase deploy                                                                                    11:02:38

=== Deploying to 'test-function-deploy-3a957'...

i  deploying functions
i  functions: ensuring necessary APIs are enabled...
✔  functions: all necessary APIs are enabled
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (22.95 KB) for uploading
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 8 function helloWorld(us-central1)...
✔  functions[helloWorld(us-central1)]: Successful update operation. 

✔  Deploy complete!

Project Console: https://console.firebase.google.com/project/test-function-deploy-3a957/overview
⋊> ~/D/f/functions

Note the deployed packaged functions size (22.95KB).

Edit firebase.json to include a 'functions.ignore' section:

{
  "functions": {
    "ignore": []
  }
}

Deploy again:

⋊> ~/D/f/functions firebase deploy                                                                                    11:03:24

=== Deploying to 'test-function-deploy-3a957'...

i  deploying functions
i  functions: ensuring necessary APIs are enabled...
✔  functions: all necessary APIs are enabled
i  functions: preparing functions directory for uploading...
i  functions: packaged functions (9.07 MB) for uploading
✔  functions: functions folder uploaded successfully
i  functions: updating Node.js 8 function helloWorld(us-central1)...
✔  functions[helloWorld(us-central1)]: Successful update operation. 

✔  Deploy complete!

Project Console: https://console.firebase.google.com/project/test-function-deploy-3a957/overview
⋊> ~/D/f/functions 

Note that the deployed package has grown from 22.95KB to 9.07MB.

Expected behavior

I expect that if things are ignored, the deployed package gets smaller rather than larger.

Actual behavior

The deployed package increases in size by several orders of magnitude from 22.95KB to 9.07MB.

For reference, the size of the functions folder including local node_modules is 29MB in this case. The node_modules folder itself is 28.9MB on disk, and is ~6MB gzipped.

This also affects TypeScript projects, presumably because the issue is triggered after the tsc part of things.

Comment

I haven't yet found any good/definitive documentation for the firebase.json configuration file and specifically the functions section. There is some documentation of the hosting section here. My understanding of thefunctions.ignore structure/usage is currently based on https://github.com/firebase/firebase-tools/issues/291#issuecomment-322526710.

There is a related issue #429 where @laurenzlong indicates that the ignore section should be operational/fixed.

Open question

Where is the deployed package built to locally, before deploying? I ran with --debug and couldn't figure out any useful file paths.

Thanks!

Thanks for considering! And my phone tethering data plan thanks you for resolving this, as my current project deploy is ~55MB before I figured out what was going on :)

[bkendall]

The behavior that you described is the behavior that I would expect.

The documentation does not describe functions.ignore, as you pointed out; however, it is used by the CLI.

Unofficially (in that it is undocumented), firebase.ignore has a default of ["node_modules"]. This lets us limit the amount of stuff we have to upload during the deploy (which you can imagine could become very large, very quickly). By setting firebase.ignore in package.json, it tells the CLI that nothing should be ignored. Now, the node_modules folder is included in the archive and sent to Functions.

To answer your question: at a high-level description of the process, the CLI doesn't build a package locally. It does roughly the following:

1. Runs any predeploy script for functions (also specified in firebase.json).
2. Parses the source code for functions to deploy.
3. Zips the source code (usually ignoring node_modules) and uploads it to Functions.

Once the source code is uploaded, the Functions infrastructure runs npm install for the package.json and the source is ready to go and serve users (at least, in this super hand-wavy explanation).

I hope that answers your questions. But since there's no CLI-related bug here, I'm going to close this issue.

[repentsinner]

Thanks for the info @bkendall.

Based on #291 I mistakenly thought that the CLI always excluded node_modules, and that the stuff in functions.ignore was additive. Adding an explicit pattern for node_modules does do what you describe.

[lookfirst]

@bkendall Why not document it?

[steveoh]

There is very little documentation about the firebase.json schema and how it is used. Overall the firebase documentation is very thorough, the organization is difficult for me to follow, and the firebase.json is almost completely missing. These defaults should be documented.

[bkendall]

A lot of times, I don't pay too much attention to closed threads, but I saw SO MANY +1s on these last comments that we really, really should actually take care of this.

I've filed an internal bug (238805712) to update the documentation around this and made sure the team knows about it. No promises on timing, but I think we can make this happen.

[lookfirst]

@bkendall Thanks for listening to your users.

[loomchild]

Perhaps it would make sense to add other log files generated by emulator apart from firebase-debug, since they can also grow a lot (not sure if it covers all of them):

"firestore-debug.log"
"firestore-debug.*.log"
"pubsub-debug.log"
"pubsub-debug.*.log"
"ui-debug.log"
"ui-debug.*.log"

Or perhaps even simpler, ignore "*.log"?

[bobatsar]

@bkendall for development, is there a way to create the sources.zip file without deploying it? I would like to adapt the functions.ignore and inspect the .zip

[maganap]

@bobatsar I don't really know how to keep the local zip file. But you can always see the function in the console: https://console.cloud.google.com/functions/list

Click on the name of the function you want to inspect, and you'll find a SOURCE tab. There you can see the uploaded source code and download a zip file as well.