Deploying a Google Cloud HttpCallbackFunction uses npm instead of yarn

0xJem commented 1 year ago

What happened?

I have a Google Cloud Function deployed by Pulumi. I use yarn the manage the dependencies, and there is a yarn.lock file present in the repo. At deployment-time, I can see in the logs that Google Cloud Build (I assume) is complaining about a missing package-lock.json file, and needing to generate one.

Expected Behavior

Yarn and the yarn.lock file should be used.

Steps to reproduce

Create project
Create GC function
Add yarn.lock file
Deploy

Output of `pulumi about`

CLI
Version 3.55.0 Go Version go1.19.5 Go Compiler gc

Plugins NAME VERSION gcp 6.45.0 nodejs unknown

Host
OS darwin Version 13.2.1 Arch arm64

This project is written in nodejs: executable='/Users/ZZZ/.nvm/versions/node/v18.12.1/bin/node' version='v18.12.1'

Backend
Name pulumi.com URL https://app.pulumi.com/0xJem User 0xJem Organizations 0xJem

Dependencies: NAME VERSION @datasert/cronjs-matcher 1.2.0 @google-cloud/opentelemetry-cloud-trace-exporter 2.0.0 @google-cloud/pubsub 3.2.1 @opentelemetry/api 1.4.0 @opentelemetry/instrumentation-express 0.32.1 @opentelemetry/instrumentation-http 0.35.1 @opentelemetry/sdk-trace-node 1.9.1 @pulumi/gcp 6.45.0 @pulumi/pulumi 3.50.2 auth0 3.1.2 axios 1.2.3 catch-unknown 1.0.0 compression 1.7.4 cors 2.8.5 cron-validate 1.4.5 date-and-time 2.4.2 email-validator 2.0.4 express 4.18.2 express-jsdoc-swagger 1.8.0 express-oauth2-jwt-bearer 1.3.0 firebase-admin 11.4.1 openapi-types 12.1.0 randomstring 1.2.3 swagger-ui-express 4.6.0 @types/auth0 2.35.9 @types/compression 1.7.2 @types/cors 2.8.13 @types/express 4.17.14 @types/jest 29.2.5 @types/jest-when 3.5.2 @types/node 18.11.18 @types/randomstring 1.1.8 @types/supertest 2.0.12 @types/swagger-ui-express 4.1.3 @typescript-eslint/eslint-plugin 5.48.0 @typescript-eslint/parser 5.48.0 dotenv 16.0.3 eslint 8.31.0 eslint-config-prettier 8.6.0 eslint-plugin-import 2.26.0 eslint-plugin-jest 27.2.1 eslint-plugin-no-relative-import-paths 1.5.2 eslint-plugin-prettier 4.2.1 eslint-plugin-simple-import-sort 8.0.0 eslint-plugin-unused-imports 2.0.0 jest 29.3.1 jest-silent-reporter 0.5.0 jest-when 3.5.2 json-to-pretty-yaml 1.2.2 prettier 2.8.1 supertest 6.3.3 ts-jest 29.0.3 ts-node 10.9.1 typescript 4.9.4

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

rquitales commented 1 year ago

Hi @0xJem

Could you provide a minimal Pulumi code sample to reproduce this issue? In particular, how are you doing steps 3 and 4 (add yarn.lock file, deploy).

I'm unable to reproduce this issue as Cloud Build appears to be using yarn to handle dependencies when I have yarn.lock present.

I'm using the https://github.com/GoogleCloudPlatform/nodejs-docs-samples repo, with the following sub-dir as example function to deploy: nodejs-docs-samples/functions/helloworld/helloworldHttp. I've run yarn install in this folder to generate a yarn.lock and zipped this folder to use it as the source for the Cloud Function I'm deploying.

This is the Pulumi code I am using:

import * as pulumi from "@pulumi/pulumi";
import * as gcp from "@pulumi/gcp";

const bucket = new gcp.storage.Bucket("bucket", {location: "US"});
const archive = new gcp.storage.BucketObject("archive", {
    bucket: bucket.name,
    source: new pulumi.asset.FileAsset("/Users/rquitales/code/gcp/source/function-source.zip"),
});
const _function = new gcp.cloudfunctions.Function("function", {
    description: "My function",
    region: "us-central1",
    runtime: "nodejs18",
    availableMemoryMb: 128,
    sourceArchiveBucket: bucket.name,
    sourceArchiveObject: archive.name,
    triggerHttp: true,
    entryPoint: "helloHttp",
});
// IAM entry for all users to invoke the function
const invoker = new gcp.cloudfunctions.FunctionIamMember("invoker", {
    project: _function.project,
    region: _function.region,
    cloudFunction: _function.name,
    role: "roles/cloudfunctions.invoker",
    member: "allUsers",
});

As an aside, GCB will use Yarn to manage dependencies if it detects a yarn.lock file in the source code it's building from. Can you double check that you are indeed adding your yarn.lock file to your Cloud Functions source.

Thanks

0xJem commented 1 year ago

Hello! Sorry for the delay with this.

I was able to create a new project and reproduce it.

package.json:

{
    "name": "pulumi-test",
    "main": "src/index.ts",
    "devDependencies": {
        "@types/express": "^4.17.17",
        "@types/node": "^16"
    },
    "dependencies": {
        "@pulumi/gcp": "^6.51.0",
        "@pulumi/pulumi": "^3.58.0",
        "express": "^4.18.2"
    }
}

src/index.ts:

import * as gcp from "@pulumi/gcp";
import express from "express";

const frontendFunction = new gcp.cloudfunctions.HttpCallbackFunction("test", {
    callback: (req: express.Request, res: express.Response) => {
        // Nothing
    }
});

Deploying this to GCP, I can clearly see in the logs that it's running npm install and ignoring the present of the yarn.lock file.

0xJem commented 1 year ago

Hi @rquitales , were you able to reproduce this?

t0yv0 commented 1 year ago

Having a quick look at this issue.

gcp.cloudfunctions.HttpCallbackFunction is implemented as a mix-in in the code below:

https://github.com/pulumi/pulumi-gcp/blob/be63ce85b7be749e1c5a41c967e9d8bb4f7a120b/sdk/nodejs/cloudfunctions/zMixins.ts#L256

HttpCallbackFunction is a convenience that makes inline functions work, but that has the cost of conflating the Pulumi project dependencies with the runtime dependencies of the callback code.

Examining the code, HttpCallbackFunction currently does not support lock files, and it only auto-detects and sends over the wire the package.json file.

I'm going to re-classify this ticket as a feature request to support detecting NPM and YARN lock files in HttpCallbackFunction.

As a workaround for now unpacking the HttpCallbackFunction convenience into the more verbose form as suggested by @rquitales above should do the trick - use separate projects for Pulumi and for the function itself, use HttpCallbackFunction and pulumi.asset.FileAsset, and include yarn.lock inside the FileAsset, which should make YARN behave correctly.

pulumi / pulumi-gcp