gcs-fetcher is missing some files during the unzipping of my App Engine source code.

[nicolasdao]

Affected builder image

gcr.io/cloud-builders/gcs-fetcher

Expected Behavior

After the nodeJS source code is zipped and successfully uploaded to Google Cloud Storage, a request to deploy that package to a new GA version via the API POST https://appengine.googleapis.com/v1/{parent=apps/*/services/*}/versions should start the first build step which fetches the zip and unzip it.

A month ago, the fetcher build step in the Google Cloud Build history would start with this:

Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
Fetching archive gs://my-bucket/my-file.zip.
******************************************************
Status: SUCCESS
...

Actual Behavior

Today, the fetcher build step in the Google Cloud Build history starts with this:

Already have image (with digest): gcr.io/cloud-builders/gcs-fetcher
WARNING: -type=Archive is deprecated; use -type=ZipArchive
Fetching archive gs://my-bucket-2/my-file.zip.
******************************************************
Status: SUCCESS
...

The deployment works and the source code is unzipped, but it is missing some files. Files in the app root folder are ok, as well as files nested under a root folder, but all files nested deeper than that are missing (e.g., src/index.js is ok, but src/repos/index.js is missing).

Steps to Reproduce the Problem

1. Create a dummy nodeJS REST API with the following code:

// index.js
const { app } = require('@neap/funky')
const fs = require('fs')
const path = require('path')

function filewalker(dir, done) {
    let results = []

    fs.readdir(dir, function(err, list) {
        if (err) return done(err)

        var pending = list.length

        if (!pending) return done(null, results)

        list.forEach(function(file){
            file = path.resolve(dir, file)

            fs.stat(file, function(err, stat){
                // If directory, execute a recursive call
                if (stat && stat.isDirectory()) {
                    // Add directory to array [comment if you need to remove the directories from the array]
                    results.push(file)

                    filewalker(file, function(err, res){
                        results = results.concat(res)
                        if (!--pending) done(null, results)
                    })
                } else {
                    results.push(file)

                    if (!--pending) done(null, results)
                }
            })
        })
    })
}

app.get('alive', (req,res) => {
    filewalker(__dirname, (err, files) => {
        if (err)
            res.status(400).send(err)   
        else
            res.status(200).send(files.join('</br>'))
    })
})

eval(app.listen(process.env.PORT || 3000))

This hosts an API at /alive that lists all the files in the app root directory. Make sure that you add a dummy file under src/test/index.js.

1. Use node archiver https://www.npmjs.com/package/archiver to zip your project.
2. Upload that zip to Google Cloud Storage.
3. Use the API described in the Google Doc here to deploy a new GAE (standard mode).
4. The deployment should work, and when you request your /alive endpoint in your browser, you should receive a list of all the files. You'll notice that the src/test/index.js is missing.

Additional Info

[bendory]

@squee1945 @ImJasonH Can you PTAL and post your thoughts?

[squee1945]

Thanks for the detailed repro.

I'm unfamiliar with https://www.npmjs.com/package/archiver - is there a CLI to use it, or does it need code? Can you provide more info on how you create your zip archive?

[bendory]

Ping @nicolasdao is this still happening? Can you please provide the requested info?

[nicolasdao]

Hi @bendory ,

I apologize, but I'm unable to confirm if this issue is still happening. Because I could not overcome it, I reimplemented my code to by-pass this issue. Instead of uploading a zip to Google Cloud Storage and then requesting a deployment to GAE from that zip, I uploaded each file one by one under a bucket, and then request the GAE deployment. That worked, and I never tried the zip approach after that incident.

[bendory]

Thanks for the reply @nicolasdao; I'm closing this out as "not reproducible."