"Unhandled error cleaning up build images." - deploying from same GCP project

[akauppi]

[REQUIRED] Environment info

• firebase-tools: 11.3.0
• Platform: Cloud Build; same GCP project as the Firebase project

[REQUIRED] Test case

• deploy a Firebase project with Cloud Functions, "onto itself", i.e. using the same GCP project that Firebase has automatically created.

[REQUIRED] Steps to reproduce

skip

[REQUIRED] Expected behavior

Deployment works, since the project has enough access rights.

Firebase would clean up Cloud Function deployment cache, as it aims to do.

[REQUIRED] Actual behavior

Deployment fails.

The cleanup fails.

Here's an excerpt from the log:

Step #6: === Deploying to 'groundlevel-jun-22'...
Step #6: 
Step #6: i  deploying firestore, functions
...
Step #6: i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
Step #6: i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
Step #6: i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
Step #6: ✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
Step #6: ✔  functions: required API cloudbuild.googleapis.com is enabled
Step #6: ✔  functions: required API cloudfunctions.googleapis.com is enabled
Step #6: i  functions: preparing codebase default for deployment
Step #6: i  functions: preparing ./functions directory for uploading...
Step #6: i  functions: packaged /workspace/packages/backend/functions (53.92 KB) for uploading
Step #6: i  firestore: latest version of ./firestore.rules already up to date, skipping upload...
Step #6: ✔  firestore: deployed indexes in ./firestore.indexes.json successfully
Step #6: ✔  functions: ./functions folder uploaded successfully
Step #6: ✔  firestore: released rules ./firestore.rules to cloud.firestore
Step #6: i  functions: updating Node.js 16 function userInfoShadow_2(europe-west2)...
Step #6: ⚠  functions: failed to update function projects/groundlevel-jun-22/locations/europe-west2/functions/userInfoShadow_2
Step #6: Failed to update function projects/groundlevel-jun-22/locations/europe-west2/functions/userInfoShadow_2
Step #6: 
Step #6: Functions deploy had errors with the following functions:
Step #6:    userInfoShadow_2(europe-west2)
Step #6: i  functions: cleaning up build files...
Step #6: ⚠  functions: Unhandled error cleaning up build images. This could result in a small monthly bill if not corrected. You can attempt to delete these images by redeploying or you can delete them manually at https://console.cloud.google.com/artifacts/docker/groundlevel-jun-22/europe-west2/gcf-artifacts
Step #6: 
Step #6: Error: There was an error deploying functions
Finished Step #6
ERROR
ERROR: build step 6 "us-central1-docker.pkg.dev/ci-builder/builders/firebase-emulators:11.3.0" failed: step exited with non-zero status: 2

The way I read that is that firebase-tools first checks that there are enough access rights:

Step #6: i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
Step #6: ✔  artifactregistry: required API artifactregistry.googleapis.com is enabled

..but it then tries to do something that's beyond those rights (delete the earlier cached images).

Screenshot of the cached images

Attempting work-around

I removed the cached images manually, and retried the deployment.

It still fails.

I removed the user_info_shadow2__ folder, leaving an empty gcf-artifacts. Retried.

It still fails in the same way.

Since there were some artifacts there from earlier runs (7 and 12 days ago), either this is something in 11.3.0 (I upgraded in the mean time), or I may have inadvertently changed some access rights.

More info

Manual deployment works, with firebase-tools 11.3.0. That is, my access rights (as the owner of the GCP project) are enough.

This also used to work, like months back, but I was using Container Registry at the time. I have not done CI deployments since moving to Artifact Registry.

---

Screenshot of the gcd-artifacts folder

Note that the deployment region is europe-west2, a bit unusual.

[akauppi]

Corrections:

It doesn't have to do with firebase-tools version.

I needed to enable Service Account User:

Since this is not mentioned in the Firebase docs (I believe), I'll leave this issue open. Let me know if you wish me to edit it.

Note: The inability to clean up was separate from the actual error that fails the deployment. Can do two tickets if you think both of these are worth doing something about.

[khenneuse]

Thanks @akauppi I created a pull request to update the documentation in the community build repo

There should be an update to the firebase tool to print a warning about the service account not having access when that happens. It would be a clearer signal as to the issue.

The build did pass when run as a single step

  - name: gcr.io/<PROJECT_ID>/firebase
  args: ["deploy", "--project=<PROJECT_ID>"]

But failed when run as individual steps

  - name: gcr.io/<PROJECT_ID>/firebase
    args:
      ["deploy", "--project=<PROJECT_ID>", "--only=functions", "--force"]
  - name: gcr.io/<PROJECT_ID>/firebase
    args: ["deploy", "--project=<PROJECT_ID>", "--only=hosting"]
  - name: gcr.io/<PROJECT_ID>/firebase
    args: ["deploy", "--project=<PROJECT_ID>", "--only=firestore"]
  - name: gcr.io/<PROJECT_ID>/firebase
    args: ["deploy", "--project=<PROJECT_ID>", "--only=storage"]

This may be an issue inside of the package that it is not capture or checking the exit code of each step. Therefore only the exit code of the last step in the chain is reported to Cloud Build.

[zouhirdev]

@akauppi hello how can i fix this please im new to firebase im having the same error i cant find the service account permissions to edit them please help

[joehan]

Hey @zouhirdev - you should be able to see and edit these permissions here: https://console.cloud.google.com/cloud-build/settings/service-account

If you are unable to see that page, you might not have the necessary permissions on your project, and should ask a project owner to help you out.

[spkrak]

Hello, I've executed configuration steps mentioned by @joehan, yet I'm still receiving same warning after deployment (Unhandled error cleaning up build images. This could result in a small monthly bill if not corrected. You can attempt to delete these images by redeploying or you can delete them manually at...) and docker image is not deleted from artifact repository from the project so I need to remove it manually every time. Any suggestions on how to proceed further?

[myconode]

Hi all, I have followed the recommended steps above for both the default and "legacy" Cloud Build service accounts, but unfortunately am still getting the same warning. I also tried granting Artifact Registry Service Agent and Artifact Registry Repository Administrator roles to the service account but deploying cloud functions using the firebase cli still yields the same error.

Any recommendations are much appreciated, thanks.

[yve-alain]

Still getting this error despite enabling the services suggested as a solution. It deploys but the repository gcf-artifacts remains and requires manual deletion.

[ziadsarour]

I have this issue too. When using --debug during deploy, I have found this log :

// Failed docker command with error  HTTP Error: 404, Not Found 
{
  "name": "FirebaseError",
  "children": [],
  "context": {
    "body": {
      "errors": [
        {
          "code": "NAME_UNKNOWN",
          "message": "Repository \"eu.gcr.io\" not found"
        }
      ],
      "error": {
        "message": "Not Found"
      }
    },
    "response": {
      "statusCode": 404
    }
  },
  "exit": 1,
  "message": "HTTP Error: 404, Not Found",
  "status": 404
}

Disabling GCR API did not work for me : https://github.com/firebase/firebase-tools/issues/7431#issuecomment-2229263787

When I try to navigate to Container Registry (https://console.cloud.google.com/gcr) using the console, I'm instantly redirected to Artifact Registry (https://console.cloud.google.com/artifacts).

---

EDIT : I guess it has something to do with this : https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr

I still have this error even if my project has been migrated from Container Registry to Artifact Registry.

[ziadsarour]

Ok, I've found the solution for my problem. Google though my transition to Artifact was set correctly but it was not (may be my fault).

1. Navigate to Artifact Registry and check if you have these 4 repositories :

2. If you don't have these 4 repositories, create one of them. Enter only the name of the repository and click on the create button.

[!NOTE]
This step may not be required, but I want to provide you the exact steps I've followed

3. Run the command gcloud artifacts settings enable-upgrade-redirection --project=PROJECT. If you have missing repositories, it will ask you if you want to create them, confirm with y. Then it will ask you if you want to continue, confirm with y.

Here is the output of this command :

gcloud artifacts settings enable-upgrade-redirection --project=PROJECT
Performing redirection enablement checks...

Project PROJECT is already using Artifact Registry for all *gcr.io traffic.
Repository report for PROJECT:

Container Registry Host  Location  Artifact Registry Repository
gcr.io                   us        None Found. Can create repo named "gcr.io"
us.gcr.io                us        None Found. Can create repo named "us.gcr.io"
asia.gcr.io              asia      None Found. Can create repo named "asia.gcr.io"
eu.gcr.io                europe    projects/PROJECT/locations/europe/repositories/eu.gcr.io

gcloud can automatically create the 3 missing repos in Artifact Registry.
If you would like to setup CMEK for these repos, exit now and create them
manually instead.

Create missing repos (recommended) (y/N)?  y

Waiting for repo creation to complete......done.
Waiting for repo creation to complete......done.
Waiting for repo creation to complete......done.

This action will redirect all Container Registry traffic to Artifact
Registry for project PROJECT. After enabling redirection, you
can route traffic back to Container Registry if needed.

Do you want to continue (y/N)?  y

legacyRedirectionState: REDIRECTION_FROM_GCR_IO_ENABLED
name: projects/PROJECT/projectSettings

[ziadsarour]

Finally, it's not working. The warning is not printed anymore but I still have an error when using --debug

https://artifactregistry.googleapis.com/v1beta2/projects/PROJECT/locations/europe-west1/repositories/gcf-artifacts/packages/routines__v1__routines_share_post {"error":{"code":404,"message":"Package \"projects/PROJECT/locations/europe-west1/repositories/gcf-artifacts/packages/routines__v1__routines_share_post\" was not found.","status":"NOT_FOUND"}}
[2024-09-17T16:21:40.152Z] <<< [apiv2][status] GET https://eu.gcr.io/v2/PROJECT/gcf/europe-west1/tags/list 200
[2024-09-17T16:21:40.152Z] <<< [apiv2][body] GET https://eu.gcr.io/v2/PROJECT/gcf/europe-west1/tags/list {"child":[],"manifest":{},"name":"PROJECT/gcf/europe-west1","tags":[]}
[2024-09-17T16:21:40.152Z] Could not find image for function projects/PROJECT/locations/europe-west1/functions/routines_v1_newsletterSubscribePost