gsutil -m rm on empty bucket fails with CommandException

[msigdel]

Running the following command fails with "CommandException: 1 files/objects could not be removed."

<gsutil -m rm gs://mybucket/empty_bucket/*>

The error isn't meaningful as in fact the issue is that there are no files, instead it says 'objects could not be removed'. Also, should the bucket not having files raise an exception?

[houglum]

Ah, interesting. Our exception handler for rm doesn't treat "No URLs matched: " exceptions any differently from other exceptions; it assumes all captured exceptions were failures from trying to remove valid files/objects.

I agree that the error message here is misleading. And as for the exception: while nothing technically went wrong in deleting all the objects/files corresponding to the command arguments (even if this evaluates to 0), we run under the assumption that if a user says, "I want to delete gs://foo/bar," then they believe that object exists - this is just a nicety to inform them that the objects they assumed were present were, in fact, not.

[LivesleyWA]

Disagree - I might not know the state of a directory that I am clearing. I have an automated build process and the target directory may or may not have something in it depending on the previous build state. In any case, the -f flag should ensure that the "error" is not reported. It is not working and I am still getting the error which kills my build process.

[andrelocher]

I'm encountering the same issue here, what's the workaround?

[houglum]

If you'd like to know whether the bucket is empty first, you could make a call to ls. The ls command doesn't currently provide a way to set max desired results, so if there will potentially be a lot of objects in the bucket, resulting in lots of listing calls, you could instead make a raw call to the JSON API's objects.list method. This would allow you to specify a value of 1 for the maxResults parameter, resulting in only one quick listing call. If you prefer the XML API, it offers this functionality via the max-keys option.

[rama-rajkumar]

I have similar issue. I wish this could be handled in a better way. We have automated build process where we clear the directory without knowing if it's empty or not. The error thrown by gsutil while trying to delete empty directory breaks the build process.

[houglum]

In the spirit of tracking how much user pain this causes, I saw another report of this behavior being confusing at https://stackoverflow.com/questions/47885197/gsutil-cp-command-error-commandexception-no-urls-matched/47895858#47895858

[nojvek]

gsutil -m cp -n -r ../assets/* gs://foo/assets
CommandException: No URLs matched: ../asset-cache/*

This is like cp -r foo bar throwing an error if foo directory is empty.

Would love to see this edge case fixed

[hobayoba]

noticied strange behavior of gsutil -m {mv, cp, rm} gs://some-bucket/dir1 gs://some-bucket/dir2 sometimes it still returns the error: "CommandException: count_of_files files/objects could not be copied\moved\removed" when I ran processing a lot of files. Usually, count_of_files is a small count, the max that I noticed is 7.

The first that I thought is network troubles, but when I started to ping-tests and moving files as it shows the above, I just got 0% of the packet loss and the error with 2 not moved files =(

it's critical moment cause I using buckets as part of the CDN network and have a lot of builds of static content during the day, so, every error just ruined build processes on my test environment.

is there another way to fast managing files remotely in GC buckets instead of gsutil?

OS: Deb 9 google-cloud-sdk version: 232.0.0-0

$ cat .boto | grep -v "^#" | sed '/^$/d' [Credentials] [Boto] https_validate_certificates = True [GoogleCompute] [GSUtil] content_language = en default_api_version = 2 [OAuth2]

gsutil version: 4.35 checksum: 3b22dd7820bcb962909f4f401010fb17 (OK) boto version: 2.49.0 python version: 2.7.15+ (default, Nov 28 2018, 16:27:22) [GCC 8.2.0] OS: Linux 4.9.0-6-amd64 multiprocessing available: True using cloud sdk: True pass cloud sdk credentials to gsutil: True config path(s): ~/.boto, gsutil path: /usr/lib/google-cloud-sdk/bin/gsutil compiled crcmod: True installed via package manager: False editable install: False

[vinaygopalkrishnan]

Hi

We are using gsutil version: 4.38, and seem to be facing the same issue:

The command: gsutil rm -f is throwing the error CommandException: 1 files/objects could not be removed.

Is this issue fixed or is there a work around

[NandoSangenetto]

Is there a plan to make it work? How can we help?

[orn688]

It would be great to at least document this behavior. There's no indication in the cp documentation that running gsutil cp -r with an empty directory as the source will fail.

[abdurrahman]

In my case, I am using Airflow and I could not realize yet when i get this CommandException, because sometimes its working, sometimes it occurring error. This could be a simple solution; gsutil -m rm -rf gs://bucket/sub_directory/ || true

[dozer-jang]

In my case, In the GCE instance to which the default service account is attached When gsutil -m option was used, there was no authority for GCS(default service account only has read only permission), but an error message was simply displayed as a CommandException. ( as CommandException: n files/objects could not be removed.)

Anyone who command in GCE, it would be good to try without -m option and check if 403 Access Denied appears.

[RezaRahmati]

This thread is started at 2017 and still issue

[vbeffa]

This might be a workaround:

touch foo && gsutil cp foo gs://${BUCKET_ID} && gsutil -m rm gs://${BUCKET_ID}/** && rm -f foo

[jeffistyping]

any update on this?

[stevenferrer]

Thanks @vbeffa, your approach works for me! To rephrase, fill the empty bucket with a dummy file and now the delete operation should work as the bucket now contains at least one object.

• Fill the empty bucket with a dummy file.

$ gsutil cp dummy.txt gs://${BUCKET}

• Perform the delete operation.

$ gsutil -m rm gs://${BUCKET}/**

[atrauzzi]

This shouldn't be failing like this. Just remove the non-zero exit status.

[atikraju]

Hi, we want delete file in google bucket without last 30 days, we are tired to make script. any one help me....Thanks