search

kestra-io / plugin-gcp

Apache License 2.0
9 stars 10 forks source link

GCS DeleteBucket expects the bucket to be empty to be deleted #321

Open shrutimantri opened 9 months ago

shrutimantri commented 9 months ago

Expected Behavior

The given bucket should be deleted. We can provide an option to delete non-empty bucket, or explicitly mention that only empty buckets will be deleted.

Actual Behaviour

The non-empty bucket does not get deleted. The task fails with the following error:

2024-01-29 16:09:11.283Deleting bucket 'smantri-gcs-kestra-test-bucket'
2024-01-29 16:09:12.910The bucket you tried to delete is not empty.
2024-01-29 16:09:12.910409 Conflict
DELETE https://storage.googleapis.com/storage/v1/b/smantri-gcs-kestra-test-bucket
{
  "code" : 409,
  "errors" : [ {
    "domain" : "global",
    "message" : "The bucket you tried to delete is not empty.",
    "reason" : "conflict"
  } ],
  "message" : "The bucket you tried to delete is not empty."
}
2024-01-29 16:09:12.910com.google.cloud.storage.StorageException: The bucket you tried to delete is not empty.
    at com.google.cloud.storage.StorageException.translate(StorageException.java:165)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:313)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.delete(HttpStorageRpc.java:642)
    at com.google.cloud.storage.StorageImpl.lambda$delete$14(StorageImpl.java:516)
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
    at com.google.cloud.storage.Retrying.run(Retrying.java:65)
    at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1515)
    at com.google.cloud.storage.StorageImpl.delete(StorageImpl.java:516)
    at io.kestra.plugin.gcp.gcs.DeleteBucket.run(DeleteBucket.java:54)
    at io.kestra.plugin.gcp.gcs.DeleteBucket.run(DeleteBucket.java:19)
    at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:684)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 409 Conflict
DELETE https://storage.googleapis.com/storage/v1/b/smantri-gcs-kestra-test-bucket
{
  "code" : 409,
  "errors" : [ {
    "domain" : "global",
    "message" : "The bucket you tried to delete is not empty.",
    "reason" : "conflict"
  } ],
  "message" : "The bucket you tried to delete is not empty."
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.delete(HttpStorageRpc.java:638)
    ... 10 more

Steps To Reproduce

in the DeleteBucket task, provide a non0-empty bucket to be deleted.

Environment Information

Example flow

id: gcp-gcs-delete-bucket
namespace: company.team
tasks:
  - id: "delete"
    type: "io.kestra.plugin.gcp.gcs.DeleteBucket"
    serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
    projectId: "<project-name>"
    name: "<bucket-name>"
anna-geller commented 9 months ago

this seems expected, I know it from S3 that this is the default behavior

initially this seems safer than deleting a non empty bucket

we could introduce an option to force delete in the style of gsutil -m rm -r gs://my-bucket