search

kestra-io / plugin-gcp

Apache License 2.0
9 stars 10 forks source link

GCS Compose fails with an 404 error #319

Open shrutimantri opened 9 months ago

shrutimantri commented 9 months ago

Expected Behavior

Given a GCS directory path containing few files, the files should concatenate into one file, and the concatenated file should be uploaded to the destination.

Actual Behaviour

The flow execution results into an error:

2024-01-29 15:00:07.828 • Using service account: kestra-gcs-svc-account@shruti-demo-project.iam.gserviceaccount.com
2024-01-29 15:00:07.836 • Using service account: kestra-gcs-svc-account@shruti-demo-project.iam.gserviceaccount.com
2024-01-29 15:00:10.163Found '4' blobs from 'gs://smantri-gcs-kestra-test-bucket/test/users/'
2024-01-29 15:00:10.167Compose from '[com.google.cloud.storage.Storage$ComposeRequest$SourceBlob@3221112c, com.google.cloud.storage.Storage$ComposeRequest$SourceBlob@1afa53f0, com.google.cloud.storage.Storage$ComposeRequest$SourceBlob@672893f7, com.google.cloud.storage.Storage$ComposeRequest$SourceBlob@1fb84247]' to 'gs://smantri-kestra-bucket/test/users.csv'
2024-01-29 15:00:10.597Object test/users/ (generation: 0) not found.
2024-01-29 15:00:10.597404 Not Found
POST https://storage.googleapis.com/storage/v1/b/smantri-kestra-bucket/o/test%2Fusers.csv/compose
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Object test/users/ (generation: 0) not found.",
    "reason" : "notFound"
  } ],
  "message" : "Object test/users/ (generation: 0) not found."
}
2024-01-29 15:00:10.597com.google.cloud.storage.StorageException: Object test/users/ (generation: 0) not found.
    at com.google.cloud.storage.StorageException.translate(StorageException.java:165)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:313)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.compose(HttpStorageRpc.java:716)
    at com.google.cloud.storage.StorageImpl.lambda$compose$16(StorageImpl.java:562)
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
    at com.google.cloud.storage.Retrying.run(Retrying.java:65)
    at com.google.cloud.storage.StorageImpl.run(StorageImpl.java:1515)
    at com.google.cloud.storage.StorageImpl.compose(StorageImpl.java:560)
    at io.kestra.plugin.gcp.gcs.Compose.run(Compose.java:102)
    at io.kestra.plugin.gcp.gcs.Compose.run(Compose.java:21)
    at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:684)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
POST https://storage.googleapis.com/storage/v1/b/smantri-kestra-bucket/o/test%2Fusers.csv/compose
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Object test/users/ (generation: 0) not found.",
    "reason" : "notFound"
  } ],
  "message" : "Object test/users/ (generation: 0) not found."
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$3.interceptResponse(AbstractGoogleClientRequest.java:466)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1111)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:552)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:493)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:603)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.compose(HttpStorageRpc.java:713)
    ... 10 more

Steps To Reproduce

  1. Create a bucket, and in a particular folder in that bucket, create a few files that need to be concatenated.
  2. Create a new bucket where the concatenated file will be uploaded.
  3. Create a flow with the task as shown in the example on this page.
  4. Execute the flow.

Environment Information

Example flow

id: gcp-gcs-compose
namespace: company.team
tasks:
  - id: "compose"
    type: "io.kestra.plugin.gcp.gcs.Compose"
    serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
    projectId: "<project_name>"
    list: 
      from: "gs://<bucket1>/test/users/"
    to: "gs://<bucket2>/test/users.csv"

This location: gs://<bucket1>/test/users/ contains 3 files: users1.csv, users2.csv, users3.csv