GoogleCloudPlatform / DataflowTemplates

Cloud Dataflow Google-provided templates for solving in-Cloud data tasks
https://cloud.google.com/dataflow/docs/guides/templates/provided-templates
Apache License 2.0
1.14k stars 950 forks source link

[Bug]: Oracle to BigQuery - "bigQuerySchemaPath" parameter issue #1438

Open fabrizio-rizzi opened 4 months ago

fabrizio-rizzi commented 4 months ago

Related Template(s)

oracle_to_bigquery

Template Version

2024-03-06-00_rc00

What happened?

I'm trying to use the "bigQuerySchemaPath" template parameter in order to create the BQ tables if needed using the json schema files stored on GCS.

This parameter is causing the job failure because I get the following error message:

com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed. java.lang.IllegalArgumentException: No filesystem found for scheme gs

If I manually creates the BQ tables and then I run the job with this parameter commented, it runs successfully.

Relevant log output

{
  "insertId": "4558186607511348110:652970:0:10150",
  "jsonPayload": {
    "line": "exec.go:66",
    "message": "com.google.cloud.teleport.v2.common.UncaughtExceptionLogger - The template launch failed.\njava.lang.IllegalArgumentException: No filesystem found for scheme gs\n\tat org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:520)\n\tat org.apache.beam.sdk.io.FileSystems.match(FileSystems.java:126)\n\tat org.apache.beam.sdk.io.FileSystems.match(FileSystems.java:147)\n\tat com.google.cloud.teleport.v2.utils.GCSUtils.getGcsFileByteChannel(GCSUtils.java:126)\n\tat com.google.cloud.teleport.v2.utils.GCSUtils.getGcsFileAsString(GCSUtils.java:98)\n\tat com.google.cloud.teleport.v2.templates.JdbcToBigQuery.writeToBQTransform(JdbcToBigQuery.java:192)\n\tat com.google.cloud.teleport.v2.templates.JdbcToBigQuery.main(JdbcToBigQuery.java:84)\n"
  },
  "resource": {
    "type": "dataflow_step",
    "labels": {
      "job_id": "2024-03-28_09_40_00-404592659939574189",
      "region": "europe-west8",
      "project_id": "",
      "step_id": "",
      "job_name": "wf-oracle-to-bigquery-consenso-fse"
    }
  },
  "timestamp": "2024-03-28T16:40:53.546325Z",
  "severity": "ERROR",
  "labels": {
    "compute.googleapis.com/resource_name": "launcher-20240328094000404592659939574189",
    "compute.googleapis.com/resource_type": "instance",
    "dataflow.googleapis.com/region": "europe-west8",
    "dataflow.googleapis.com/job_id": "2024-03-28_09_40_00-404592659939574189",
    "compute.googleapis.com/resource_id": "4558186607511348110",
    "dataflow.googleapis.com/job_name": "wf-oracle-to-bigquery-consenso-fse"
  },
  "logName": "",
  "receiveTimestamp": "2024-03-28T16:40:54.134868383Z",
  "errorGroups": [
    {
      "id": "CL6l4pGGtKuUoQE"
    }
  ]
}
stankiewicz commented 4 months ago

this is resolved with #1452