goccy / bigquery-emulator

BigQuery emulator server implemented in Go
MIT License
831 stars 109 forks source link

How to run a load job in bq emulator with auth service account json? #347

Open Bitsy-Chuck opened 3 months ago

Bitsy-Chuck commented 3 months ago

I am trying to load a file in bigquery emulator using a load job

 LoadJobConfiguration loadConfig =
                    LoadJobConfiguration.newBuilder(tableIdObj, csvFilePath)
                            .setFormatOptions(FormatOptions.csv())
                            .setCreateDisposition(JobInfo.CreateDisposition.CREATE_IF_NEEDED)
                            .setWriteDisposition(JobInfo.WriteDisposition.WRITE_APPEND)
                            .setAutodetect(true)  // Automatically detect schema
                            .build();

bigquery.create(JobInfo.of(loadConfig));

fails with

com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request
POST http://localhost:57058/bigquery/v2/projects/test-project/jobs?prettyPrint=false
{
  "code": 400,
  "errors": [
    {
      "location": "",
      "message": "failed to import from gcs: dialing: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information",
      "reason": "jobInternalError",
      "debugInfo": ""
    }
  ],
  "message": "failed to import from gcs: dialing: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information"
}

But when running normal queries like bigquery.query(conf) works with auth. How to pass the auth? bigquery client is init with NoCredentials()

Bitsy-Chuck commented 3 months ago
public BigQuery getBqService(BigQueryEmulatorContainer bigQueryEmulatorContainer) {
        String url = bigQueryEmulatorContainer.getEmulatorHttpEndpoint();
        BigQueryOptions options = BigQueryOptions
                .newBuilder()
                .setProjectId(bigQueryEmulatorContainer.getProjectId())
                .setHost(url)
                .setLocation(url)
                .setCredentials(NoCredentials.getInstance())
                .build();
        return options.getService();
    }