GoogleCloudDataproc / flink-bigquery-connector

BigQuery integration to Apache Flink's Table API
Apache License 2.0
23 stars 14 forks source link

IllegalArgumentException: Project ID required for service #153

Open HitenUnify opened 2 months ago

HitenUnify commented 2 months ago
Caused by: java.lang.IllegalArgumentException: A project ID is required for this service but could not be determined from the builder or the environment.  Please set a project ID using the builder.
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:143)
    at com.google.cloud.ServiceOptions.<init>(ServiceOptions.java:340)
    at com.google.cloud.bigquery.BigQueryOptions.<init>(BigQueryOptions.java:94)
    at com.google.cloud.bigquery.BigQueryOptions.<init>(BigQueryOptions.java:31)
    at com.google.cloud.bigquery.BigQueryOptions$Builder.build(BigQueryOptions.java:89)
    at com.google.cloud.flink.bigquery.services.BigQueryServicesImpl$QueryDataClientImpl.<init>(BigQueryServicesImpl.java:248)
    at com.google.cloud.flink.bigquery.services.BigQueryServicesImpl.createQueryDataClient(BigQueryServicesImpl.java:89)
    at com.google.cloud.flink.bigquery.services.BigQueryServicesFactory.queryClient(BigQueryServicesFactory.java:94)
    at com.google.cloud.flink.bigquery.sink.serializer.BigQuerySchemaProviderImpl.getTableSchemaFromOptions(BigQuerySchemaProviderImpl.java:153)
    at com.google.cloud.flink.bigquery.sink.serializer.BigQuerySchemaProviderImpl.<init>(BigQuerySchemaProviderImpl.java:60)

happens even though it is clearly defined here

CredentialsOptions credentials = getCredentialsOptions(destination);
BigQueryConnectOptions sinkConnectOptions =
                BigQueryConnectOptions.builder().setProjectId(projectId)
                    .setDataset(dataset).setTable(destinationTableId).setCredentialsOptions(credentials).build();
BigQuerySchemaProvider schemaProvider = new BigQuerySchemaProviderImpl(sinkConnectOptions);

Core Issue seems to be at

com.google.cloud.flink.bigquery.services.BigQueryServicesImpl.QueryDataClientImpl#QueryDataClientImpl

public QueryDataClientImpl(CredentialsOptions options) {
            bigQuery =
                    BigQueryOptions.newBuilder()
                            .setCredentials(options.getCredentials())
                            .build()
                            .getService();
            bigquery = BigQueryUtils.newBigqueryBuilder(options).build();
        }

The builder

BigQueryOptions.newBuilder()
                            .setCredentials(options.getCredentials())
                            .build()
                            .getService();

doesn't have projectID context, unless passed through ENV, which I can't do in my flink project.

Thanks!

jayehwhyehentee commented 1 month ago

@HitenUnify Can you share the entire application code that you are trying to run here?