search

trinodb / trino

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
https://trino.io
Apache License 2.0
9.92k stars 2.86k forks source link

BigQuery table listing not robust against schema drops #8795

Open electrum opened 2 years ago

electrum commented 2 years ago

In the this test failure, the method com.google.cloud.bigquery.BigQueryImpl.listTables() is failing due to the schema test_drop_case_sensitive getting dropped concurrently while the test is running. It is called in a loop over the schema names via BigQueryMetadata.listTables().

When we get a "not found" exception from listTables(), we can fetch the list of schema names again. If the schema still exists, something weird happened, so throw the error. Otherwise, there are two options:

  1. Retry the entire list tables operation with the fresh list of schema names.
  2. Skip the schema that no longer exists. This is the simplest and most efficient, though slightly less consistent, but we're not consistent anyway since tables are listed individually across schemas.
Error:  testSelectInformationSchemaColumns(io.trino.plugin.bigquery.TestBigQueryIntegrationSmokeTest)  Time elapsed: 8.403 s  <<< FAILURE!
java.lang.AssertionError: Expected query to succeed: SELECT * FROM information_schema.columns
    at org.testng.Assert.fail(Assert.java:83)
    at io.trino.testing.QueryAssertions.assertQuerySucceeds(QueryAssertions.java:278)
    at io.trino.testing.AbstractTestQueryFramework.assertQuerySucceeds(AbstractTestQueryFramework.java:253)
    at io.trino.testing.AbstractTestQueryFramework.assertQuerySucceeds(AbstractTestQueryFramework.java:248)
    at io.trino.testing.AbstractTestIntegrationSmokeTest.testSelectInformationSchemaColumns(AbstractTestIntegrationSmokeTest.java:428)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:104)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:645)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:851)
    at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1177)
    at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:129)
    at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:112)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.RuntimeException: Not found: Dataset development-226103:test_drop_case_sensitive
    at io.trino.testing.AbstractTestingTrinoClient.execute(AbstractTestingTrinoClient.java:120)
    at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:476)
    at io.trino.testing.QueryAssertions.assertQuerySucceeds(QueryAssertions.java:275)
    ... 16 more
    Suppressed: java.lang.Exception: SQL: SELECT * FROM information_schema.columns
        at io.trino.testing.DistributedQueryRunner.execute(DistributedQueryRunner.java:479)
        ... 17 more
Caused by: com.google.cloud.bigquery.BigQueryException: Not found: Dataset development-226103:test_drop_case_sensitive
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.listTables(HttpBigQueryRpc.java:326)
    at com.google.cloud.bigquery.BigQueryImpl$20.call(BigQueryImpl.java:886)
    at com.google.cloud.bigquery.BigQueryImpl$20.call(BigQueryImpl.java:880)
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
    at com.google.cloud.bigquery.BigQueryImpl.listTables(BigQueryImpl.java:878)
    at com.google.cloud.bigquery.BigQueryImpl.listTables(BigQueryImpl.java:821)
    at io.trino.plugin.bigquery.BigQueryClient.listTables(BigQueryClient.java:218)
    at io.trino.plugin.bigquery.BigQueryMetadata.listTables(BigQueryMetadata.java:151)
    at io.trino.plugin.bigquery.BigQueryMetadata.lambda$listTableColumns$13(BigQueryMetadata.java:274)
    at java.base/java.util.Optional.orElseGet(Optional.java:369)
    at io.trino.plugin.bigquery.BigQueryMetadata.listTableColumns(BigQueryMetadata.java:274)
    at io.trino.spi.connector.ConnectorMetadata.streamTableColumns(ConnectorMetadata.java:254)
    at io.trino.metadata.MetadataManager.listTableColumns(MetadataManager.java:671)
    at io.trino.metadata.MetadataListing.listTableColumns(MetadataListing.java:159)
    at io.trino.connector.informationschema.InformationSchemaPageSource.addColumnsRecords(InformationSchemaPageSource.java:251)
    at io.trino.connector.informationschema.InformationSchemaPageSource.buildPages(InformationSchemaPageSource.java:216)
    at io.trino.connector.informationschema.InformationSchemaPageSource.getNextPage(InformationSchemaPageSource.java:183)
    at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:311)
    at io.trino.operator.Driver.processInternal(Driver.java:387)
    at io.trino.operator.Driver.lambda$processFor$9(Driver.java:291)
    at io.trino.operator.Driver.tryWithLock(Driver.java:683)
    at io.trino.operator.Driver.processFor(Driver.java:284)
    at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1076)
    at io.trino.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
    at io.trino.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
    at io.trino.$gen.Trino_testversion____20210804_194153_3.run(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 404 Not Found
GET https://www.googleapis.com/bigquery/v2/projects/development-226103/datasets/test_drop_case_sensitive/tables?prettyPrint=false
{
  "code" : 404,
  "errors" : [ {
    "domain" : "global",
    "message" : "Not found: Dataset development-226103:test_drop_case_sensitive",
    "reason" : "notFound"
  } ],
  "message" : "Not found: Dataset development-226103:test_drop_case_sensitive",
  "status" : "NOT_FOUND"
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:118)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:37)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:428)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1108)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:514)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:455)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565)
    at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.listTables(HttpBigQueryRpc.java:305)
    ... 30 more
hashhar commented 2 years ago

I think the cause is different. The tests in TestBigQueryIntegrationSmokeTest don't use case-insensitive-name-mapping while the schema created in TestBigQueryCaseInsensitiveMapping with a non-lowercase name so the QueryRunner in TestBigQueryIntegrationSmokeTest cannot "find" the schema since it sends the request to BigQuery using the lowercased name when the actual name is Test_Drop_Case_Sensitive.

The solution is to ensure isolation of TestBigQueryIntegrationSmokeTest and TestBigQueryCaseInsensitiveMapping which we do using the surefire setup in trino-bigquery.

Same problem manifests in base-jdbc based connectors if you start re-using the containers since the non-lowercase schema names will then cause query failures for any query runner not configured with case-insensitive-name-mapping. But we don't have that problem since most connector tests are containerised and each test spins up it's own container.