search

google / zetasql

ZetaSQL - Analyzer Framework for SQL
Apache License 2.0
2.32k stars 219 forks source link

RESOURCE_EXHAUSTED: Received message larger than max (4224741 vs. 4194304) #143

Open Longxiang-Bai opened 1 year ago

Longxiang-Bai commented 1 year ago

Hi, we are using zetasql 2023.04.1 and we are facing the following error: com.google.zetasql.io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Received message larger than max (4224741 vs. 4194304) when processing a huge sql with Analyzer.extractTableNamesFromScript(). I am wondering if anyone could advise how can we make it work? Thanks!

matthewcbrown commented 1 year ago

Getting this error on extractTableNamesFromScript is very unexpected, since it doesn't actually serialize the AST. I would expect this on almost any other API call, are you sure it's that call that is failing?

I assume the raw sql itself doesn't acceed this size?

Longxiang-Bai commented 1 year ago

Hi @matthewcbrown, thanks for your reply. I am pretty sure it is this API. Since our sql has over 400k parameters lol. So it is possible it actually exceeds the size. Do you have any comment or suggestions?

kylase commented 9 months ago

I experienced this error as well. However, the circumstances which it arises is different.

We are only running a simple SELECT 1 statement with ZetaSQL Toolkit, but with minor difference with regards how the catalog is build with.

If we build the catalog with a project with a lot of tables (415 tables) (:

catalog.addAllTablesInProject("projectId")

(note that this is ZetaSQL Toolkit API and what it does is to populate the SimpleCatalog)

We will encounter similar error:

Exception in thread "main" com.google.zetasql.toolkit.AnalysisException: Received message larger than max (4266687 vs. 4194304)
    at com.google.zetasql.toolkit.ZetaSQLToolkitAnalyzer$StatementAnalyzer.analyzeNextStatement(ZetaSQLToolkitAnalyzer.java:232)
    at com.google.zetasql.toolkit.ZetaSQLToolkitAnalyzer$StatementAnalyzer.next(ZetaSQLToolkitAnalyzer.java:211)
    at com.google.zetasql.toolkit.ZetaSQLToolkitAnalyzer$StatementAnalyzer.next(ZetaSQLToolkitAnalyzer.java:148)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
    at com.google.zetasql.toolkit.examples.AnalyzeBigQuery.main(AnalyzeBigQuery.java:79)
Caused by: com.google.zetasql.SqlException: Received message larger than max (4266687 vs. 4194304)
    at com.google.zetasql.Analyzer.analyzeNextStatement(Analyzer.java:220)
    at com.google.zetasql.toolkit.ZetaSQLToolkitAnalyzer$StatementAnalyzer.analyzeNextStatement(ZetaSQLToolkitAnalyzer.java:229)
    ... 4 more
Caused by: com.google.zetasql.io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Received message larger than max (4266687 vs. 4194304)
    at com.google.zetasql.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
    at com.google.zetasql.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
    at com.google.zetasql.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
    at com.google.zetasql.ZetaSqlLocalServiceGrpc$ZetaSqlLocalServiceBlockingStub.analyze(ZetaSqlLocalServiceGrpc.java:1506)
    at com.google.zetasql.Analyzer.analyzeNextStatement(Analyzer.java:218)
    ... 5 more

If the catalog has small number of tables (45 in our case), this error will not occur.