eddumelendez
commented
1 year ago JFTR, it starts with version 374
Open hgschmie opened 1 year ago
eddumelendez
commented
1 year ago JFTR, it starts with version 374
Gingernaut
commented
1 year ago +1 on this issue
eddumelendez
commented
1 year ago At the time, I posted the previous comment tags from 374 to 405 were not working. However, taking a look at this again today seems that they are working fine 🤔. Logs doesn't contain the digest and during my tests didn't store them neither so not sure what happened here.
@hgschmie can you confirm this in your side, please? I've run 20 repeated tests and all green
hgschmie
commented
1 year ago will try.
hgschmie
commented
1 year ago still breaks for me:
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: 2023-01-11T19:07:06.962Z INFO main io.trino.server.Server ======== SERVER STARTED ========
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: 2023-01-11T19:07:07.995Z ERROR query-execution-6 io.trino.execution.scheduler.PipelinedQueryScheduler Failure in distributed stage for query 20230111_190706_00058_7skfx
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: io.trino.spi.TrinoException: No nodes available to run query
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at io.trino.execution.scheduler.UniformNodeSelector.computeAssignments(UniformNodeSelector.java:216)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at io.trino.execution.scheduler.DynamicSplitPlacementPolicy.computeAssignments(DynamicSplitPlacementPolicy.java:41)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at io.trino.execution.scheduler.SourcePartitionedScheduler.schedule(SourcePartitionedScheduler.java:292)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at io.trino.execution.scheduler.SourcePartitionedScheduler$1.schedule(SourcePartitionedScheduler.java:172)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at io.trino.execution.scheduler.PipelinedQueryScheduler$DistributedStagesScheduler.schedule(PipelinedQueryScheduler.java:1235)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[docker-java-stream-1217532399] INFO org.jdbi.v3.testing.junit5.tc.TrinoJdbiTestContainersExtensionTest - STDERR: at java.base/java.lang.Thread.run(Thread.java:833)This seems to be an inherent trino problem. The server is ready but has not yet registered any processing nodes.
It is harder to reproduce in 405 (probably 10 runs until I got the error) but not gone.
szymonm
commented
1 year ago I'm hitting the same issue.
Created the test that fails pretty often on my machine (Intel Mac). Please, have a look here: https://github.com/testcontainers/testcontainers-java/pull/6563/files
I think I understand the issue. The query code we use for checking that container is ready is:
boolean testQuerySucceeded = statement.execute(this.getTestQueryString());
if (testQuerySucceeded) {
logger().info("Container is started (JDBC URL: {})", this.getJdbcUrl());
return;
}Unfortunately, Trino enters some state when it returns from the execute method but it fails with No nodes available to run query when trying to fetch the data from the returned ResultSet. Thus, our check succeeds but Trino is not ready to serve queries that return data. In some other tests I successfully create tables and insert data, but fai later on querying it.
@findepi I see you contributed some code to TrinoContainer and know Trino well. What are your thoughts?
Solutions that come to my mind include:
WDYT?
findepi
commented
1 year ago @findepi I see you contributed some code to TrinoContainer and know Trino well. What are your thoughts?
thanks @szymonm for the ping. yes, i think i contributed that class
the container defines the readiness check
public String getTestQueryString() {
return "SELECT count(*) FROM tpch.tiny.nation";i think that when I was working on the container, this was sufficient.
I think it's not very practical for Trino in a single-node setup to accept a query and fail with No nodes available to run query, so I would treat it as a Trino problem, that we may or may not choose to workaround within testcontainers.
@szymonm @hgschmie if you're able to reproduce the problem using trinodb/trino docker image directly (without testcontainers library), I would recommend filing the bug report with the Trino project.
Module
Trino
Testcontainers version
1.17.6
Using the latest Testcontainers version?
Yes
Host OS
MacOS
Host Arch
x86
Docker version
What happened?
Trino container starts up and reports as "ready". Running a query fails with "No nodes available to run query". Waiting a few seconds before running the query allows it to succeed.
-> The trino container reports "ready" before it is actually ready.
The log below shows that the query that testcontainers uses to check that the engine is ready actually fails with the same error (No nodes available to run query) but testcontainers still contains is ready to use
Relevant log output
Additional Information
starting the container suffices. Sometimes it does work out, so it is a timing issue.