SQLancer Logs for Postgres and Terrier

[jazzplato]

I tried running the sqlancer on the postgres and the terrier and here are the logs generated.

According to the sqlancer readme, the database<xx>-cur.log files are the sql queries used on each of the databases created during the tests and the database<xx>.log files are the execution logs of the sqlancer process should there be any logic bugs. It is notable that such logic bugs in the .log files are expressed in the java exceptions, which may not directly link back to the potential bugs in the terrier code base in c++. For example, here is a sample log for the terrier.

--org.postgresql.util.PSQLException: The connection attempt failed.
--  at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:297)
--  at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
--  at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:217)
--  at org.postgresql.Driver.makeConnection(Driver.java:458)
--  at org.postgresql.Driver.connect(Driver.java:260)
--  at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
--  at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
--  at sqlancer.postgres.PostgresProvider.createDatabase(PostgresProvider.java:274)
--  at sqlancer.postgres.PostgresProvider.createDatabase(PostgresProvider.java:1)
--  at sqlancer.Main$DBMSExecutor.run(Main.java:312)
--  at sqlancer.Main$1.runThread(Main.java:421)
--  at sqlancer.Main$1.run(Main.java:414)
--  at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
--  at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
--  at java.base/java.lang.Thread.run(Thread.java:834)
--Caused by: java.io.EOFException
--  at org.postgresql.core.PGStream.receiveChar(PGStream.java:372)
--  at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:510)
--  at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:146)
--  at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:197)
--  ... 14 more
---- Time: 2020/07/08 07:10:56
-- Database: database0
-- Database version: null
-- seed value: 1594192255931

Since I remember that @apavlo mentioned that someone in the CMU DB Group had started working on building the interface layer of the terrier for the sqlancer. It can be seen that if we are going to write our own interface layer of the terrier for the sqlancer, we can customize the java exception to be more informative.

[jazzplato]

the sqlancer logs for the duckdb is now available at here. It worked remarkably well with sqlancer such that no logic error ever thrown!

[mrigger]

Hi @jazzplato, I just noticed this and thought that my input might help. Feel free to open GitHub issues in the SQLancer repository, or ask in our Slack workspace if anything does not work as you would expect.

In principle, a Java exception should always correspond to a bug in the DBMS. Judging from the output above, and from the terrier log files, an exception is thrown since SQLancer fails to connect to the DBMS. A meaningful error message for this scenario is on our TODO list (see https://github.com/sqlancer/sqlancer/issues/92).

I had a look at the PostgreSQL log output. SQLancer generates queries and statements assuming a newer PostgreSQL version (version 12), which explains the exceptions. We actually test that SQLancer does not throw any exceptions for PostgreSQL 12 (see https://github.com/sqlancer/sqlancer/blob/86647df8aa2dd8d167b5c3ce3297290f5b0b2bcd/.travis.yml#L79).