search

apache / polaris

Apache Polaris, the interoperable, open source catalog for Apache Iceberg
https://polaris.apache.org/
Apache License 2.0
1.13k stars 124 forks source link

[BUG] The existing local derby directory fails the catalog creation #217

Closed flyrain closed 2 months ago

flyrain commented 2 months ago

Is this a possible security vulnerability?

Describe the bug

Here are stack traces. The issue is resolved by deleting the derby dir. We may clean up it very time we run the app.

Caused by: java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
    at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.createDatabase(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
    at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
    at org.apache.derby.jdbc.InternalDriver$1.run(Unknown Source)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
    at org.apache.derby.jdbc.InternalDriver.getNewEmbedConnection(Unknown Source)
    at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
    at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
    at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source)
    at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:683)
    at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:191)
    at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
    at com.jolbox.bonecp.BoneCP.<init>(BoneCP.java:416)
    ... 92 more
Caused by: ERROR XJ041: Failed to create database 'metastore_db', see the next exception for details.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown Source)
    ... 108 more
Caused by: ERROR XBM0A: The database directory '/private/tmp/derby/metastore_db' exists. However, it does not contain the expected 'service.properties' file. Perhaps Derby was brought down in the middle of creating this database. You may want to delete this directory and try creating the database again.
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
    at org.apache.derby.impl.services.monitor.StorageFactoryService.vetService(Unknown Source)
    at org.apache.derby.impl.services.monitor.StorageFactoryService.access$900(Unknown Source)
    at org.apache.derby.impl.services.monitor.StorageFactoryService$10.run(Unknown Source)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
    at org.apache.derby.impl.services.monitor.StorageFactoryService.createServiceRoot(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
    at org.apache.derby.impl.services.monitor.BaseMonitor.createPersistentService(Unknown Source)
    at org.apache.derby.impl.services.monitor.FileMonitor.createPersistentService(Unknown Source)
    at org.apache.derby.iapi.services.monitor.Monitor.createPersistentService(Unknown Source)
    at org.apache.derby.impl.jdbc.EmbedConnection$5.run(Unknown Source)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:571)
    at org.apache.derby.impl.jdbc.EmbedConnection.createPersistentService(Unknown Source)
    ... 105 more
MonkeyCanCode commented 2 months ago

@flyrain here is a sample fix: https://github.com/apache/polaris/pull/228

So derby is used by spark by default for metastore management. I am assuming u ran the setup.sh under regtests dir and aborted it in between thus caused an incomplete derby dir to get created. Thus, the following runs with spark will all fail until a manual cleanup is being performed. This PR will cleanup derby home dir if existed.