Open ingomueller-net opened 3 years ago
Duplicate of #15382.
While #15501 may have fixed my issue (still not sure), I do think that the documentation has to be improved after all. Things that should be mentioned here, IMO:
enum-types-table-name
?I think I get further with version 0.246, but I am stuck again. Now, I get the following:
2021-01-28T20:44:42.130+0100 INFO main com.facebook.presto.server.PluginManager -- Loading plugin ...presto-server-0.246/data/plugin/function-namespace-managers --
2021-01-28T20:44:42.146+0100 INFO main com.facebook.presto.server.PluginManager Installing com.facebook.presto.functionNamespace.FunctionNamespaceManagerPlugin
2021-01-28T20:44:42.158+0100 INFO main com.facebook.presto.server.PluginManager Registering function namespace manager mysql 2021-01-28T20:44:42.158+0100 INFO main com.facebook.presto.server.PluginManager -- Finished loading plugin ...presto-server-0.246/data/plugin/function-namespace-managers --
...
2021-01-28T20:44:42.716+0100 INFO main com.facebook.presto.server.PluginManager -- Loading plugin ...presto-server-0.246/data/plugin/mysql --
2021-01-28T20:44:42.732+0100 INFO main com.facebook.presto.server.PluginManager Installing com.facebook.presto.plugin.mysql.MySqlPlugin
2021-01-28T20:44:42.742+0100 INFO main com.facebook.presto.server.PluginManager Registering connector mysql
2021-01-28T20:44:42.744+0100 INFO main com.facebook.presto.server.PluginManager -- Finished loading plugin ...presto-server-0.246/data/plugin/mysql --
...
2021-01-28T20:44:43.340+0100 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog properties etc/catalog/mysql.properties --
2021-01-28T20:44:43.341+0100 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Loading catalog mysql --
2021-01-28T20:44:43.765+0100 INFO main Bootstrap PROPERTY DEFAULT RUNTIME DESCRIPTION
2021-01-28T20:44:43.765+0100 INFO main Bootstrap case-insensitive-name-matching false false
2021-01-28T20:44:43.765+0100 INFO main Bootstrap case-insensitive-name-matching.cache-ttl 1.00m 1.00m
2021-01-28T20:44:43.765+0100 INFO main Bootstrap connection-password [REDACTED] [REDACTED]
2021-01-28T20:44:43.765+0100 INFO main Bootstrap connection-url ---- jdbc:mysql://hostname:3306
2021-01-28T20:44:43.765+0100 INFO main Bootstrap connection-user ---- root
2021-01-28T20:44:43.765+0100 INFO main Bootstrap allow-drop-table false false Allow connector to drop tables
2021-01-28T20:44:43.765+0100 INFO main Bootstrap mysql.auto-reconnect true true
2021-01-28T20:44:43.765+0100 INFO main Bootstrap mysql.connection-timeout 10.00s 10.00s
2021-01-28T20:44:43.765+0100 INFO main Bootstrap mysql.max-reconnects 3 3
2021-01-28T20:44:43.957+0100 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle starting...
2021-01-28T20:44:43.957+0100 INFO main com.facebook.airlift.bootstrap.LifeCycleManager Life cycle startup complete. System ready.
2021-01-28T20:44:43.959+0100 INFO main com.facebook.presto.metadata.StaticCatalogStore -- Added catalog mysql using connector mysql --
2021-01-28T20:44:43.973+0100 INFO main com.facebook.presto.metadata.StaticFunctionNamespaceStore -- Loading function namespace manager for catalog example --
2021-01-28T20:44:44.399+0100 INFO main Bootstrap PROPERTY DEFAULT RUNTIME DESCRIPTION
2021-01-28T20:44:44.399+0100 INFO main Bootstrap function-cache-expiration 5.00m 5.00m
2021-01-28T20:44:44.399+0100 INFO main Bootstrap function-instance-cache-expiration 8.00h 8.00h
2021-01-28T20:44:44.399+0100 INFO main Bootstrap supported-function-languages [sql] [sql]
2021-01-28T20:44:44.399+0100 INFO main Bootstrap type-cache-expiration 1.00h 1.00h
2021-01-28T20:44:44.399+0100 INFO main Bootstrap sql.function-implementation-type SQL SQL
2021-01-28T20:44:44.399+0100 INFO main Bootstrap database-url ---- jdbc:mysql://hostname:3306
2021-01-28T20:44:44.399+0100 INFO main Bootstrap enum-types-table-name enum_types enum_types
2021-01-28T20:44:44.399+0100 INFO main Bootstrap function-namespaces-table-name function_namespaces example_function_namespaces
2021-01-28T20:44:44.399+0100 INFO main Bootstrap functions-table-name sql_functions example_sql_functions
2021-01-28T20:44:44.890+0100 ERROR main com.facebook.presto.server.PrestoServer Unable to create injector, see the following errors:
1) Error notifying ProvisionListener com.facebook.airlift.bootstrap.LifeCycleModule$$Lambda$1903/1032026871 of com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManager.
Reason: com.facebook.airlift.bootstrap.LifeCycleStartException: Exception in PostConstruct method com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManager::initialize()
at com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManagerModule.configure(MySqlFunctionNamespaceManagerModule.java:43)
while locating com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManager
1 error
com.google.inject.CreationException: Unable to create injector, see the following errors:
1) Error notifying ProvisionListener com.facebook.airlift.bootstrap.LifeCycleModule$$Lambda$1903/1032026871 of com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManager.
Reason: com.facebook.airlift.bootstrap.LifeCycleStartException: Exception in PostConstruct method com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManager::initialize()
at com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManagerModule.configure(MySqlFunctionNamespaceManagerModule.java:43)
while locating com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManager
1 error
at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:186)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
at com.google.inject.Guice.createInjector(Guice.java:87)
at com.facebook.airlift.bootstrap.Bootstrap.initialize(Bootstrap.java:251)
at com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManagerFactory.create(MySqlFunctionNamespaceManagerFactory.java:59)
at com.facebook.presto.metadata.FunctionAndTypeManager.loadFunctionNamespaceManager(FunctionAndTypeManager.java:166)
at com.facebook.presto.metadata.StaticFunctionNamespaceStore.loadFunctionNamespaceManager(StaticFunctionNamespaceStore.java:81)
at com.facebook.presto.metadata.StaticFunctionNamespaceStore.loadFunctionNamespaceManagers(StaticFunctionNamespaceStore.java:63)
at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:152)
at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:79)
Caused by: com.facebook.airlift.bootstrap.LifeCycleStartException: Exception in PostConstruct method com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManager::initialize()
at com.facebook.airlift.bootstrap.LifeCycleManager.startInstance(LifeCycleManager.java:245)
at com.facebook.airlift.bootstrap.LifeCycleManager.addInstance(LifeCycleManager.java:211)
at com.facebook.airlift.bootstrap.LifeCycleModule.provision(LifeCycleModule.java:62)
at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:120)
at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:211)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:182)
... 9 more
Caused by: org.jdbi.v3.core.ConnectionException: java.sql.SQLException: No suitable driver found for jdbc:mysql://hostname:3306
at org.jdbi.v3.core.Jdbi.open(Jdbi.java:300)
at org.jdbi.v3.core.LazyHandleSupplier.initHandle(LazyHandleSupplier.java:58)
at org.jdbi.v3.core.LazyHandleSupplier.getHandle(LazyHandleSupplier.java:46)
at org.jdbi.v3.sqlobject.statement.internal.CustomizingStatementHandler.invoke(CustomizingStatementHandler.java:148)
at org.jdbi.v3.sqlobject.statement.internal.SqlUpdateHandler.invoke(SqlUpdateHandler.java:30)
at org.jdbi.v3.sqlobject.SqlObjectFactory.lambda$null$13(SqlObjectFactory.java:163)
at org.jdbi.v3.core.internal.JdbiThreadLocals.invokeInContext(JdbiThreadLocals.java:27)
at org.jdbi.v3.core.LazyHandleSupplier.lambda$invokeInContext$1(LazyHandleSupplier.java:72)
at org.jdbi.v3.core.internal.JdbiThreadLocals.invokeInContext(JdbiThreadLocals.java:27)
at org.jdbi.v3.core.LazyHandleSupplier.invokeInContext(LazyHandleSupplier.java:71)
at org.jdbi.v3.sqlobject.SqlObjectFactory.lambda$createInvocationHandler$14(SqlObjectFactory.java:162)
at org.jdbi.v3.core.OnDemandExtensions.invoke(OnDemandExtensions.java:76)
at org.jdbi.v3.core.OnDemandExtensions.lambda$null$0(OnDemandExtensions.java:63)
at org.jdbi.v3.core.internal.JdbiThreadLocals.invokeInContext(JdbiThreadLocals.java:27)
at org.jdbi.v3.core.OnDemandExtensions.lambda$null$1(OnDemandExtensions.java:62)
at org.jdbi.v3.core.Jdbi.withExtension(Jdbi.java:439)
at org.jdbi.v3.core.OnDemandExtensions.lambda$create$2(OnDemandExtensions.java:61)
at com.sun.proxy.$Proxy201.createFunctionNamespacesTableIfNotExists(Unknown Source)
at com.facebook.presto.functionNamespace.mysql.MySqlFunctionNamespaceManager.initialize(MySqlFunctionNamespaceManager.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.facebook.airlift.bootstrap.LifeCycleManager.startInstance(LifeCycleManager.java:240)
... 20 more
Caused by: java.sql.SQLException: No suitable driver found for jdbc:mysql://hostname:3306
at java.sql.DriverManager.getConnection(DriverManager.java:689)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at com.facebook.presto.functionNamespace.mysql.MySqlConnectionModule.lambda$setup$0(MySqlConnectionModule.java:41)
at org.jdbi.v3.core.Jdbi.open(Jdbi.java:285)
... 43 more
I have configured the MySQL connector in case that has anything to do. The log shows that this loads fine and (if I remove the function manager) I can actually run queries through that connector. However, the MySQL JDBC driver does not seem to be found when the function manager needs it.
I have tried many different formats of the database-url string. I have also tried copying mysql-connector-java.jar into various places. None of that has worked. What is the problem?
Aha, I made some progress. I found the documentation on Alluxio mentioning adding something to the classpath, so I tried adding the following to etc/jvm.properties
:
-Xbootclasspath/a:/path/to/presto-server-0.246/plugin/mysql/mysql-connector-java-5.1.48.jar
Now, I get this:
Caused by: java.sql.SQLException: Access denied for user ''@'10.111.1.11' (using password: NO)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3933)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3869)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:864)
at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1707)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1217)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2189)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2220)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2015)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:768)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:385)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:323)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:270)
at com.facebook.presto.functionNamespace.mysql.MySqlConnectionModule.lambda$setup$0(MySqlConnectionModule.java:41)
at org.jdbi.v3.core.Jdbi.open(Jdbi.java:285)
... 43 more
So back to my earlier question: How do I specify the credentials?
OK, I think I found it. The connection string in etc/function-namespace/example.properties
needs to provide it:
database-url=jdbc:mysql://hostname:3306/dbname?user=root&password=password
Note that this also required to create a new database and provide that in the connection string as well...
For completeness and to help with Google searches, this is the message I get if I don't specify a database name:
Caused by: java.sql.SQLException: No database selected
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3933)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3869)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2675)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1912)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1251)
at org.jdbi.v3.core.statement.SqlLoggerUtil.wrap(SqlLoggerUtil.java:29)
at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1381)
... 46 more
OK, I finally made it work. The final obstacle was that you have to manually create an entry in one of the database tables in order to create a function namespace (and you can't create a function without such a namespace). Otherwise, you get:
Query 20210129_095004_00001_n4n2d failed: Function namespace not found: example.default
While this is mentioned in the documentation, I think (1) it shouldn't be under Configuration, but in a dedicated section Creating Function Namespaces or similar, and (2) I think that the default namespace should be created by default when the corresponding table gets created.
In summary: I managed to make the function namespace manager work and I can run UDFs now, but the documentation is very incomplete and should be improved.
I put together a working Docker image in case someone is curious: https://github.com/ingomueller-net/docker-presto.
Sorry about the rough experience. Engineers are in general not famous for writing good documentations. It would be great if you can help us improve the documentation by sending us a PR! Thanks a lot!
Hi @ingomueller-net This is a great article and very easy to consume for users. Can you please send us PR and I can get this added to community documentation.
I encountered the same issue, the documentation does not work at all.
For the issue: "Explicit bindings are required and com.facebook.drift.transport.client.MethodInvokerFactory
binder.bind(new TypeLiteral<MethodInvokerFactory<Annotation>>(){})
.toInstance((MethodInvokerFactory<Annotation>) DriftNettyMethodInvokerFactory.createStaticDriftNettyMethodInvokerFactory(new DriftNettyClientConfig()));
configBinder(binder).bindConfig(DriftNettyClientConfig.class);
and we need to add the following new dependency:
<dependency>
<groupId>com.facebook.drift</groupId>
<artifactId>drift-transport-netty</artifactId>
<version>1.33</version>
<type>jar</type>
</dependency>
@igor-suhorukov how do you solve it? @rongrong do you have recommended solution?
I am failing to make the function namespace manager work. It is quite obvious that the documentation is incomplete; it should for the very least say that the official releases do not include that functionality yet.
It would also help if there was a description of how to build a release that includes the function namespace manager. When I compile with
mvn package -DskipTests
, I do see that corresponding jars are built inpresto-function-namespace-managers/target/
. However, they are not copied into theplugin
folder of thepresto-server
target.If I copy manually
presto-function-namespace-managers/target/presto-function-namespace-managers-0.245.1/*.jar
from the build folder toplugin/function-namespace-managers/
of my Presto installation, Presto fails to start. (Configuration as recommended in the documentation). Inserver.log
, I get:Note that if I change the configuration to
experimental-functions-enabled=true
, the same thing happens. What is missing?