search

garyfeng / flink-playgrounds

Apache Flink Playgrounds
https://flink.apache.org/
Apache License 2.0
0 stars 0 forks source link

planner missing #3

Closed garyfeng closed 4 years ago

garyfeng commented 4 years ago

After fixing #1 and #2, I tried to run the python code and got the following error

>>> from pyflink.table.descriptors import Schema, Rowtime, Json, Kafka
>>> Json()
<pyflink.table.descriptors.Json object at 0x7fe8671620f0>
>>> Kafka()
<pyflink.table.descriptors.Kafka object at 0x7fe866c92240>
>>> import os
>>>
... from pyflink.datastream import StreamExecutionEnvironment, TimeCharacteristic
>>> from pyflink.table import StreamTableEnvironment, CsvTableSink, DataTypes
>>> from pyflink.table.descriptors import Schema, Rowtime, Json, Kafka
>>> from pyflink.table.window import Tumble
>>>
>>> s_env = StreamExecutionEnvironment.get_execution_environment()
>>> s_env.set_parallelism(1)
<pyflink.datastream.stream_execution_environment.StreamExecutionEnvironment object at 0x7fe8667264e0>
>>> s_env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
>>> # Stream Table
... st_env = StreamTableEnvironment.create(s_env)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/apache_flink-1.10.dev0-py3.7.egg/pyflink/util/exceptions.py", line 147, in deco
    return f(*a, **kw)
  File "/usr/local/lib/python3.7/dist-packages/py4j-0.10.8.1-py3.7.egg/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.flink.table.api.java.StreamTableEnvironment.create.
: org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
        at org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.lookupExecutor(StreamTableEnvironmentImpl.java:147)
        at org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.create(StreamTableEnvironmentImpl.java:116)
        at org.apache.flink.table.api.java.StreamTableEnvironment.create(StreamTableEnvironment.java:112)
        at org.apache.flink.table.api.java.StreamTableEnvironment.create(StreamTableEnvironment.java:83)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
        at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
        at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)

It looks like it is missing a planner module. Is it because it doesn't find a planner JAR?

garyfeng commented 4 years ago

from the error message it looks like it was trying to look for a connector to elasticsearch6. Full error, but still not showing 29 others

... st_env = StreamTableEnvironment.create(s_env)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/apache_flink-1.10.dev0-py3.7.egg/pyflink/util/exceptions.py", line 147, in deco
    return f(*a, **kw)
  File "/usr/local/lib/python3.7/dist-packages/py4j-0.10.8.1-py3.7.egg/py4j/protocol.py", line 328, in get_return_value
    format(target_id, ".", name), value)
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.flink.table.api.java.StreamTableEnvironment.create.
: org.apache.flink.table.api.TableException: Could not instantiate the executor. Make sure a planner module is on the classpath
        at org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.lookupExecutor(StreamTableEnvironmentImpl.java:147)
        at org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.create(StreamTableEnvironmentImpl.java:116)
        at org.apache.flink.table.api.java.StreamTableEnvironment.create(StreamTableEnvironment.java:112)
        at org.apache.flink.table.api.java.StreamTableEnvironment.create(StreamTableEnvironment.java:83)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
        at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
        at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282)
        at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79)
        at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.flink.table.api.TableException: Could not load service provider for table factories.
        at org.apache.flink.table.factories.TableFactoryService.discoverFactories(TableFactoryService.java:218)
        at org.apache.flink.table.factories.TableFactoryService.findAllInternal(TableFactoryService.java:170)
        at org.apache.flink.table.factories.TableFactoryService.findAll(TableFactoryService.java:125)
        at org.apache.flink.table.factories.ComponentFactoryService.find(ComponentFactoryService.java:48)
        at org.apache.flink.table.api.java.internal.StreamTableEnvironmentImpl.lookupExecutor(StreamTableEnvironmentImpl.java:138)
        ... 14 more
Caused by: java.util.ServiceConfigurationError: org.apache.flink.table.factories.TableFactory: Provider org.apache.flink.streaming.connectors.elasticsearch6.Elasticsearch6UpsertTableSinkFactory could not be instantiated
        at java.util.ServiceLoader.fail(ServiceLoader.java:232)
        at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
        at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
        at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at org.apache.flink.table.factories.TableFactoryService.discoverFactories(TableFactoryService.java:214)
        ... 18 more
Caused by: java.lang.NoClassDefFoundError: org/elasticsearch/common/xcontent/XContentType
        at org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchUpsertTableSinkFactoryBase.<clinit>(ElasticsearchUpsertTableSinkFactoryBase.java:105)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
        ... 22 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.common.xcontent.XContentType
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
        ... 29 more
garyfeng commented 4 years ago

Turns out I was wrong in #1 and #2. The FLINK_HOME should be set to /opt/flink. Then I just need to copy the missing JAR files to /opt/flink/lib, instead of the complicated python packing step.

Tried with kafka -- this fixed the Kafka() issue but left the same error for Json(). Copied json formmater JAR and this solves the problem.

garyfeng commented 4 years ago

Problem fixed. Shouldn't have changed FLINK_HOME to the python lib. It should have been the /opt/flink directory, just needed to copy the missing JARs there.