search

docker-flink / examples

Examples for how to use the Flink Docker images in a variety of ways
Apache License 2.0
91 stars 46 forks source link

HA mode not activated java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.SecurityUtil #4

Open Vince-Cercury opened 6 years ago

Vince-Cercury commented 6 years ago

I've setup a zookeeper:

helm --namespace vince2 install --name vince2-zookeeper incubator/zookeeper

created a values.yaml

cat > values.yaml <<EOF flink: num_taskmanagers: 3 highavailability: enabled: true zookeeper_quorum: vince2-zookeeper-zookeeper state_s3_bucket: s3://my-bucket/vince2/ha EOF

( don't need AWS keys, I have instanceProfile allowing any pod to read/write into that S3. Testing with another component)

Deploy FLink:

helm --namespace vince2 install --name=vince2-flink-ha --values=values.yaml flink-1.4.0.tgz

But the job manager logs says it is not starting with HA

2018-04-16 04:59:11,606 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager without high-availability 2018-04-16 04:59:11,609 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager on vince2-flink-ha-flink-jobmanager:6123 with execution mode CLUSTER

It seems the HA settings didn't make it to the configmap:

flink-conf.yaml: | blob.server.port: 6124 jobmanager.rpc.address: vince2-flink-ha-flink-jobmanager jobmanager.rpc.port: 6123 jobmanager.heap.mb: 1024 taskmanager.heap.mb: 1024 taskmanager.numberOfTaskSlots: 1

Vince-Cercury commented 6 years ago

I could load the right settings in configmap.

However with HA, it's missing a SecurityUtil

Exception in thread "main" java.lang.NoClassDefFoundError: Could not initialize class org.apache.hadoop.security.SecurityUtil

Has someone got it working with HA?

patricklucas commented 6 years ago

Hmm. Usually, I would say that this is a consequence of using Hadoop-less Flink but using an s3:// filesystem URI without copying the Presto S3 filesystem implementation from opt to lib, but if you're using the Flink images hosted on Docker Hub, they call include Hadoop.

Have you searched the Flink JIRA issues and mailing list? Someone might have run into this before.