search

pinpoint-apm / pinpoint

APM, (Application Performance Management) tool for large-scale distributed systems.
https://pinpoint-apm.gitbook.io/
Apache License 2.0
13.42k stars 3.76k forks source link

Application Inspector flink version supported #7550

Open marty-macfly opened 3 years ago

marty-macfly commented 3 years ago

Thanks for the work done on Pinpoint, I've just setup application inspector job on Flink version 1.3.1 which is the Flink version used inside the pinpoint-apm/pinpoint-docker, it's working properly but Flink version 1.3.1 is really old released in 08/2017.

So I've tried to bump the Flink version to see how far I can go and I reached to update flink to version 1.8.3 which has been released on 12/2019 which is better but still an outdated version.

I've tried to update to version 1.9 or higher and job loading failed with the following error:

pinpoint-flink-jobmanager_1   | java.lang.RuntimeException: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
pinpoint-flink-jobmanager_1   |         at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36)
pinpoint-flink-jobmanager_1   |         at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
pinpoint-flink-jobmanager_1   | Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:152)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:83)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:376)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34)
pinpoint-flink-jobmanager_1   |         ... 7 more
pinpoint-flink-jobmanager_1   | Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'Window(TumblingEventTimeWindows(10000), EventTimeTrigger, ApplicationStatBoWindow) -> Sink: Unnamed': Configuring the output format (null) failed: The runtime context has not been initialized yet. Try accessing it in one of the other life cycle methods.
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:218)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:106)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.scheduler.LegacyScheduler.createExecutionGraph(LegacyScheduler.java:207)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.scheduler.LegacyScheduler.createAndRestoreExecutionGraph(LegacyScheduler.java:184)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.scheduler.LegacyScheduler.<init>(LegacyScheduler.java:176)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.scheduler.LegacySchedulerFactory.createInstance(LegacySchedulerFactory.java:70)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:266)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:146)
pinpoint-flink-jobmanager_1   |         ... 10 more
pinpoint-flink-jobmanager_1   | Caused by: java.lang.Exception: Configuring the output format (null) failed: The runtime context has not been initialized yet. Try accessing it in one of the other life cycle methods.
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:96)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:214)
pinpoint-flink-jobmanager_1   |         ... 20 more
pinpoint-flink-jobmanager_1   | Caused by: java.lang.IllegalStateException: The runtime context has not been initialized yet. Try accessing it in one of the other life cycle methods.
pinpoint-flink-jobmanager_1   |         at org.apache.flink.api.common.io.RichOutputFormat.getRuntimeContext(RichOutputFormat.java:47)
pinpoint-flink-jobmanager_1   |         at com.navercorp.pinpoint.flink.dao.hbase.StatisticsDao.configure(StatisticsDao.java:56)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:93)
pinpoint-flink-jobmanager_1   |         ... 21 more
pinpoint-flink-jobmanager_1   | 2021-01-13 06:23:13,590 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler    - Unhandled exception.
pinpoint-flink-jobmanager_1   | org.apache.flink.runtime.client.JobSubmissionException: Failed to submit job.
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$internalSubmitJob$2(Dispatcher.java:333)
pinpoint-flink-jobmanager_1   |         at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
pinpoint-flink-jobmanager_1   |         at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
pinpoint-flink-jobmanager_1   |         at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:44)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
pinpoint-flink-jobmanager_1   |         at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
pinpoint-flink-jobmanager_1   | Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
pinpoint-flink-jobmanager_1   |         at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:36)
pinpoint-flink-jobmanager_1   |         at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
pinpoint-flink-jobmanager_1   |         ... 6 more
pinpoint-flink-jobmanager_1   | Caused by: org.apache.flink.runtime.client.JobExecutionException: Could not set up JobManager
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:152)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.dispatcher.DefaultJobManagerRunnerFactory.createJobManagerRunner(DefaultJobManagerRunnerFactory.java:83)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.dispatcher.Dispatcher.lambda$createJobManagerRunner$5(Dispatcher.java:376)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.util.function.CheckedSupplier.lambda$unchecked$0(CheckedSupplier.java:34)
pinpoint-flink-jobmanager_1   |         ... 7 more
pinpoint-flink-jobmanager_1   | Caused by: org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'Window(TumblingEventTimeWindows(10000), EventTimeTrigger, ApplicationStatBoWindow) -> Sink: Unnamed': Configuring the output format (null) failed: The runtime context has not been initialized yet. Try accessing it in one of the other life cycle methods.
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:218)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:106)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.scheduler.LegacyScheduler.createExecutionGraph(LegacyScheduler.java:207)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.scheduler.LegacyScheduler.createAndRestoreExecutionGraph(LegacyScheduler.java:184)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.scheduler.LegacyScheduler.<init>(LegacyScheduler.java:176)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.scheduler.LegacySchedulerFactory.createInstance(LegacySchedulerFactory.java:70)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:278)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.JobMaster.<init>(JobMaster.java:266)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:98)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.createJobMasterService(DefaultJobMasterServiceFactory.java:40)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobmaster.JobManagerRunner.<init>(JobManagerRunner.java:146)
pinpoint-flink-jobmanager_1   |         ... 10 more
pinpoint-flink-jobmanager_1   | Caused by: java.lang.Exception: Configuring the output format (null) failed: The runtime context has not been initialized yet. Try accessing it in one of the other life cycle methods.
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:96)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:214)
pinpoint-flink-jobmanager_1   |         ... 20 more
pinpoint-flink-jobmanager_1   | Caused by: java.lang.IllegalStateException: The runtime context has not been initialized yet. Try accessing it in one of the other life cycle methods.
pinpoint-flink-jobmanager_1   |         at org.apache.flink.api.common.io.RichOutputFormat.getRuntimeContext(RichOutputFormat.java:47)
pinpoint-flink-jobmanager_1   |         at com.navercorp.pinpoint.flink.dao.hbase.StatisticsDao.configure(StatisticsDao.java:56)
pinpoint-flink-jobmanager_1   |         at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:93)
pinpoint-flink-jobmanager_1   |         ... 21 more

Is it because I'm missing something or just becvause pinpoint-flink-job is not compatible with flink version higher than 1.8, if so is it in the pipe to make it compatible ?

minwoo-jung commented 3 years ago

Hello @marty-macfly Pinpoint has been officially tested up to version 1.7.1. So, it is recommended to use 1.7.1 by upgrading the version instead of 1.3.1.

For the newer version, we haven't tested it yet, so we recommend looking at the problem. Perhaps that is the case that there is no problem with the pinpoint logic and the context cannot be handled inside the flink.

@RoySRose It is better to upgrade the flink version used in the docker image.

marty-macfly commented 3 years ago

Hello @minwoo-jung

Thanks for your feedback, I've update my PR #100 on pinpoint-docker to bump Flink to version 1.7 and load the job when flink startup so @RoySRose can have a look.

journeagle commented 3 years ago

I upgraded flink v1.12.3 and got the same error. The answer I got through the query is that getRuntimeContext can only be called within the open method. 12

I'm trying to move the code under the configure method to the open method, and it works, but I don't know if this is the right way to handle it.

flink

unc1 commented 3 years ago

Is there any change in the supported Flink version? Or is it still 1.7.2? Thanks.