search

peelframework / peel

Peel is a framework that helps you to define, execute, analyze, and share experiments for distributed systems and algorithms.
http://peel-framework.org
Apache License 2.0
27 stars 32 forks source link

Using different/custom version of Apache Flink #110

Closed mnoumanshahzad closed 7 years ago

mnoumanshahzad commented 7 years ago

Hi, I am trying to benchmark a custom Apache Flink build and Apache Flink v1.1.4. Apparently the current Peel Framework code runs experiments on Apache Flink v1.1 Can you kindly point out the changes I need to make in the project to be able to use the other versions of Apache Flink.

Thanks

aalexandrov commented 7 years ago

@mnoumanshahzad We can certainly help.

The lasest version of Peel added support for Flink v1.1.4.

To migrate to Flink v1.1.4 you have to:

1. Update Peel

First you need to update the peel-core and peel-extensions versions used in your project. To do this, change the peel.version to 1.1.4 in your main pom.xml (FYI, the fact that the Flink version you want to use and the Peel version you have to use to get it coincide is a pure coincidence).

To verify that this works, re-build the project

rm -Rf $BUNDLE_BIN/$BUNDLE_NAME/lib
mvn clean deploy -Pdev

and then from the $BUNDLE_BIN/$BUNDLE_NAME run

./peel.sh sys:setup flink-1.1.4

At this point Flink 1.1.4 should be up and running on your developer machine.

2. Possibly Redefine the Flink 1.1.4 Bean

The latest version of peel-extensions already contains a definition of a flink-1.1.4 bean that you can use in your Experiments Definitions.

However, in case in your bundle defines experiments where Flink always depends on other systems (e.g. HDFS or Zookeeper), chances are you will find a file called systems.scala (if you use Scala-based) configuration, or in your systems.xml (if you use an XML-based configuration) that overrides the flink-1.1 bean definition in order to add the additional system dependencies. If this is the case, replicate the code snippet and adapt it in order to override the flink-1.1.4 bean definition in a similar manner.

3. Adapt The Experiment Definitions

In your experiments definition, refer to flink-1.1.4 as the Experiment runner, e.g. in Scala the syntax will be

runner = ctx.getBean("flink-1.1.4", classOf[Flink])

whereas before it was (probably)

runner = ctx.getBean("flink-1.1.0", classOf[Flink])

Once you adapt the experiment definitions, go to $BUNDLE_BIN/$BUNDLE_NAME and from the console type

find config/ -name '*.class' -delete

Then you should be able to run the experiment with flink-1.1.4.