GMAP / DSPBench

a suite of benchmark applications for distributed data stream processing systems
MIT License
27 stars 3 forks source link

Update of Apache Storm to a more recent version #3

Closed crommands closed 2 years ago

crommands commented 3 years ago

The version of Storm this project currently uses is from 2014. I didn't see any mention in the paper of why this specific version was selected, so I wanted to ask if there is any plan to update it?

I had a look at it myself because the framework independent architecture makes this a far smaller task than in other benchmarks, but I haven't been able to figure out all the details.

For the most part, updating the dependency versions seems to work without any API breaking changes, until Storm 1.2.3, where the development of the module storm-kafka was stopped and replaced with storm-kafka-client. This only affects the StormKafkaSpout, but I haven't fully figured out how to update it with the changed API. Additionally, following version 1.0.0 of Storm the backtype.storm.* import statements should be, and for later versions have to be, replaced with org.apache.storm.* imports. All the APIs stay identical, so that is a relatively quick process.

Some of the other dependencies also need to be updated or provide benefits, such as being able to build the project with newer JDKs than 8:

mockito-all 1.10.8 -> mockito-core 3.11.2
commons-lang3.version->3.8.1
org.geotools gt-shapefile&gt-xml -> 25.2
kafka -> something recent

That is everything I have figured so far which would be necessary to reach Storm 2.2.0 and be up to date, but there are probably things I overlooked and again I haven't figured out the StormKafkaSpout. Additionally, I don't know how these dependency changes affect the other platforms, given that they currently aren't in the repository.

mayconbordin commented 3 years ago

Hi, we are working on a new version of the framework with an updated version of Storm, as well as the other libraries we use.

dalvangriebler commented 2 years ago

@crommands we are in the process to release consolidated and updated versions of Storm in the next weeks. You can follow this on branch V2.

mayconbordin commented 2 years ago

4