search

CSIRT-MU / Stream4Flow

A framework for the real-time network traffic analysis based on world-leading technologies for distributed stream processing, network traffic monitoring, and visualization.
https://csirt.muni.cz/?lang=en
MIT License
99 stars 36 forks source link

Enable dynamic allocation of the Spark resources #73

Closed cermmik closed 6 years ago

cermmik commented 6 years ago

Set spark.streaming.dynamicAllocation.enabled=true in the run_application.sh to allow Spark decide the correct number of resources.

To enable dynamic allocation it is necessary to modify the microbatch size to fit the window size for all applications.

cermmik commented 6 years ago

Created branch dynamic-resourcess-allocation to update all applications.

tomjirsa commented 6 years ago

merged in #79. Closing