search

igabriel85 / DICE-Monitoring

IeAT Repository for H2020 DICE Project
http://www.dice-h2020.eu/
Apache License 2.0
2 stars 2 forks source link

Add capability to collect Storm logs via dmon-agent #152

Open igabriel85 opened 8 years ago

igabriel85 commented 8 years ago

Extend the capabilities of dmon-agent so that it can locate (if necessary) and then forward storm worker logs to dmon. Implement method

GET -> /v1/node/stwork

Return the log of current storm worker.

Worker logs are usually located in ../apache-storm-0.9.3/logs. The ones with the format worker*.logs.

The logging directory can also be set from the config yaml file at:

storm.log.dir: /path/to/storm/logdir

or using the command line with:

-Dstorm.log.dir=/opt/storm/apache-storm-0.9.3/logs

matejart commented 7 years ago

When deploying the Storm worker nodes, will we need to add any steps additional to the current ones? Right now we basically register a Storm nimbus node with the DMon and request a restart of logstash. In the next releases, will we need to be careful to also prepare a dmon configuration for Storm workers? Will the dmon-agent serve the logs or will some other log pushing mechanism exist?

igabriel85 commented 7 years ago

I have just pushed our solution to collect to this repository. Details are in the I have just sent. We only need to set the STORM_LOG environmental variable.

@matejart Do you set it when installing Storm?