search

cisagov / LME

Logging Made Easy (LME) is a no-cost and open logging and protective monitoring solution serving all organizations.
https://www.cisa.gov/resources-tools/services/logging-made-easy
Other
831 stars 65 forks source link

Maintenance - Safe Shutdown / Preserve state after server downtime #436

Open arunseelagan-cisa opened 1 month ago

arunseelagan-cisa commented 1 month ago

Is your feature request related to a problem? Please describe. When we need to take down the server hosting LME for maintenance purposes, what is the best way to stop the LME docker stack without completely destroying the existing containers?

We want to be able to re-run docker stack after the downtime and still have access to all the data that was present beforehand.

Describe the solution you'd like Please add a section under the "Maintenance" section of https://github.com/cisagov/LME, where you provide guidance on how to properly stop LME in a manner that preserves state.

Describe alternatives you've considered Tried to run "docker stop " but the containers come right back up automatically.

docker remove commands are not a good alternative, because that would theoretically destroy all the data inside the containers.

Additional context N/A

aarz-snl commented 3 weeks ago

May want to look into docker stack commands...

Something like

docker stack services lme -q | xargs docker service update --replicas 0

Which would temporarily stop the containers

And then to restart

docker stack services lme -q | xargs docker service update --replicas 1

aarz-snl commented 3 weeks ago

This worked for me:

docker service scale lme_elasticsearch=0 lme_kibana=0 lme_logstash=0

Then

docker service scale lme_elasticsearch=1 lme_kibana=1 lme_logstash=1 To restart

took about a minute to stop them, and about 3 minutes to start back up