ChrisM-S
commented
7 years ago Coincidentally, we were discussing this as you wrote the ticket! IMAT has currently got about about 60GB in its EPICS logs directory (30GB IOC logs, 30GB conserver logs). By a strange quirk of fate these are all on an even more giant data disk so won't be a problem - but it would have sunk most other instruments by now. The conserver logs, although few, come at the top of the treesize top 100. The IOCS are all small(ish) in comparison and chopped into daily chunks but there are way to many of them. I guess a few other strategies to handle fully this might be:
- [ ] 1. Close and re-open all log files into bite sized chunks at a fixed size limit ( then it is easy to trim away the closed files even if there is a lot of random data being written.
- [ ] 2. Make console logs and others also save new files daily/hourly like the IOCS.
- [ ] 3. Keep the log routines only writing lines if they are really different (judged by a regex maybe to exclude changing times or digits which change insignificantly because of jitter).
- [ ] 4. Keep the logging frequency deliberately slow (e. g. only 1 message per second regardless of what it is) this throttles to prevent a runaway log file - so at 132 characters per line say, there would only be 10MB/day max.
Last but not least
- [ ] 5. Make all log files cycle so they clean themselves up at what is judged to be an appropriate interval. For example, leave the year, month or even week out of the name so that they overwrite monthly, weekly or daily. e.g. SKFCHOPPER_03-17.log (day 17 of a monthly log).
It is possible for a log file under IBEX to grow large. This often means 1 or more pvs that can not read. If this happens it would be good to have a way of being alerted to this problem and potentially be able to fix it.
This can not be done under Nagios because it does not play well with file sizes and wildcards.
It might be good as a PV under the blockserver. This could report the top ten largest files today and their sizes.