Issue:
The lmod tracking has stopped logging after Oct 27th. The issue seems to be with lmod mysql DB reading the config file that has mysql credentials. This lead to the lmod tracking data not being ingested into the database from lmod log files. After certain period, the log rotate started clearing the old files which are yet to push the data into mysql DB because of issue with reading encoded password. However, we've been rsyncing /var/log on cm01 to a backup location on /data/rc nightly. Mike (@flakrat) copied all of the missing moduleUsage.log-%DATE% files to /data/rc/lmod_logs. We should restore missing data into the DB with them.
Work around
Modify the log rotate config to move old files after log rotation into a different location so that log files will be preserved until we fix this issue.
Solution
Fix the error we see with python-mysql client not being able to read the creds from config file.
Issue: The lmod tracking has stopped logging after Oct 27th. The issue seems to be with lmod mysql DB reading the config file that has mysql credentials. This lead to the lmod tracking data not being ingested into the database from lmod log files. After certain period, the log rotate started clearing the old files which are yet to push the data into mysql DB because of issue with reading encoded password. However, we've been rsyncing /var/log on cm01 to a backup location on /data/rc nightly. Mike (@flakrat) copied all of the missing moduleUsage.log-%DATE% files to /data/rc/lmod_logs. We should restore missing data into the DB with them.
Work around Modify the log rotate config to move old files after log rotation into a different location so that log files will be preserved until we fix this issue.
Solution Fix the error we see with python-mysql client not being able to read the creds from config file.