Optimize CRON File and Logging

[lovelysandlonelys]

Two distinct tasks

1. Rather than having so many subfolders have a logs directory, each log has it's own directory, and then there are log files with the date inside of that model file rather than having so many nested folders.
   • logs
   • • mlp model
   • • • mlm-model-yymmdd
2. Automate the creation of the cron file by reading the timing information from the dspec
   • have an enabled boolean inside of the dspec to make sure that we can show what models we want to be running to the script that builds the cron file

NOTE: the CRON logs are just logging only the standard error not the cron log errors so they are kinda useless

• test, look at it again, maybe pipe to the same log file, make sure that the log files are logging as expected

NOTE: the offset value for each inundation model should be offset by some seconds so that all the models are not all running at the same time and utilizing 80% of the CPU

[matdenkas]

lead_times = [12, 24, 48]
for lead_time in lead_times:

    FOLDER_STRUCTURE = f'Magnolia/{lead_time}'

    min = 20
    file_path = f'Magnolia/{lead_time}/magnolia_{lead_time}'
    model = f'{min} * * * * mkdir -p ./logs/$(date "+\%Y")/$(date "+\%m")/{FOLDER_STRUCTURE} && docker exec semaphore-core python3 src/semaphoreRunner.py -d ./{file_path}.json >> ./logs/$(date "+\%Y")/$(date "+\%m")/{file_path}.log'

    min = 25
    file_path = f'Magnolia/{lead_time}/magnolia_transform_{lead_time}'
    transform = f'{min} * * * * mkdir -p ./logs/$(date "+\%Y")/$(date "+\%m")/{FOLDER_STRUCTURE} && docker exec semaphore-core python3 src/semaphoreRunner.py -d ./{file_path}.json >> ./logs/$(date "+\%Y")/$(date "+\%m")/{file_path}.log'

    print(model)
    print(transform)

[lovelysandlonelys]

• [x] Discuss CRON_log -> Joy!
• [x] Go over proposed flow of init_cron.py
  • [x] Discuss headers in the cron file
  • [x] Discuss and reformat cronjob string to reflect new log file hierarchy
  • [x] Discuss adding a marker to the dspec for weather or not we want a model to run
• [x] Stagger the timing information on the inundation models as to not overwhelm the server
• [x] Refine init_cron.py plan
• [x] Add model run bool
• [x] Add comment jsons
• [ ] Fix cron strings
• [x] Fill in init_cron.py functions

[lovelysandlonelys]

Meeting Notes:

• Use One file for cron log rather than each model having it's own file because we only want to catch when cron is having issues running models not when there is an issue with each model individually
• We're going to use a recursive function to walk through the dspc file structure for comments in the cron file
  • we want to add a comment.json file to logs for more descriptive comments about models
    • we want to add a run T/F marker

[lovelysandlonelys]

If the interval is minutely:

• just have a special case for when the interval is less than an hour?
  • 5-59/10
  • This would run correctly on our version of cron for running every ten minutes with an offset of five minutes
  • so if the interval is less than an hour we set the offset to offset-59 so that it evaluates correctly?

[matdenkas]

print(format_timing(300, 3600)) # Hourly run with offset of 10 min
print(format_timing(0, 360)) # No offset run every six min print(format_timing(1200, 10800)) # 20 min offset run every 3 hours print(format_timing(0, 3600)) # No offset run hourly print(format_timing(60, 360)) # No offset run every six min

5 /1 0-59/6 20 /3 0 /1 1-59/6