After thousands of log files have accumulated, opening the Historical page becomes slow. A lot of the wait time comes from the list_log_files function, which scans the log files and summarizes them. I profiled this function and found that nearly half of the execution time is spent in dateutil.parser.parse, which is used to parse the time from the beginning of the log filename. I presume the parse function is slow because it must guess what time format has been used.
I tried switching to datetime.datetime.strptime, and this gave a significant speedup, but still a decent amount of time was spent in strptime, perhaps because it needs to parse the format string argument. Manually converting the pieces to integers and passing them directly into datetime.datetime() reduced the processing time to a negligible amount, and reduced the execution time of list_log_files by about 45%.
After thousands of log files have accumulated, opening the Historical page becomes slow. A lot of the wait time comes from the
list_log_filesfunction, which scans the log files and summarizes them. I profiled this function and found that nearly half of the execution time is spent indateutil.parser.parse, which is used to parse the time from the beginning of the log filename. I presume theparsefunction is slow because it must guess what time format has been used.I tried switching to
datetime.datetime.strptime, and this gave a significant speedup, but still a decent amount of time was spent instrptime, perhaps because it needs to parse the format string argument. Manually converting the pieces to integers and passing them directly intodatetime.datetime()reduced the processing time to a negligible amount, and reduced the execution time oflist_log_filesby about 45%.