│ 2024-02-06 06:45:09,228 - MainThread - INFO -
--------------------------------------------
│
│ Exception in thread producer:
│
│ Traceback (most recent call last):
│
│ File "/usr/local/lib/python3.10/threading.py", line 1016, in
_bootstrap_inner │
│ self.run()
│
│ File "/app/src/pay/payment_producer.py", line 241, in run
│
│ get_verbose_log_helper().reset(pymnt_cycle)
│
│ File "/app/src/verbose_logging_helper.py", line 75, in reset
│
│ self.archive(old_path)
│
│ File "/app/src/verbose_logging_helper.py", line 96, in archive
│
│ self.remove_oldest(archive_dir)
│
│ File "/app/src/verbose_logging_helper.py", line 108, in
remove_oldest
│
│ self.remove_oldest(archive_dir)
│
│ File "/app/src/verbose_logging_helper.py", line 108, in
remove_oldest
│
│ self.remove_oldest(archive_dir)
│
│ File "/app/src/verbose_logging_helper.py", line 108, in
remove_oldest
│
│ self.remove_oldest(archive_dir)
│
│ [Previous line repeated 988 more times]
│
│ File "/app/src/verbose_logging_helper.py", line 99, in remove_oldest
│
│ files = [
│
│ File "/app/src/verbose_logging_helper.py", line 100, in <listcomp>
│
│ os.path.join(archive_dir, f)
│
│ File "/usr/local/lib/python3.10/posixpath.py", line 77, in join
│
│ sep = _get_sep(a)
│
│ File "/usr/local/lib/python3.10/posixpath.py", line 42, in _get_sep
│
│ if isinstance(path, bytes):
│
│ RecursionError: maximum recursion depth exceeded while calling a
Python object
After updating to latest master.
Explanation: TRD has a mechanism to zip up logs. there is a function to remove logs that are too old. This function was searching for files with zip extension. But, these logs are in gz extension. Viktor fixed this in his PR "Record and playback".
But in my production bakers, I have a ton of these gz files. This deletion function calls itself recursively, but it goes over the recursion limit of python.
This PR fixes the function to turn it into a simple loop rather than a recursive loop.
I'll let chatGPT explain:
The error you're encountering, RecursionError: maximum recursion depth exceeded while calling a Python object, indicates that your remove_oldest function is causing a stack overflow due to too many recursive calls. This typically happens when the base case of the recursion is not reached or not defined properly, causing the function to call itself indefinitely.
In your remove_oldest function, you recursively call self.remove_oldest(archive_dir) within itself. This will continue indefinitely if the condition len(sorted_files) > self.keep_at_most remains True. If files are not being removed as expected, or if self.keep_at_most is set incorrectly, this can lead to an infinite loop and thus a recursion error.
To fix this issue, you need to ensure that each recursive call makes progress towards a base case where the recursion will stop. Here's a revised version of your method with a loop instead of recursion, which is generally safer for this kind of operation:
def remove_oldest(self, archive_dir):
while True:
files = [
os.path.join(archive_dir, f)
for f in os.listdir(archive_dir)
if self.is_archive_file(f)
]
sorted_files = sorted(files, key=os.path.getmtime)
if len(sorted_files) <= self.keep_at_most:
break # Exit the loop if the number of files is within the limit
os.remove(sorted_files[0]) # Remove the oldest file
In this revised version:
The function uses a while True loop to continually check and remove the oldest file until the number of files is within the desired limit (self.keep_at_most). This approach avoids recursion, eliminating the risk of a stack overflow error. The loop will exit (break) when the condition len(sorted_files) <= self.keep_at_most is met, meaning the number of files is now within the limit, and no more files need to be removed. This should resolve the RecursionError and ensure that your function properly maintains the number of archive files within the specified limit.
I got this:
After updating to latest master.
Explanation: TRD has a mechanism to zip up logs. there is a function to remove logs that are too old. This function was searching for files with zip extension. But, these logs are in gz extension. Viktor fixed this in his PR "Record and playback".
But in my production bakers, I have a ton of these gz files. This deletion function calls itself recursively, but it goes over the recursion limit of python.
This PR fixes the function to turn it into a simple loop rather than a recursive loop.
I'll let chatGPT explain:
The error you're encountering, RecursionError: maximum recursion depth exceeded while calling a Python object, indicates that your remove_oldest function is causing a stack overflow due to too many recursive calls. This typically happens when the base case of the recursion is not reached or not defined properly, causing the function to call itself indefinitely.
In your remove_oldest function, you recursively call self.remove_oldest(archive_dir) within itself. This will continue indefinitely if the condition len(sorted_files) > self.keep_at_most remains True. If files are not being removed as expected, or if self.keep_at_most is set incorrectly, this can lead to an infinite loop and thus a recursion error.
To fix this issue, you need to ensure that each recursive call makes progress towards a base case where the recursion will stop. Here's a revised version of your method with a loop instead of recursion, which is generally safer for this kind of operation:
In this revised version:
The function uses a while True loop to continually check and remove the oldest file until the number of files is within the desired limit (self.keep_at_most). This approach avoids recursion, eliminating the risk of a stack overflow error. The loop will exit (break) when the condition len(sorted_files) <= self.keep_at_most is met, meaning the number of files is now within the limit, and no more files need to be removed. This should resolve the RecursionError and ensure that your function properly maintains the number of archive files within the specified limit.
Work effort: 1 hour