HaveAGitGat
commented
1 month ago The point of the logs files is it contains enough info to diagnose issues. Maybe for your use case some of the stuff isn't needed, but when someone sends me a log file it's useful to have full context and all of it has come in useful at some point or another (hence it being there).
It's all relative. I have a few thousand job reports averaging around 10KB each with the largest being 700KB. If someone is transcoding a library of tens of thousands of videos, they most likely have tens of terrabytes of storage so a few GB for job reports is relatively little.
I don't think most people would like their original directories being filled up with thousands of job reports. At the moment all job reports are all stored in a single location which also makes it easy to search with the Report Searcher on the tools tab. Spreading the files all over the place would significantly slow down the search function and report cleaning. Also some people's work flows include deleting the original file/directory which in this case would delete the job reports too. So doing that makes things very messy.
The job reports are plain text files in server/Tdarr/DB2/JobReports, they don't use SQLite. You can change the size on the Options tab Total job history size limit (GB).
I don't think this makes a lot of sense as it would reduce performance and cause issues.
You could create a flow plugin which copies the job report from it's usual folder to next to the original file if it's really necessary.
The default config for log retention is 10GB, which seems like a lot of storage for log files. I have run into troubleshooting items and could not view a log file as it had already been pruned. Part of the cause is that pruning is turned on by default -this does make sense as you don't want log files filling up a drive, there needs to be some kind of cap.
But, a big part of this being pruned so early is down to how much data is stored in each of the log files. A lot of the content being irrelevant and not useful in debugging a flow. I'm all for detail in logs but there's a lot there that just serves no purpose.
Aside from optimizing the content of the logs I would also like to see an option where the log file could be stored with the resulting file in the original file location. At least then we can easily find the log file and we know that it will be there if it is needed. There could also be a function in the tools section to manually prune the log files from the library if one wanted to do that.
These log files could also be used by Tdarr to manage what files have gone through a flow already rather than keeping it in a DB. In the event that a DB corrupts (not out of the ordinary with sqlite) or is accidentally deleted, these files could act as a distributed DB instead.