Closed ctengel closed 3 years ago
ability to combine this with other log/retries (see journal)
Stick with flat files now for output - DB later
Next steps:
don't count an ignoreerrors #16 run as "successful"
From #4:
Cause #15 and related tools to output backend for said file
i.e. output backends where relevant into retry/refresh files etc and definetely csvs
look through attempt files /logs.... a. look at url and find urls that aren't in DB summary (either vid URL, playlist URL, etc) b. deeper analysis of data...reconstruct playlists, etc
worker logs a. quieter b. success dl vs failed
flask logs anything???
then we need to somehow dump all this in a way that it can be stored by higher level maths