search

guidopetri / chess-pipeline

Pulling games from the Lichess API into a PostgreSQL database for data analysis.
GNU General Public License v3.0
20 stars 2 forks source link

Add perftype to filenames to prevent unfulfilled dependency errors #61

Closed guidopetri closed 3 years ago

guidopetri commented 3 years ago

For some time now, I've been getting spurious "unfulfilled dependency at runtime" errors, and I think I've finally figured it out: A task starts because its self.complete function returns False, but then during the runtime of that task, another task starts that writes to the exact same file, and thus makes the first task's self.complete function return True.

I think the above is the case, but I need to investigate closer. If so, the culprit is not adding perftype to the filenames that all tasks write to - I run both blitz and bullet for a handful of users, and if one of them contains more games than the other, it'll "run over" to the next task's allotted time, thus creating the above error.

guidopetri commented 3 years ago

Obviously, if the entire pipeline runs at the same time that another instance of the pipeline is running, then the temp files will get purged... which breaks the longer-running pipeline.

guidopetri commented 3 years ago

Hopefully closed with 7f84fb3 . This commit makes a CopyWrapper only remove the targets that are involved in its completion, not completely clean the temp files.

guidopetri commented 3 years ago

(Testing this tonight by playing some bullet games!)

guidopetri commented 3 years ago

I neglected the fact that PostgresTargets won't have a .path attribute. Fixed this in f4993ea .

I also neglected the fact that CopyWrapper's inputs are all PostgresTargets, and I should actually be looking at the jobs list itself and iterating over the functions there. Fixed in 34fce4d .

Hopefully this is the actual end of this issue's saga. The lesson: I need a better testing, probably through a staging/test environment.

guidopetri commented 3 years ago

Alright, now I think I've actually fixed it. The method is: look at the job list, iterate over each function and (recursively) get the local files. This is finished with commit ab6728c .