Python variables are scoped by function. This meant that in the
loop submitting jobs to the executor the value of the variable
"query_copy" would have an unexpected value if the jobs were ever
blocked. This would happen after 140 * 50 = 7000 observations, as
the executor would be out of workers by then.
Now we create all queries separately and put them in a list, which
the jobs can pop from one at a time (lists are thread-safe in
Python). This should solve the problem.
Python variables are scoped by function. This meant that in the loop submitting jobs to the executor the value of the variable "query_copy" would have an unexpected value if the jobs were ever blocked. This would happen after 140 * 50 = 7000 observations, as the executor would be out of workers by then.
Now we create all queries separately and put them in a list, which the jobs can pop from one at a time (lists are thread-safe in Python). This should solve the problem.