JobDispatcher does not return all of the results

[lbabetto]

JobDispatcher does not return all of the results in a calculation.

MRE:

from jobdispatcher import JobDispatcher, Job

def myfunc(idx):
    return idx

job_list = [
    Job(name=f"{idx}", function=myfunc, arguments=[idx],  cores=1)
    for idx in range(1,11)
]

jd = JobDispatcher(job_list, maxcores=2)

results = jd.run()

print(results)

Expected output:

{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9, '10': 10}

Actual output:

{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}

Increasing the number of maxcores lowers the amount of returned data.

maxcore=2: {'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}

maxcore=3: {'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8}

Furthermore, the amount of returned output is not reproducible run-to-run. Running the same script with maxcore=8 back to back results in the following behaviour:

(compchem) lbabetto@TSP620:test$ python test.py
{'1': 1, '2': 2, '5': 5, '4': 4, '3': 3, '7': 7, '6': 6}
(compchem) lbabetto@TSP620:test$ python test.py
{'3': 3, '2': 2, '1': 1, '4': 4, '7': 7}
(compchem) lbabetto@TSP620:test$ python test.py
{'4': 4, '2': 2, '5': 5, '1': 1, '3': 3, '6': 6, '7': 7}
(compchem) lbabetto@TSP620:test$ python test.py
{'1': 1, '3': 3, '2': 2, '4': 4, '6': 6, '7': 7}
(compchem) lbabetto@TSP620:test$ python test.py
{'1': 1, '2': 2, '3': 3, '5': 5, '4': 4}
(compchem) lbabetto@TSP620:test$ python test.py
{'2': 2, '1': 1, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7}
(compchem) lbabetto@TSP620:test$ python test.py
{'1': 1, '2': 2, '4': 4, '5': 5, '3': 3, '6': 6}
(compchem) lbabetto@TSP620:test$ python test.py
{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5}
(compchem) lbabetto@TSP620:test$ python test.py
{'1': 1, '3': 3, '2': 2, '4': 4}
(compchem) lbabetto@TSP620:test$ python test.py
{'1': 1, '2': 2, '3': 3, '4': 4}

This is a critical bug :(

[lbabetto]

Solved by #31