Closed lbabetto closed 2 years ago
JobDispatcher does not return all of the results in a calculation.
MRE:
from jobdispatcher import JobDispatcher, Job def myfunc(idx): return idx job_list = [ Job(name=f"{idx}", function=myfunc, arguments=[idx], cores=1) for idx in range(1,11) ] jd = JobDispatcher(job_list, maxcores=2) results = jd.run() print(results)
Expected output:
{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9, '10': 10}
Actual output:
{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}
Increasing the number of maxcores lowers the amount of returned data.
maxcores
maxcore=2: {'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}
maxcore=2
maxcore=3: {'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8}
maxcore=3
{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8}
Furthermore, the amount of returned output is not reproducible run-to-run. Running the same script with maxcore=8 back to back results in the following behaviour:
maxcore=8
(compchem) lbabetto@TSP620:test$ python test.py {'1': 1, '2': 2, '5': 5, '4': 4, '3': 3, '7': 7, '6': 6} (compchem) lbabetto@TSP620:test$ python test.py {'3': 3, '2': 2, '1': 1, '4': 4, '7': 7} (compchem) lbabetto@TSP620:test$ python test.py {'4': 4, '2': 2, '5': 5, '1': 1, '3': 3, '6': 6, '7': 7} (compchem) lbabetto@TSP620:test$ python test.py {'1': 1, '3': 3, '2': 2, '4': 4, '6': 6, '7': 7} (compchem) lbabetto@TSP620:test$ python test.py {'1': 1, '2': 2, '3': 3, '5': 5, '4': 4} (compchem) lbabetto@TSP620:test$ python test.py {'2': 2, '1': 1, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7} (compchem) lbabetto@TSP620:test$ python test.py {'1': 1, '2': 2, '4': 4, '5': 5, '3': 3, '6': 6} (compchem) lbabetto@TSP620:test$ python test.py {'1': 1, '2': 2, '3': 3, '4': 4, '5': 5} (compchem) lbabetto@TSP620:test$ python test.py {'1': 1, '3': 3, '2': 2, '4': 4} (compchem) lbabetto@TSP620:test$ python test.py {'1': 1, '2': 2, '3': 3, '4': 4}
This is a critical bug :(
Solved by #31
JobDispatcher does not return all of the results in a calculation.
MRE:
Expected output:
{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9, '10': 10}
Actual output:
{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}
Increasing the number of
maxcores
lowers the amount of returned data.maxcore=2
:{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8, '9': 9}
maxcore=3
:{'1': 1, '2': 2, '3': 3, '4': 4, '5': 5, '6': 6, '7': 7, '8': 8}
Furthermore, the amount of returned output is not reproducible run-to-run. Running the same script with
maxcore=8
back to back results in the following behaviour:This is a critical bug :(