fpsync: start all idle workers before refresh

[jcphill]

There is no reason to call work_list_refresh after every job start; only call it on loop iterations when more idle workers are needed.

[martymac]

Hello Jim,

Thanks for your patch.

Function work_list_refresh() updates ${WORK_NUM}.

Your patch would lead to updating ${WORK_NUM} only when ${WORK_NUM} >= ${OPT_JOBS} but that is not always the case. We need to detect finished jobs when ${WORK_NUM} < ${OPT_JOBS} too. I can see two cases :

• when we are emptying the queue and processing the last jobs
• when we initially pass a number of workers greater than the total number of jobs to process

With your patch, when there are more workers available than jobs, you skip detecting finished jobs and last ${OPT_JOBS} will silently terminate without being detected.

[jcphill]

Thank you for your reply. Note that the loop exits based on line 918: while [ "${_NEXT}" != "fp_done" ] && [ "${_NEXT}" != "sl_stop" ] so work_list_refresh() is not called more than once after the job_queue_next() returns fp_done or sl_stop, and instead wait is called at line 954 to detect when the remaining jobs have finished. ${WORK_NUM} and ${WORK_LIST} are not used later.

[martymac]

Hello again,

Good catch ! I have overlooked my code, sorry. So everything seems good to me :)

Thanks for your contribution !

Best regards,

Ganael.

[jcphill]

Thanks. Since work_list_refresh() is O(WORK_NUM) it could be a bottleneck when called after every job launch and not all workers were kept busy. With this change I see a substantial performance increase with 80 workers.

[martymac]

That's right! Thanks again for your feedback!