search

BOINC / boinc

Open-source software for volunteer computing and grid computing.
https://boinc.berkeley.edu
GNU Lesser General Public License v3.0
2.02k stars 446 forks source link

Improve the efficiency/scalabilty of submit_rpc_handler.php #2791

Closed lfield closed 2 months ago

lfield commented 6 years ago

When submitting 30K jobs, the submit_rpc_handler.php exceeded the default memory limit of 128M on line 230. The XML file is only 8.7MB in size.

lfield commented 6 years ago

As an aside using create work and passing command line arguments does not seem like a good idea to me. At least on our target machine the maximum number of arguments is O(2M).

davidpanderson commented 5 years ago

It doesn't use command line arguments.

lfield commented 5 years ago

It uses the command line in the function _submitjobs

https://github.com/BOINC/boinc/blob/master/html/user/submit_rpc_handler.php#L204 https://github.com/BOINC/boinc/blob/master/html/user/submit_rpc_handler.php#L211 https://github.com/BOINC/boinc/blob/master/html/user/submit_rpc_handler.php#L248 https://github.com/BOINC/boinc/blob/master/html/user/submit_rpc_handler.php#L278

In any case this would be a big change. This issue is about the memory efficiency of handling a large submission.

davidpanderson commented 5 years ago

Per-job parameters are not passed to create_work on the command line; they're passed via stdin.

davidpanderson commented 2 months ago

After the initial report I increased the script's mem limit to 4GB. Did that solve the problem?