CPUQuota error

[Bartman0]

I configured the systemdspawner to use CPU limits.

c.SystemdSpawner.cpu_limit = 2.0

However I get this after logging in as a jupyterhub user:

CPU quota '200.0%' invalid.
Failed to create bus message: Invalid argument
[E 2017-09-10 22:59:38.791 JupyterHub user:251] Unhandled error starting rkooijman's server: Command '['/usr/bin/systemd-run', '--unit', 'jupyter-rkooijman-singleuser', '--uid', '1000', '--gid', '1000', '--property=PrivateTmp=yes', '--property=PrivateDevices=yes', '--setenv=PATH=/home/rkooijman/conda/bin:/usr/local/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin', '--setenv=LANG=en_US.UTF-8', '--setenv=JUPYTERHUB_API_TOKEN=918e8f94b54e4838bc4effe834fecb50', '--setenv=JPY_API_TOKEN=918e8f94b54e4838bc4effe834fecb50', '--setenv=MEM_LIMIT=2147483648', '--setenv=CPU_LIMIT=2.0', '--setenv=SHELL=/bin/bash', '--property=MemoryAccounting=yes', '--property=MemoryLimit=2147483648', '--property=CPUAccounting=yes', '--property=CPUQuota=200.0%', '--property=NoNewPrivileges=yes', '--property=ReadOnlyDirectories=-/', '--property=ReadWriteDirectories=/home/rkooijman', '/bin/bash', '-c', 'cd /home/rkooijman && exec jupyterhub-singleuser \'--user="rkooijman"\' \'--cookie-name="jupyter-hub-token-rkooijman"\' \'--base-url="/user/rkooijman"\' \'--hub-host=""\' \'--hub-prefix="/hub/"\' \'--hub-api-url="http://127.0.0.1:8081/hub/api"\' \'--ip="127.0.0.1"\' --port=34619']' returned non-zero exit status 1.

According to the systemd-run docs, this seems allright. But it fails nonetheless.

Any ideas?

[Bartman0]

I tried isolating the issue using simple tests:

root@ubuntu:~# systemd-run --property=CPUQuota=200.0% test CPU quota '200.0%' invalid. Failed to create bus message: Invalid argument root@ubuntu:~# systemd-run --property=CPUQuota='200.0%' test CPU quota '200.0%' invalid. Failed to create bus message: Invalid argument root@ubuntu:~# systemd-run --property=CPUQuota='200.0' test CPU quota '200.0' invalid. Failed to create bus message: Invalid argument root@ubuntu:~# systemd-run --property=CPUQuota='2' test CPU quota '2' invalid. Failed to create bus message: Invalid argument root@ubuntu:~# systemd-run --property=CPUQuota=2 test CPU quota '2' invalid. Failed to create bus message: Invalid argument

I have no idea what value is acceptable for CPU quota.

[astrodb]

I see the same error. Would be nice to know if it can be fixed.

[willingc]

Hi @Bartman0 and @astrodb,

Here's the section of the JupyterHub code that sets the CPU quota: https://github.com/jupyterhub/jupyterhub/blob/6fb31cc6138e784afb3eff204b27019ea3d0006c/jupyterhub/spawner.py#L453

I believe that this spawner would need some updates to support the CPU quota. Pinging @yuvipanda for verification or alternatives.

[ixjlyons]

Setting the CPU limit with this spawner does work (I have tested), but it appears that there may be some systemd versions that don't allow for non-integer quota percentages.

For example, the following:

systemd-run --property="CPUAccounting=yes" --property="CPUQuota=200.0%" echo

works for me with systemd 229 but not 234. Using --property="CPUQuota=200%" works fine in both cases.

[ixjlyons]

I tracked this down a bit more. Here are some notes:

• Here is the context in which this error occurs.
• This commit did a few things: 1. added some more robust parsing code, 2. switched from double to int for holding the percentage, and 3. broke support for percentages > 100%. This commit made it into systemd 231.
• This commit fixed the > 100% issue, but the value is still stored as an integer. This commit made it into systemd 232 and the behavior has not changed since.

To summarize, it seems to me then that systemd==231 does not support percentages > 100% at all, and that systemd>231 on does not support non-integer percentages.

Two simple approaches to fixing it for systemdspawner would be to convert to the nearest whole percentage when appending the --property=CPUQuota... argument, or to have systemdspawner users directly specify the integer percentage. The former maintains backward compatibility so it might be preferable.

edit: fixed link to the second referenced commit

[ixjlyons]

By the way @Bartman0 and @astrodb in the meantime, unless you're running systemd 231, you could set c.SystemdSpawner.cpu_limit = 2 (instead of 2.0) and it should work since '{}%'.format(2 * 100) == '200%'.

[willingc]

@ixjlyons Thanks so much for the helpful (and well documented) responses. :cake:

[yuvipanda]

w00t, thank you for the detailed digging, @ixjlyons!

To confirm, something like 150% will work, but 150.0% will not work. Is that accurate? If so I think having SystemdSpawner do that conversion explicitly seems like the right move.

[ixjlyons]

Yes, 150% should work. Except for systemd 231 which doesn't accept anything > 100%--I don't know how widespread 231 is but maybe adding a note in the README or the config description about it would be worthwhile.

[astrodb]

I've changed the setting to c.SystemdSpawner.cpu_limit = 2 add the error doesn't appear anymore. Thanks!

[astrodb]

Actually, I'm still seeing the error "CPU quota '200.0%' invalid" with that setting.

[ixjlyons]

Ah, yes the cpu_limit option is a float. Sorry, I guess there's not a workaround currently then. I can submit a PR later today.