Doesn't work with systemd 219

[KhoaLeTea]

In the start() function you use the command '/usr/bin/systemd-run', but I'm getting a CalledProcessError from Python. I'm not sure if I read the code wrong, but it seems that systemd-run was passed arguments, but you never passed in the service that you wanted run. So I believe you want to do a "systemd-run jupyterhub" if I understand this correctly.

[yuvipanda]

Hello! Can you paste the entire stacktrace for the CalledProcessError you were getting? systemd-run is passed a list of arguments to start, including path to the jupyter-singleuser script...

[KhoaLeTea]

Here you go. I'm running it with no extra options.

Unknown assignment WorkingDirectory=/home/philliple. Failed to create bus message: No such device or address [E 2016-10-12 10:33:37.618 JupyterHub user:244] Unhandled error starting philliple's server: Command '['/usr/bin/sudo', '/usr/bin/systemd-run', '--unit', 'jupyter-philliple-singleuser', '--uid', '1001', '--gid', '1001', '--setenv=LANG=en_US.utf8', '--setenv=JPY_API_TOKEN=32ed5ad4af764767b64886e2c9da2354', '--setenv=PATH=/home/philliple/.nvm/versions/node/v6.5.0/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/home/philliple/.local/bin:/home/philliple/bin', '--property=WorkingDirectory=/home/philliple', '--setenv=SHELL=/bin/bash', '--property=MemoryAccounting=yes', '--property=MemoryLimit=90%', 'jupyterhub-singleuser', '--user=philliple', '--cookie-name=jupyter-hub-token-philliple', '--base-url=/user/philliple', '--hub-host=', '--hub-prefix=/hub/', '--hub-api-url=http://127.0.0.1:8081/hub/api', '--ip=127.0.0.1', '--port=56214']' returned non-zero exit status 1 [E 2016-10-12 10:33:37.705 JupyterHub web:1548] Uncaught exception POST /hub/login?next= (127.0.0.1) HTTPServerRequest(protocol='http', host='localhost:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Cache-Control': 'max-age=0', 'Upgrade-Insecure-Requests': '1', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8', 'Content-Type': 'application/x-www-form-urlencoded', 'Referer': 'http://localhost:8000/hub/login', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36', 'Origin': 'http://localhost:8000', 'Content-Length': '43', 'Host': 'localhost:8000', 'Accept-Language': 'en-US,en;q=0.8', 'Connection': 'close'}) Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/tornado/web.py", line 1469, in _execute result = yield result File "/home/philliple/jupyterhub/jupyterhub/jupyterhub/handlers/login.py", line 84, in post yield self.spawn_single_user(user) File "/home/philliple/jupyterhub/jupyterhub/jupyterhub/handlers/base.py", line 326, in spawn_single_user yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f) File "/home/philliple/jupyterhub/jupyterhub/jupyterhub/user.py", line 254, in spawn raise e File "/home/philliple/jupyterhub/jupyterhub/jupyterhub/user.py", line 228, in spawn ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f) File "/usr/lib/python3.4/site-packages/systemdspawner/systemdspawner.py", line 188, in start subprocess.check_output(cmd) File "/usr/lib64/python3.4/subprocess.py", line 620, in check_output raise CalledProcessError(retcode, process.args, output=output) subprocess.CalledProcessError: Command '['/usr/bin/sudo', '/usr/bin/systemd-run', '--unit', 'jupyter-philliple-singleuser', '--uid', '1001', '--gid', '1001', '--setenv=LANG=en_US.utf8', '--setenv=JPY_API_TOKEN=32ed5ad4af764767b64886e2c9da2354', '--setenv=PATH=/home/philliple/.nvm/versions/node/v6.5.0/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/home/philliple/.local/bin:/home/philliple/bin', '--property=WorkingDirectory=/home/philliple', '--setenv=SHELL=/bin/bash', '--property=MemoryAccounting=yes', '--property=MemoryLimit=90%', 'jupyterhub-singleuser', '--user=philliple', '--cookie-name=jupyter-hub-token-philliple', '--base-url=/user/philliple', '--hub-host=', '--hub-prefix=/hub/', '--hub-api-url=http://127.0.0.1:8081/hub/api', '--ip=127.0.0.1', '--port=56214']' returned non-zero exit status

[yuvipanda]

Aaah, interesting. The issue is Unknown assignment WorkingDirectory=/home/ philliple..

Can you tell me what operating system / distro you are using? And the systemd version as well? (You can get it with systemd-run --version)

On Wed, Oct 12, 2016 at 9:35 AM, Khoa notifications@github.com wrote:

Here you go. I'm running it with no extra options.

Unknown assignment WorkingDirectory=/home/philliple. Failed to create bus message: No such device or address [E 2016-10-12 10:33:37.618 JupyterHub user:244] Unhandled error starting philliple's server: Command '['/usr/bin/sudo', '/usr/bin/systemd-run', '--unit', 'jupyter-philliple-singleuser', '--uid', '1001', '--gid', '1001', '--setenv=LANG=en_US.utf8', '--setenv=JPY_API_TOKEN= 32ed5ad4af764767b64886e2c9da2354', '--setenv=PATH=/home/ philliple/.nvm/versions/node/v6.5.0/bin:/usr/lib64/qt-3.3/ bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/ sbin:/home/philliple/.local/bin:/home/philliple/bin', '--property=WorkingDirectory=/home/philliple', '--setenv=SHELL=/bin/bash', '--property=MemoryAccounting=yes', '--property=MemoryLimit=90%', 'jupyterhub-singleuser', '--user=philliple', '--cookie-name=jupyter-hub-token-philliple', '--base-url=/user/philliple', '--hub-host=', '--hub-prefix=/hub/', '--hub-api-url=http://127.0.0.1:8081/hub/api', '--ip=127.0.0.1', '--port=56214']' returned non-zero exit status 1 [E 2016-10-12 10:33:37.705 JupyterHub web:1548] Uncaught exception POST /hub/login?next= (127.0.0.1) HTTPServerRequest(protocol='http', host='localhost:8000', method='POST', uri='/hub/login?next=', version='HTTP/1.1', remoteip='127.0.0.1', headers={'Cache-Control': 'max-age=0', 'Upgrade-Insecure-Requests': '1', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,/_;q=0.8', 'Content-Type': 'application/x-www-form-urlencoded', 'Referer': ' http://localhost:8000/hub/login', 'Accept-Encoding': 'gzip, deflate', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36', 'Origin': ' http://localhost:8000', 'Content-Length': '43', 'Host': 'localhost:8000', 'Accept-Language': 'en-US,en;q=0.8', 'Connection': 'close'}) Traceback (most recent call last): File "/usr/lib64/python3.4/site-packages/tornado/web.py", line 1469, in _execute result = yield result File "/home/philliple/jupyterhub/jupyterhub/jupyterhub/handlers/login.py", line 84, in post yield self.spawn_single_user(user) File "/home/philliple/jupyterhub/jupyterhub/jupyterhub/handlers/base.py", line 326, in spawn_single_user yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), f) File "/home/philliple/jupyterhub/jupyterhub/jupyterhub/user.py", line 254, in spawn raise e File "/home/philliple/jupyterhub/jupyterhub/jupyterhub/user.py", line 228, in spawn ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f) File "/usr/lib/python3.4/site-packages/systemdspawner/systemdspawner.py", line 188, in start subprocess.check_output(cmd) File "/usr/lib64/python3.4/subprocess.py", line 620, in check_output raise CalledProcessError(retcode, process.args, output=output) subprocess.CalledProcessError: Command '['/usr/bin/sudo', '/usr/bin/systemd-run', '--unit', 'jupyter-philliple-singleuser', '--uid', '1001', '--gid', '1001', '--setenv=LANG=en_US.utf8', '--setenv=JPY_API_TOKEN=32ed5ad4af764767b64886e2c9da2354', '--setenv=PATH=/home/philliple/.nvm/versions/node/ v6.5.0/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/ sbin:/usr/bin:/usr/sbin:/bin:/sbin:/home/philliple/.local/bin:/home/philliple/bin', '--property=WorkingDirectory=/home/philliple', '--setenv=SHELL=/bin/bash', '--property=MemoryAccounting=yes', '--property=MemoryLimit=90%', 'jupyterhub-singleuser', '--user=philliple', '--cookie-name=jupyter-hub-token-philliple', '--base-url=/user/philliple', '--hub-host=', '--hub-prefix=/hub/', '--hub-api-url=http://127.0.0.1:8081/hub/api', '--ip=127.0.0.1', '--port=56214']' returned non-zero exit status

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jupyterhub/systemdspawner/issues/9#issuecomment-253266529, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB23h6VBh3rctSwMpd_bldAi8TmZp66ks5qzQxggaJpZM4KM1yK .

Yuvi Panda T http://yuvi.in/blog

[KhoaLeTea]

I'm on CentOS 7 right now. I'm running systemd 219

[yuvipanda]

Looks like systemd 219 doesn't support setting WorkingDirectory on transient units via systemd-run. I'll try to find a workaround for this soon

[KhoaLeTea]

Awesome thanks!

[yuvipanda]

@Amerterasu can you test it with this branch: https://github.com/jupyterhub/systemdspawner/tree/centos-fix and let me know if that fixes your issue?

[KhoaLeTea]

It spawned it perfectly. I'm curious though, what does changing it to quote do?

[yuvipanda]

The primary change is to use bash to run a 'cd' before spawning the notebook, since the WorkingDirectory systemd property wasn't available for that version of systemd. https://github.com/jupyterhub/systemdspawner/commit/a184f690da76bf63a71475ae4a43a83c1736e4ec is the meaty change :)

I'll merge it to master shortly!

[KhoaLeTea]

Thank you so much!

[yuvipanda]

Done! \o/ Thanks for testing it on Centos and helping me debug! <3

[lachlansimpson]

If I'm still seeing this issue - CentOS 7, python3.6, jupyterhub 0.9.2 - what should I do?

Unknown assignment WorkingDirectory=/home/ladmin.
Failed to create bus message: No such device or address

I'm not seeing change a184f69 in systemdspawner/systemdspawner.py either.

[lachlansimpson]

Oh! I see that it's moved into systemdspawner/systemd.py Still seeing this issue though.

[adsche]

@datakid I had the same issue. If you comment out (i.e., prefix with #) line 51 in file systemdspawner/systemd.py it works on version 219.

Right now, it seems to use both ways, the bash way and the WorkingDirectory property which won't work on 219.

[lachlansimpson]

I'll give it a try - thanks

[lachlansimpson]

Nope, still got the same result?

Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/lib64/python3.6/asyncio/base_events.py", line 1434, in _run_once
Oct 24 08:42:02 compute39 jupyterhub:    handle._run()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/lib64/python3.6/asyncio/events.py", line 145, in _run
Oct 24 08:42:02 compute39 jupyterhub:    self._callback(*self._args)
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/app.py", line 1956, in launch_instance_async
Oct 24 08:42:02 compute39 jupyterhub:    await self.start()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/app.py", line 1865, in start
Oct 24 08:42:02 compute39 jupyterhub:    await self.proxy.start()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/proxy.py", line 535, in start
Oct 24 08:42:02 compute39 jupyterhub:    self._check_previous_process()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/proxy.py", line 484, in _check_previous_process
Oct 24 08:42:02 compute39 jupyterhub:    self.log.warning("Proxy no longer running at pid=%s", pid)
Oct 24 08:42:02 compute39 jupyterhub: Message: 'Proxy no longer running at pid=%s'
Oct 24 08:42:02 compute39 jupyterhub: Arguments: (1112,)
Oct 24 08:42:02 compute39 jupyterhub: --- Logging error ---
Oct 24 08:42:02 compute39 jupyterhub: Traceback (most recent call last):
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/app.py", line 1865, in start
Oct 24 08:42:02 compute39 jupyterhub:    await self.proxy.start()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/proxy.py", line 601, in start
Oct 24 08:42:02 compute39 jupyterhub:    await server.wait_up(1)
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/utils.py", line 165, in wait_for_server
Oct 24 08:42:02 compute39 jupyterhub:    timeout=timeout
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff
Oct 24 08:42:02 compute39 jupyterhub:    raise TimeoutError(fail_message)
Oct 24 08:42:02 compute39 jupyterhub: TimeoutError: Server at 10.126.17.229:443 didn't respond in 1 seconds
Oct 24 08:42:02 compute39 jupyterhub: During handling of the above exception, another exception occurred:
Oct 24 08:42:02 compute39 jupyterhub: Traceback (most recent call last):
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/lib64/python3.6/logging/__init__.py", line 993, in emit
Oct 24 08:42:02 compute39 jupyterhub:    msg = self.format(record)
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/lib64/python3.6/logging/__init__.py", line 839, in format
Oct 24 08:42:02 compute39 jupyterhub:    return fmt.format(record)
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib64/python3.6/site-packages/tornado/log.py", line 193, in format
Oct 24 08:42:02 compute39 jupyterhub:    formatted = self._fmt % record.__dict__
Oct 24 08:42:02 compute39 jupyterhub: KeyError: 'highlevel'
Oct 24 08:42:02 compute39 jupyterhub: Call stack:
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/bin/jupyterhub", line 4, in <module>
Oct 24 08:42:02 compute39 jupyterhub:    main()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/app.py", line 1968, in launch_instance
Oct 24 08:42:02 compute39 jupyterhub:    loop.start()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib64/python3.6/site-packages/tornado/platform/asyncio.py", line 132, in start
Oct 24 08:42:02 compute39 jupyterhub:    self.asyncio_loop.run_forever()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/lib64/python3.6/asyncio/base_events.py", line 422, in run_forever
Oct 24 08:42:02 compute39 jupyterhub:    self._run_once()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/lib64/python3.6/asyncio/base_events.py", line 1434, in _run_once
Oct 24 08:42:02 compute39 jupyterhub:    handle._run()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/lib64/python3.6/asyncio/events.py", line 145, in _run
Oct 24 08:42:02 compute39 jupyterhub:    self._callback(*self._args)
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/app.py", line 1956, in launch_instance_async
Oct 24 08:42:02 compute39 jupyterhub:    await self.start()
Oct 24 08:42:02 compute39 jupyterhub:  File "/usr/local/lib/python3.6/site-packages/jupyterhub/app.py", line 1867, in start
Oct 24 08:42:02 compute39 jupyterhub:    self.log.critical("Failed to start proxy", exc_info=True)
Oct 24 08:42:02 compute39 jupyterhub: Message: 'Failed to start proxy'
Oct 24 08:42:02 compute39 jupyterhub: Arguments: ()
Oct 24 08:42:02 compute39 systemd: jupyterhub.service: Main process exited, code=exited, status=1/FAILURE
Oct 24 08:42:02 compute39 systemd: jupyterhub.service: Unit entered failed state.
Oct 24 08:42:02 compute39 audit: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=jupyterhub comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Oct 24 08:42:02 compute39 systemd: jupyterhub.service: Failed with result 'exit-code'.
Oct 24 08:52:12 compute39 systemd: Starting Cleanup of Temporary Directories...
Oct 24 08:52:12 compute39 systemd: Started Cleanup of Temporary Directories.
Oct 24 08:52:12 compute39 audit: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-tmpfiles-clean comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'

[hansen-m]

Same issue with RHEL 7 (systemd 219) and the fix from @adsche for commenting out line 51 worked for me.

jupyterhub-systemdspawner version 0.11

@yuvipanda

[meeseeksmachine]

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/unable-to-use-systemdspawner-on-centos-7/8057/3