search

jupyterhub / yarnspawner

Spawn JupyterHub single user notebook servers in Hadoop/YARN containers.
https://jupyterhub-yarnspawner.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
19 stars 16 forks source link

error when configuring the yarnspawner for a cluster #5

Closed jrcblue closed 5 years ago

jrcblue commented 5 years ago

According to the document of the yarnspawner, we created and packed the environment via conda and applied to a cluster with hadoop. Also,we configured the jupyterhub_config.py following the example. However, when a user log in, it comes to "internal server error" and ：

[E 2019-03-06 13:32:19.652 JupyterHub base:1001] Preventing implicit spawn for jrc because last spawn failed: 'source' must be a string [E 2019-03-06 13:32:19.652 JupyterHub web:1788] Uncaught exception GET /hub/user/jrc/ (127.0.0.1) ........ ........ TypeError: 'source' must be a string But we believe the source is a string indeed and we tried configuring the environment and spawner several times. I appreciate it if you can share some suggestion/solutions with us. Also, maybe this is a bug of the spawner. Hope to hear back from you and thanks!

Our jupyterhub_config.py is shown as follows:

c.Authenticator.whitelist = {'jrc'} c.Authenticator.admin_users = {'jrc'}

c.JupyterHub.spawner_class = 'yarnspawner.YarnSpawner'

c.YarnSpawner.mem_limit = '2 G' c.YarnSpawner.cpu_limit =1

c.YarnSpawner.queue ='default'

c.YarnSpawner.localize_files = { 'environment': { 'source':'hdfs:///jupyterhub/conda-envs/example-env.tar.gz', 'visibility': 'public' } } c.YarnSpawner.prologue = 'source example-env/bin/activate'

jcrist commented 5 years ago

This is a duplicate of https://github.com/jcrist/skein/issues/142 - the documentation is for git master, which supports dicts for values in localize_files, while the current release only supports strings or skein.File objects. See https://github.com/jcrist/skein/issues/142#issuecomment-461485595 for more information.

I'll try to get a new release out today or tomorrow to prevent this from happening to others.

jrcblue commented 5 years ago

Hello,

Thank you for your reply! I installed the new version yarnspawner. However there is still some other confusing error. I am confused why Unable to connect to driver and permission denied. Thank you very much

The following is my proxy configuration and I configured the environment:

hadoop.proxyuser.jrc.hosts * hadoop.proxyuser.jrc.groups *

hdfs dfsadmin -refreshSuperUserGroupsConfiguration

yarn rmadmin –refreshSuperUserGroupsConfiguration

root@xxx:/opt/work/hadoop-2.7.2/etc/hadoop# hdfs dfsadmin -refreshSuperUserGroupsConfiguration

Refresh super user groups configuration successful

root@xxx:/opt/work/hadoop-2.7.2/etc/hadoop# yarn rmadmin -refreshSuperUserGroupsConfiguration

19/03/07 14:20:01 INFO client.RMProxy: Connecting to ResourceManager at xxx/172.168.2.165:8033

export CONDA_HOME=/home/jrc/miniconda3

export PATH=：/opt/work/hadoop-2.7.2/bin：${CONDA_HOME}/bin:${PATH}

export HADOOP_HOME=/opt/work/hadoop-2.7.2

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/Hadoop

The error is shown as follows:

(jh4y) jrc@Gondolin-Node-065:~$ jupyterhub –debug

[I 2019-03-07 14:27:35.819 JupyterHub base:499] User logged in: jrc

[I 2019-03-07 14:27:35.820 JupyterHub log:158] 302 POST /hub/login?next= -> /user/jrc/ (jrc@10.239.196.34) 334.27ms

[I 2019-03-07 14:27:35.866 JupyterHub log:158] 302 GET /user/jrc/ -> /hub/user/jrc/ (@10.239.196.34) 1.40ms

[D 2019-03-07 14:27:35.965 JupyterHub base:603] Initiating spawn for jrc

[D 2019-03-07 14:27:35.965 JupyterHub base:609] 0/100 concurrent spawns

[D 2019-03-07 14:27:35.966 JupyterHub base:612] 0 active servers

Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.2.0-35-generic x86_64)

sh: 1: cannot create /run/motd.dynamic.new: Permission denied

[W 2019-03-07 14:27:36.266 JupyterHub spawner:696] Setting port from user.server is deprecated as of JupyterHub 0.7.

19/03/07 14:27:38 INFO client.RMProxy: Connecting to ResourceManager at xxx/172.168.2.165:8032

19/03/07 14:27:38 INFO skein.Driver: Driver started, listening on 33443

19/03/07 14:27:38 INFO skein.Driver: Driver shut down

[E 2019-03-07 14:27:39.187 JupyterHub user:477] Unhandled error starting jrc's server: Unable to connect to driver

19/03/07 14:27:40 INFO client.RMProxy: Connecting to ResourceManager at xxx/172.168.2.165:8032

19/03/07 14:27:41 INFO skein.Driver: Driver started, listening on 34711

19/03/07 14:27:41 INFO skein.Driver: Driver shut down

[E 2019-03-07 14:27:41.895 JupyterHub user:486] Failed to cleanup jrc's server that failed to start

Traceback (most recent call last):

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/user.py", line 482, in spawn

    await self.stop()

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/user.py", line 554, in stop

    await spawner.stop()

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/yarnspawner/spawner.py", line 266, in stop

    client = await self._get_client()

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/yarnspawner/spawner.py", line 146, in _get_client

    None, lambda: skein.Client(**kwargs)

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/concurrent/futures/thread.py", line 56, in run

    result = self.fn(*self.args, **self.kwargs)

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/yarnspawner/spawner.py", line 146, in <lambda>

    None, lambda: skein.Client(**kwargs)

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 346, in __init__

    self._call('ping', proto.Empty())

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 270, in _call

    raise ConnectionError("Unable to connect to %s" % self._server_name)

skein.exceptions.ConnectionError: Unable to connect to driver

[E 2019-03-07 14:27:41.899 JupyterHub web:1788] Uncaught exception GET /hub/user/jrc/ (10.239.196.34)

HTTPServerRequest(protocol='http', host='172.168.2.165:8000', method='GET', uri='/hub/user/jrc/', version='HTTP/1.1', remote_ip='10.239.196.34')

Traceback (most recent call last):

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/tornado/web.py", line 1699, in _execute

    result = await result

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 1052, in get

    await self.spawn_single_user(user)

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 705, in spawn_single_user

    timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn

    await spawn_future

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/user.py", line 489, in spawn

    raise e

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/user.py", line 409, in spawn

    url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/yarnspawner/spawner.py", line 210, in start

    client = await self._get_client()

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/yarnspawner/spawner.py", line 146, in _get_client

    None, lambda: skein.Client(**kwargs)

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/concurrent/futures/thread.py", line 56, in run

    result = self.fn(*self.args, **self.kwargs)

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/yarnspawner/spawner.py", line 146, in <lambda>

    None, lambda: skein.Client(**kwargs)

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 346, in __init__

    self._call('ping', proto.Empty())

  File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 270, in _call

    raise ConnectionError("Unable to connect to %s" % self._server_name)

skein.exceptions.ConnectionError: Unable to connect to driver

[D 2019-03-07 14:27:41.904 JupyterHub base:880] No template for 500

[E 2019-03-07 14:27:41.920 JupyterHub log:150] {

  "Connection": "close",

  "Cache-Control": "max-age=0",

  "X-Forwarded-For": "10.239.196.34",

  "Via": "1.1 Gondolin-Gateway (squid/3.3.8)",

  "Cookie": "jupyterhub-hub-login=\"2|1:0|10:1551940055|20:jupyterhub-hub-login|44:ZGZkYmVmYjQxMDI2NDAyYTg5NjU3MTNiNGIzMjcyZWY=|d65a107df85b403b81d800eb5963095fc455696184f5bb5ce2e5274baf7d69f5\"; _xsrf=2|a71cc953|ac1e49181f83bdfcf5839a04a4a3c048|1551840886; jupyterhub-session-id=ff249360bbbf43a19b89a28c1e63107a",

  "Accept-Language": "en-US,en;q=0.9,zh-CN;q=0.8,zh-TW;q=0.7,zh;q=0.6",

  "Accept-Encoding": "gzip, deflate",

  "Referer": "http://172.168.2.165:8000/hub/login",

  "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",

  "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36",

  "Upgrade-Insecure-Requests": "1",

  "Host": "172.168.2.165:8000"

}

[E 2019-03-07 14:27:41.920 JupyterHub log:158] 500 GET /hub/user/jrc/ (jrc@10.239.196.34) 6016.24ms

-----Original Messages----- From:"Jim Crist" notifications@github.com Sent Time:2019-03-06 23:46:48 (Wednesday) To: jcrist/yarnspawner yarnspawner@noreply.github.com Cc: RCJiao 3150104926@zju.edu.cn, Author author@noreply.github.com Subject: Re: [jcrist/yarnspawner] error when configuring the yarnspawner for a cluster (#5)

This is a duplicate of jcrist/skein#142 - the documentation is for git master, which supports dicts for values in localize_files, while the current release only supports strings or skein.File objects. See jcrist/skein#142 (comment) for more information.

I'll try to get a new release out today or tomorrow to prevent this from happening to others.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Best wishes!

Ruochen Jiao

Chu Kochen Honors College,Zhejiang University 3150104926@zju.edu.cn

jcrist commented 5 years ago

Hmmm, I'm not sure what that issue is. It looks like the skein driver is starting but then failing to connect. To help debug I'll need more information.

What is the output of?

$ SKEIN_LOG_LEVEL=debug skein application ls

Also, when posting things like code or tracebacks, its generally nice to surround them in markup (like I've done above) so they display nicely. For more information on how to do this see https://guides.github.com/features/mastering-markdown/.

jrcblue commented 5 years ago

Thank you for your reply. I tried the command:

$ SKEIN_LOG_LEVEL=debug skein application ls and the output is:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38

Unexpected Error: Traceback (most recent call last): File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/cli.py", line 106, in get_driver return Client.from_global_driver() File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 359, in from_global_driver raise DriverNotRunningError("No driver currently running") skein.exceptions.DriverNotRunningError: No driver currently running

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/cli.py", line 426, in main func(kwargs) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/cli.py", line 312, in application_ls apps = get_driver().get_applications(states=state) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/cli.py", line 108, in get_driver return Client() File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 334, in init java_options=java_options) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 200, in _start_driver classpath = (subprocess.check_output(['yarn', 'classpath', '--glob']) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/subprocess.py", line 336, in check_output kwargs).stdout File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/subprocess.py", line 403, in run with Popen(*popenargs, **kwargs) as process: File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/subprocess.py", line 709, in init restore_signals, start_new_session) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/subprocess.py", line 1344, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: 'yarn': 'yarn'

Exception ignored in: <object repr() failed> Traceback (most recent call last): File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 472, in del self.close() File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 461, in close if self._proc is not None: AttributeError: _proc |

Also, there is always a error indicating no such file or dictionary 'yarn': 'yarn' ,but the instruction doesn't mention I need to configure this.

Thank you!

-----Original Messages----- From:"Jim Crist" notifications@github.com Sent Time:2019-03-07 15:29:18 (Thursday) To: jcrist/yarnspawner yarnspawner@noreply.github.com Cc: RCJiao 3150104926@zju.edu.cn, Author author@noreply.github.com Subject: Re: [jcrist/yarnspawner] error when configuring the yarnspawner for a cluster (#5)

Hmmm, I'm not sure what that issue is. It looks like the skein driver is starting but then failing to connect. To help debug I'll need more information.

What is the output of?

$ SKEIN_LOG_LEVEL=debug skein application ls

Also, when posting things like code or tracebacks, its generally nice to surround them in markup (like I've done above) so they display nicely. For more information on how to do this see https://guides.github.com/features/mastering-markdown/.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Best wishes!

Ruochen Jiao

Chu Kochen Honors College,Zhejiang University 3150104926@zju.edu.cn

jcrist commented 5 years ago

Looks like the yarn commandline application isn't on $PATH, which is causing the daemon startup to fail. I'm not sure what's up with your hadoop installation, but the yarn CLI comes standard and is usually available. You likely need to add the hadoop /bin directory to your $PATH.

jrcblue commented 5 years ago

yes,I added the yarn commandline to $PATH. And it looks close to working. Now the output of $ SKEIN_LOG_LEVEL=debug skein application ls is :

1 2 3 4 5 6 7 8

19/03/08 16:47:43 DEBUG skein.Driver: Logging in using ticket cache 19/03/08 16:47:44 INFO client.RMProxy: Connecting to ResourceManager at Gondolin-Node-065/172.168.2.165:8032 19/03/08 16:47:45 INFO skein.Driver: Driver started, listening on 45627 19/03/08 16:47:45 DEBUG skein.Driver: Reporting gRPC server port back to the launching process E0308 16:47:45.315065977 10280 http_proxy.cc:62] 'https' scheme not supported in proxy URI 19/03/08 16:47:45 DEBUG skein.Driver: Starting process disconnected, shutting down 19/03/08 16:47:45 INFO skein.Driver: Driver shut down APPLICATION_ID NAME STATE STATUS CONTAINERS VCORES MEMORY RUNTIM |

So it means the problem is to set a suitable proxy?

Thank you!

-----Original Messages----- From:"Jim Crist" notifications@github.com Sent Time:2019-03-08 11:05:56 (Friday) To: jcrist/yarnspawner yarnspawner@noreply.github.com Cc: RCJiao 3150104926@zju.edu.cn, Author author@noreply.github.com Subject: Re: [jcrist/yarnspawner] error when configuring the yarnspawner for a cluster (#5)

Looks like the yarn commandline application isn't on $PATH, which is causing the daemon startup to fail. I'm not sure what's up with your hadoop installation, but the yarn CLI comes standard and is usually available. You likely need to add the hadoop /bin directory to your $PATH.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Best wishes!

Ruochen Jiao

Chu Kochen Honors College,Zhejiang University 3150104926@zju.edu.cn

jcrist commented 5 years ago

So it means the problem is to set a suitable proxy?

I assume you're talking about this line:

E0308 16:47:45.315065977 10280 http_proxy.cc:62] 'https' scheme not supported in proxy URI

This is due to some proxy set by environment variables like HTTP_PROXY/HTTPS_PROXY. You may want to unset or alter these environment variables before starting skein or jupyterhub. However, everything ran fine in the above, that error is innocuous. Does yarnspawner still not work after adding yarn to your $PATH?

jrcblue commented 5 years ago

Yes, there is still something wrong:

[E 2019-03-12 14:30:12.489 JupyterHub user:477] Unhandled error starting jrc's server: Failed to submit application, exception: User: jrc is not allowed to impersonate jrc [E 2019-03-12 14:30:12.577 JupyterHub user:486] Failed to cleanup jrc's server that failed to start Traceback (most recent call last): File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/user.py", line 482, in spawn await self.stop() File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/user.py", line 554, in stop await spawner.stop() File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/yarnspawner/spawner.py", line 268, in stop None, client.kill_application, self.app_id File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 624, in kill_application self._call('kill', proto.KillRequest(id=app_id, user=user)) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 278, in _call raise context.ValueError(exc.details()) ValueError: Invalid ApplicationId 'PENDING'

[E 2019-03-12 14:30:12.580 JupyterHub web:1788] Uncaught exception GET /hub/user/jrc/ (127.0.0.1) HTTPServerRequest(protocol='http', host='127.0.0.1:19999', method='GET', uri='/hub/user/jrc/', version='HTTP/1.1', remote_ip='127.0.0.1') Traceback (most recent call last): File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/tornado/web.py", line 1699, in _execute result = await result File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 1052, in get await self.spawn_single_user(user) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 705, in spawn_single_user timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn await spawn_future File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/user.py", line 489, in spawn raise e File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/jupyterhub/user.py", line 409, in spawn url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/yarnspawner/spawner.py", line 211, in start app_id = await loop.run_in_executor(None, client.submit, spec) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/concurrent/futures/thread.py", line 56, in run result = self.fn(*self.args, **self.kwargs) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 490, in submit resp = self._call('submit', spec.to_protobuf()) File "/home/jrc/miniconda3/envs/jh4y/lib/python3.6/site-packages/skein/core.py", line 280, in _call raise self._server_error(exc.details()) skein.exceptions.DriverError: Failed to submit application, exception: User: jrc is not allowed to impersonate jrc

[E 2019-03-12 14:30:12.596 JupyterHub log:150] { "Cookie": "jupyterhub-hub-login=\"2|1:0|10:1552372208|20:jupyterhub-hub-login|44:ZGZkYmVmYjQxMDI2NDAyYTg5NjU3MTNiNGIzMjcyZWY=|7d5a114311624cefb1dd81728add9d410a296f5f8ad83904f42f4cd3a39f650d\"; _xsrf=2|d8d45906|b01faa2c02009321af61a33ef9aed088|1551146410; username-127-0-0-1-18889=\"2|1:0|10:1552370290|24:username-127-0-0-1-18889|44:NmEzMmE3M2VhYjA4NDY0YzliMzc5NzQ3MmIxZWIxODg=|072bd41295f3da23b5f45f50edbc97c4344547772f396dcc701ee3a203b0c856\"; jupyterhub-session-id=4dffe99026204c7e80e05ff082a0f8e1", "Accept-Language": "en-US,en;q=0.9,zh;q=0.8", "Accept-Encoding": "gzip, deflate, br", "Referer": "http://127.0.0.1:19999/hub/login", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36", "Upgrade-Insecure-Requests": "1", "Cache-Control": "max-age=0", "Connection": "close", "Host": "127.0.0.1:19999" }

Thank you very much!

-----Original Messages----- From:"Jim Crist" notifications@github.com Sent Time:2019-03-09 03:00:54 (Saturday) To: jcrist/yarnspawner yarnspawner@noreply.github.com Cc: RCJiao 3150104926@zju.edu.cn, Author author@noreply.github.com Subject: Re: [jcrist/yarnspawner] error when configuring the yarnspawner for a cluster (#5)

So it means the problem is to set a suitable proxy?

I assume you're talking about this line:

E0308 16:47:45.315065977 10280 http_proxy.cc:62] 'https' scheme not supported in proxy URI

This is due to some proxy set by environment variables like HTTP_PROXY/HTTPS_PROXY. You may want to unset or alter these environment variables before starting skein or jupyterhub. However, everything ran fine in the above, that error is innocuous. Does yarnspawner still not work after adding yarn to your $PATH?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Best wishes!

Ruochen Jiao

Chu Kochen Honors College,Zhejiang University 3150104926@zju.edu.cn

jcrist commented 5 years ago

Ah, yeah, you need to be running jupyterhub as a different user than the requesting user. In my setup I have jupyterhub running as user jupyterhub. See the documentation: https://jcrist.github.io/yarnspawner/#id4.

I could change things to allow the jupyterhub user to also request containers, but I don't currently see a use case for that, as you'd really want a new privileged user to run jupyterhub for security reasons.

jrcblue commented 5 years ago

Thank you for your reply. However, I tried this way before. I had jupyterhub running in root and the logged in as user 'jrc'. And in the proxy configuration I set like this :hadoop.proxyuser.root.hosts ......

The error is shown as follows:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

19/03/13 11:48:35 INFO skein.Driver: Uploading application resources to hdfs://Gondolin-Node-065:9000/user/jrc/.skein/application_1551765709575_0113 [E 2019-03-13 11:48:35.839 JupyterHub user:477] Unhandled error starting jrc's server: Failed to submit application, exception: Permission denied: user=jrc, access=WRITE, inode="/user/jrc/.skein/application_1551765709575_0113":root:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1704) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1687) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3894) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:983) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

[E 2019-03-13 11:48:35.906 JupyterHub user:486] Failed to cleanup jrc's server that failed to start Traceback (most recent call last): File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 482, in spawn await self.stop() File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 554, in stop await spawner.stop() File "/root/miniconda3/lib/python3.7/site-packages/yarnspawner/spawner.py", line 268, in stop None, client.kill_application, self.app_id File "/root/miniconda3/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/root/miniconda3/lib/python3.7/site-packages/skein/core.py", line 624, in kill_application self._call('kill', proto.KillRequest(id=app_id, user=user)) File "/root/miniconda3/lib/python3.7/site-packages/skein/core.py", line 278, in _call raise context.ValueError(exc.details()) ValueError: Invalid ApplicationId 'PENDING'

[E 2019-03-13 11:48:35.908 JupyterHub web:1788] Uncaught exception GET /hub/user/jrc/ (127.0.0.1) HTTPServerRequest(protocol='http', host='127.0.0.1:19999', method='GET', uri='/hub/user/jrc/', version='HTTP/1.1', remote_ip='127.0.0.1') Traceback (most recent call last): File "/root/miniconda3/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute result = await result File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 1052, in get await self.spawn_single_user(user) File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 705, in spawn_single_user timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 626, in finish_user_spawn await spawn_future File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 489, in spawn raise e File "/root/miniconda3/lib/python3.7/site-packages/jupyterhub/user.py", line 409, in spawn url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f) File "/root/miniconda3/lib/python3.7/site-packages/yarnspawner/spawner.py", line 211, in start app_id = await loop.run_in_executor(None, client.submit, spec) File "/root/miniconda3/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/root/miniconda3/lib/python3.7/site-packages/skein/core.py", line 490, in submit resp = self._call('submit', spec.to_protobuf()) File "/root/miniconda3/lib/python3.7/site-packages/skein/core.py", line 280, in _call raise self._server_error(exc.details()) skein.exceptions.DriverError: Failed to submit application, exception: Permission denied: user=jrc, access=WRITE, inode="/user/jrc/.skein/application_1551765709575_0113":root:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1720) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1704) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1687) at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3894) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:983) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

[E 2019-03-13 11:48:35.955 JupyterHub log:150] { "Cookie": "jupyterhub-hub-login=\"2 1:0 10:1552448912 20:jupyterhub-hub-login 44:ZDc4NmIwYjc2NDEzNGQwY2JmNzczNzhiNTUyNDNiMmM= a2cc798326b0575935e569da41d8db93b77ab45454ce81d8e490d322f536e249\"; _xsrf=2 d8d45906 b01faa2c02009321af61a33ef9aed088 1551146410; jupyterhub-session-id=3bfbe44c746944ada694f960801ebdf8", "Accept-Language": "en-US,en;q=0.9,zh;q=0.8", "Accept-Encoding": "gzip, deflate, br", "Referer": "http://127.0.0.1:19999/hub/login", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36", "Upgrade-Insecure-Requests": "1", "Cache-Control": "max-age=0", "Connection": "close", "Host": "127.0.0.1:19999" } [E 2019-03-13 11:48:35.955 JupyterHub log:158] 500 GET /hub/user/jrc/ (jrc@127.0.0.1) 3077.63ms

I had two environments for jupyterhub-yarnspawners: one in root and one in user jrc. Above is running in root and logging in as another user. It's interesting the errors are different in these two ways. -----Original Messages----- From:"Jim Crist" notifications@github.com Sent Time:2019-03-12 22:14:26 (Tuesday) To: jcrist/yarnspawner yarnspawner@noreply.github.com Cc: RCJiao 3150104926@zju.edu.cn, Author author@noreply.github.com Subject: Re: [jcrist/yarnspawner] error when configuring the yarnspawner for a cluster (#5)

Ah, yeah, you need to be running jupyterhub as a different user than the requesting user. In my setup I have jupyterhub running as user jupyterhub. See the documentation: https://jcrist.github.io/yarnspawner/#id4.

I could change things to allow the jupyterhub user to also request containers, but I don't currently see a use case for that, as you'd really want a new privileged user to run jupyterhub for security reasons.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Best wishes!

Ruochen Jiao

Chu Kochen Honors College,Zhejiang University 3150104926@zju.edu.cn

jcrist commented 5 years ago

From this line:

Permission denied: user=jrc, access=WRITE, inode="/user/jrc/.skein/application_1551765709575_0113":root:supergroup:drwxr-xr-x

it looks like user jrc doesn't have write access to a subdirectory of its own home directory (generally user jrc should always have write access to everything under /user/jrc/). This is an odd state, and seems to me indicative that something in your environment is incorrect. Perhaps something was left in a bad state during a past attempt.

Can you try completely deleting the /user/jrc/.skein directory and trying again?

Also, reiterating what I said before - when posting things like code or tracebacks, its generally nice to surround them in markup (as I've done above) so they display nicely. For more information on how to do this see https://guides.github.com/features/mastering-markdown/.

jcrist commented 5 years ago

@jrcblue, any updates here?

jcrist commented 5 years ago

Closing as stale, feel free to reopen if you can provide more information.