search

mhammond / pywin32

Python for Windows (pywin32) Extensions
5.06k stars 798 forks source link

pywintypes.error: (50, 'GetFileSecurity', 'The request is not supported.') kernel error with jupyter lab version 1.1.4 #1415

Closed wikwok closed 5 years ago

wikwok commented 5 years ago

Description

Launching jupyter notebook v1.1.4 python3.7 from Anaconda3 throws kernel error on windows 8.1

Default installation of jupyterlab v1.0.2 compiled with Anaconda3 works fine and not prone to this error.

Also Mac seems not prone to this issue when testing it on Mac.

inspecting the terminal feed, it seems that pywin32 has a bug on windows8.1 causing the kernal to fail.

Please see full details below:

Traceback (most recent call last):
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\tornado\web.py", line 1699, in _execute
    result = await result
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\tornado\gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\notebook\services\sessions\handlers.py", line 72, in post
    type=mtype))
  File "C:\Users\sxxx\.conda\envs\logAna\lib\site-packages\tornado\gen.py", line 735, in run
    value = future.result()
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\tornado\gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 88, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\tornado\gen.py", line 735, in run
    value = future.result()
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\tornado\gen.py", line 742, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 101, in start_kernel_for_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\tornado\gen.py", line 735, in run
    value = future.result()
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\tornado\gen.py", line 209, in wrapper
    yielded = next(result)
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\notebook\services\kernels\kernelmanager.py", line 168, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\jupyter_client\multikernelmanager.py", line 110, in start_kernel
    km.start_kernel(**kwargs)
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\jupyter_client\manager.py", line 240, in start_kernel
    self.write_connection_file()
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\jupyter_client\connect.py", line 547, in write_connection_file
    kernel_name=self.kernel_name
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\jupyter_client\connect.py", line 212, in write_connection_file
    with secure_write(fname) as f:
  File "C:\Users\xxx\.conda\envs\logAna\lib\contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\jupyter_client\connect.py", line 100, in secure_write
    win32_restrict_file_to_user(fname)
  File "C:\Users\xxx\.conda\envs\logAna\lib\site-packages\jupyter_client\connect.py", line 61, in win32_restrict_file_to_user
    sd = win32security.GetFileSecurity(fname, win32security.DACL_SECURITY_INFORMATION)
pywintypes.error: (50, 'GetFileSecurity', 'The request is not supported.')

Reproduce

At first instance jupyter was automatically updated when installing other Anaconda3 packages which seemed to cause the error. However, after testing lots of setups and finally installing jupyterlab on a fresh clean anaconda environment determined that the new version has got a bug running on windows 8.1 Enterprise 64bit SP0 at the least. I don't know if other versions of windows are prone to this.

Expected behavior

create new python3.7 notebooks without any kernel errors due to pywin32.

Context

Troubleshoot Output 

(base) C:\WINDOWS\system32>activate test

(test) C:\WINDOWS\system32>jupyter troubleshoot
$PATH:
        C:\Users\xxx\.conda\envs\test
        C:\Users\xxx\.conda\envs\test\Library\mingw-w64\bin
        C:\Users\xxx\.conda\envs\test\Library\usr\bin
        C:\Users\xxx\.conda\envs\test\Library\bin
        C:\Users\xxx\.conda\envs\test\Scripts
        C:\Users\xxx\.conda\envs\test\bin
        C:\ProgramData\Anaconda3\condabin
        C:\oracle\Oracle11g\product\11.2.0\client_1\bin
        C:\Rtools\bin
        C:\ProgramData\Oracle\Java\javapath
        C:\WINDOWS\system32
        C:\WINDOWS
        C:\WINDOWS\System32\Wbem
        C:\WINDOWS\System32\WindowsPowerShell\v1.0
        C:\Program Files (x86)\ZANTAZ\EAS Outlook Addin
        C:\Program Files\Liquid Technologies\Liquid Studio 2018\XmlDataBinder16\
Redist16\cpp\win32\bin
        C:\Program Files\Liquid Technologies\Liquid Studio 2018\XmlDataBinder16\
Redist16\cpp\win64\bin
        C:\Program Files\PostgreSQL\11\bin
        C:\Program Files\SAS94\Secure\ccme4
        C:\Program Files\SAS94\x86\Secure\ccme4
        C:\oracle\10.2.0\bin
        C:\BI_HOME\bin
        C:\BI_HOME\jre\1.4.2\bin
        C:\BI_HOME\jre\1.4.2\bin\client
        C:\BI_HOME\jlib
        C:\OpenSSL-Win64\bin
        C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn
        C:\Users\xxx\AppData\Local\Programs\Microsoft VS Code\bin

sys.path:
        C:\Users\xxx\.conda\envs\test\Scripts
        C:\Users\xxx\.conda\envs\test\python37.zip
        C:\Users\xxx\.conda\envs\test\DLLs
        C:\Users\xxx\.conda\envs\test\lib
        C:\Users\xxx\.conda\envs\test
        C:\Users\xxx\.conda\envs\test\lib\site-packages
        C:\Users\xxx\.conda\envs\test\lib\site-packages\win32
        C:\Users\xxx\.conda\envs\test\lib\site-packages\win32\lib
        C:\Users\xxx\.conda\envs\test\lib\site-packages\Pythonwin

sys.executable:
        C:\Users\xxx\.conda\envs\test\python.exe

sys.version:
        3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)]

platform.platform():
        Windows-8.1-6.3.9600-SP0

where jupyter:
        C:\Users\xxx\.conda\envs\test\Scripts\jupyter.exe

pip list:
        Package           Version
        ----------------- ---------
        attrs             19.1.0
        backcall          0.1.0
        bleach            3.1.0
        certifi           2019.9.11
        colorama          0.4.1
        decorator         4.4.0
        defusedxml        0.6.0
        entrypoints       0.3
        ipykernel         5.1.2
        ipython           7.8.0
        ipython-genutils  0.2.0
        jedi              0.15.1
        Jinja2            2.10.1
        json5             0.8.5
        jsonschema        3.0.2
        jupyter-client    5.3.3
        jupyter-core      4.5.0
        jupyterlab        1.1.4
        jupyterlab-server 1.0.6
        MarkupSafe        1.1.1
        mistune           0.8.4
        nbconvert         5.6.0
        nbformat          4.4.0
        notebook          6.0.1
        pandocfilters     1.4.2
        parso             0.5.1
        pickleshare       0.7.5
        pip               19.2.3
        prometheus-client 0.7.1
        prompt-toolkit    2.0.9
        Pygments          2.4.2
        pyrsistent        0.15.4
        python-dateutil   2.8.0
        pywin32           223
        pywinpty          0.5.5
        pyzmq             18.1.0
        Send2Trash        1.5.0
        setuptools        41.2.0
        six               1.12.0
        terminado         0.8.2
        testpath          0.4.2
        tornado           6.0.3
        traitlets         4.3.2
        wcwidth           0.1.7
        webencodings      0.5.1
        wheel             0.33.6
        wincertstore      0.2

(test) C:\WINDOWS\system32>
Command Line Output 

(base) C:\WINDOWS\system32>activate test

(test) C:\WINDOWS\system32>jupyter lab
[I 15:50:55.678 LabApp] JupyterLab extension loaded from C:\Users\xxx\
.conda\envs\test\lib\site-packages\jupyterlab
[I 15:50:55.678 LabApp] JupyterLab application directory is C:\Users\xxx\.conda\envs\test\share\jupyter\lab
[I 15:50:55.682 LabApp] Serving notebooks from local directory: C:\WINDOWS\syste
m32
[I 15:50:55.682 LabApp] The Jupyter Notebook is running at:
[I 15:50:55.682 LabApp] http://localhost:8888/?token=e2f1c73d291f7e884d3c618eb43
83196c2d75484a0f88d96
[I 15:50:55.682 LabApp]  or http://127.0.0.1:8888/?token=e2f1c73d291f7e884d3c618
eb4383196c2d75484a0f88d96
[I 15:50:55.682 LabApp] Use Control-C to stop this server and shut down all kern
els (twice to skip confirmation).
[C 15:50:55.914 LabApp]

    To access the notebook, open this file in a browser:
        file://A-FILEDATA-COV-P/Profile/xxx/AppData/Roaming/jupyter/ru
ntime/nbserver-8544-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=e2f1c73d291f7e884d3c618eb4383196c2d75484a0f
88d96
     or http://127.0.0.1:8888/?token=e2f1c73d291f7e884d3c618eb4383196c2d75484a0f
88d96
[W 15:51:04.157 LabApp] Could not determine jupyterlab build status without node
js
[E 15:51:13.236 LabApp] Uncaught exception POST /api/sessions?1569509473167 (::1
)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='POST', uri
='/api/sessions?1569509473167', version='HTTP/1.1', remote_ip='::1')
    Traceback (most recent call last):
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\we
b.py", line 1699, in _execute
        result = await result
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 742, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\notebook\s
ervices\sessions\handlers.py", line 72, in post
        type=mtype))
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 735, in run
        value = future.result()
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 742, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\notebook\s
ervices\sessions\sessionmanager.py", line 88, in create_session
        kernel_id = yield self.start_kernel_for_session(session_id, path, name,
type, kernel_name)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 735, in run
        value = future.result()
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 742, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\notebook\s
ervices\sessions\sessionmanager.py", line 101, in start_kernel_for_session
        self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_na
me)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 735, in run
        value = future.result()
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 209, in wrapper
        yielded = next(result)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\notebook\s
ervices\kernels\kernelmanager.py", line 168, in start_kernel
        super(MappingKernelManager, self).start_kernel(**kwargs)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\multikernelmanager.py", line 110, in start_kernel
        km.start_kernel(**kwargs)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\manager.py", line 240, in start_kernel
        self.write_connection_file()
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\connect.py", line 547, in write_connection_file
        kernel_name=self.kernel_name
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\connect.py", line 212, in write_connection_file
        with secure_write(fname) as f:
      File "C:\Users\xxx\.conda\envs\test\lib\contextlib.py", line 112
, in __enter__
        return next(self.gen)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\connect.py", line 100, in secure_write
        win32_restrict_file_to_user(fname)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\connect.py", line 61, in win32_restrict_file_to_user
        sd = win32security.GetFileSecurity(fname, win32security.DACL_SECURITY_IN
FORMATION)
    pywintypes.error: (50, 'GetFileSecurity', 'The request is not supported.')
[W 15:51:13.245 LabApp] Unhandled error
[E 15:51:13.247 LabApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Content-Length": "97",
      "Authorization": "token e2f1c73d291f7e884d3c618eb4383196c2d75484a0f88d96",

      "Origin": "http://localhost:8888",
      "X-Xsrftoken": "2|e0797587|bfb0295d2cfd16737a016247e6e4c3bb|1569403253",
      "User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36
 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36",
      "Content-Type": "application/json",
      "Accept": "*/*",
      "Referer": "http://localhost:8888/lab",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "_xsrf=2|e0797587|bfb0295d2cfd16737a016247e6e4c3bb|1569403253; u
sername-localhost-8889=\"2|1:0|10:1569504770|23:username-localhost-8889|44:NGY0Y
jJmZTI2MDZiNGU1ZDk5MTNjYjY3MTcwYjdlMDM=|caa84e45c938a438e74fe746b2d337a2e50d7882
a22c70ec0f114d22aa337feb\"; username-localhost-8888=\"2|1:0|10:1569509473|23:use
rname-localhost-8888|44:MDFkMmNlYjcwYzgxNGY3M2JlYmFjYTA4OTE2MGMwNDI=|d3bdb147f2b
afff4d6e7528326a74e6b7d13199b6f63400e84e2c00af8337dbf\""
    }
[E 15:51:13.248 LabApp] 500 POST /api/sessions?1569509473167 (::1) 69.19ms refer
er=http://localhost:8888/lab
[E 15:51:38.978 LabApp] Uncaught exception POST /api/sessions?1569509497781 (::1
)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='POST', uri
='/api/sessions?1569509497781', version='HTTP/1.1', remote_ip='::1')
    Traceback (most recent call last):
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\we
b.py", line 1699, in _execute
        result = await result
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 742, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\notebook\s
ervices\sessions\handlers.py", line 72, in post
        type=mtype))
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 735, in run
        value = future.result()
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 742, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\notebook\s
ervices\sessions\sessionmanager.py", line 88, in create_session
        kernel_id = yield self.start_kernel_for_session(session_id, path, name,
type, kernel_name)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 735, in run
        value = future.result()
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 742, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\notebook\s
ervices\sessions\sessionmanager.py", line 101, in start_kernel_for_session
        self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_na
me)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 735, in run
        value = future.result()
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\tornado\ge
n.py", line 209, in wrapper
        yielded = next(result)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\notebook\s
ervices\kernels\kernelmanager.py", line 168, in start_kernel
        super(MappingKernelManager, self).start_kernel(**kwargs)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\multikernelmanager.py", line 110, in start_kernel
        km.start_kernel(**kwargs)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\manager.py", line 240, in start_kernel
        self.write_connection_file()
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\connect.py", line 547, in write_connection_file
        kernel_name=self.kernel_name
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\connect.py", line 212, in write_connection_file
        with secure_write(fname) as f:
      File "C:\Users\xxx\.conda\envs\test\lib\contextlib.py", line 112
, in __enter__
        return next(self.gen)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\connect.py", line 100, in secure_write
        win32_restrict_file_to_user(fname)
      File "C:\Users\xxx\.conda\envs\test\lib\site-packages\jupyter_cl
ient\connect.py", line 61, in win32_restrict_file_to_user
        sd = win32security.GetFileSecurity(fname, win32security.DACL_SECURITY_IN
FORMATION)
    pywintypes.error: (50, 'GetFileSecurity', 'The request is not supported.')
[W 15:51:38.984 LabApp] Unhandled error
[E 15:51:38.985 LabApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Content-Length": "97",
      "Pragma": "no-cache",
      "Cache-Control": "no-cache",
      "Authorization": "token e2f1c73d291f7e884d3c618eb4383196c2d75484a0f88d96",

      "Origin": "http://localhost:8888",
      "X-Xsrftoken": "2|e0797587|bfb0295d2cfd16737a016247e6e4c3bb|1569403253",
      "User-Agent": "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36
 (KHTML, like Gecko) Chrome/73.0.3683.75 Safari/537.36",
      "Content-Type": "application/json",
      "Accept": "*/*",
      "Referer": "http://localhost:8888/lab",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "_xsrf=2|e0797587|bfb0295d2cfd16737a016247e6e4c3bb|1569403253; u
sername-localhost-8889=\"2|1:0|10:1569504770|23:username-localhost-8889|44:NGY0Y
jJmZTI2MDZiNGU1ZDk5MTNjYjY3MTcwYjdlMDM=|caa84e45c938a438e74fe746b2d337a2e50d7882
a22c70ec0f114d22aa337feb\"; username-localhost-8888=\"2|1:0|10:1569509486|23:use
rname-localhost-8888|44:OWMzYTU5MjgzZDQ0NDQ5Zjk3ZDZkYzUzNTE5MDYxMjQ=|fc18aa7b12f
dfb162c64a2236e57c93d1088d13b7ae52fd63a37ffd9e815de38\""
    }
[E 15:51:38.986 LabApp] 500 POST /api/sessions?1569509497781 (::1) 25.07ms refer
er=http://localhost:8888/lab
Browser Output 
Starting application in workspace: "/lab"
api/sessions?1569509473167:1 
Failed to load resource: the server responded with a status of 500 (Internal Server Error)
serverconnection.js:193 
POST http://localhost:8888/api/sessions?1569509497781 500 (Internal Server Error)
handleRequest @ serverconnection.js:193
makeRequest @ serverconnection.js:75
startSession @ default.js:636
startNew @ default.js:614
startNew @ default.js:378
startNew @ session.js:50
startNew @ manager.js:166
_startSession @ clientsession.js:418
_changeKernel @ clientsession.js:370
(anonymous) @ clientsession.js:403
serverconnection.js:89 
Uncaught (in promise) Error: Invalid response: 500 Internal Server Error
    at default.js:639
    at async SessionManager.startNew (manager.js:166)
ResponseError @ serverconnection.js:89
(anonymous) @ default.js:639
mhammond commented 5 years ago

I see no reason to believe this is a bug in pywin32 - the win32 function GetFileSecurity() is just failing when used in this context. I suggest you open a jupyter bug.

wikwok commented 4 years ago

@mhammond you were right! It is NOT a pywin32 issue!

I found the problem and a solution as described below in comment for anyone that may land here looking for an answer.

Many thanks goes to @kevin-bates for finding this solution!

This is my story for anyone else to learn (I run Anconda which comes pre installed with Jupyter):

1- When checking the environment paths by using this command: jupyter --paths I get two roaming locations one for runtime and one for data! This is because I use a corporate laptop and Windows is setup to store user data files in a roaming location on the network! 

(base) C:\>jupyter --paths
config:
    C:\Users\user_x\.jupyter
    C:\ProgramData\Anaconda3\etc\jupyter
    C:\ProgramData\jupyter
data:
    \\A-FILEDATA-P\Profile\use_x\AppData\Roaming\jupyter
    C:\ProgramData\Anaconda3\share\jupyter
    C:\ProgramData\jupyter
runtime:
    \\A-FILEDATA-P\Profile\use_x\AppData\Roaming\jupyter\runtime

(base) C:\>

2- So, I tried modifying C:\Users\user_xxx.jupyter\jupyter_notebook_config.py by adding JUPYTER_RUNTIME_DIR = 'C:\Users\user_x\AppData\Roaming\jupyter\runtime' but it did not work!

2- However, Big note here: I noticed that jupyter lab command fire a "stand alone" Chrome I have on a removable usb harddisk! which under the corporate Windows setup does not have any rights to write to other folder location on the laptop, thus the security error!

3- The solution was to setup two folders on my usb disk, where stand alone Chrome is found and set the environment variables to point to those locations like this:

set JUPYTER_RUNTIME_DIR=D:\jupyter\runtime
set JUPYTER_DATA_DIR=D:\jupyter

4- The only catch is that you have to do these two commands at the start of every new session as they don't persist say when you restart Jupyter lab! and as @Shongololo pointed out you can only run other programs like Spyder from the command line only for it to read the correct settings!

It remains to be found out how this could be resolved permanently.