Cloning large repo never finishes

brandonmcclure commented 3 years ago

Description

Clone appears part of the way, but the UI never finishes. It says it is "Cloning repository..." over night. This is only happening on a large repo (1.4GB) and is not happening for smaller repos

Reproduce

Click the clone repo button
enter url and auth for large repo (1GB)
Cloning happens without errors, and if you observe the files I can see the files getting created, and the repo is the expected size, but the UI/extension never moves out of the "Cloning repository" status.

Expected behavior

Cloning to finish and allow me to perform git operations on it.

Context

Python package version: Version: 0.30.1

Extension version:

JupyterLab v3.0.16
/opt/conda/share/jupyter/labextensions
    jupyter-matplotlib v0.9.0 enabled OK
    nbdime-jupyterlab v2.1.0 enabled OK
    @jupyter-widgets/jupyterlab-manager v3.0.0 enabled OK (python, jupyterlab_widgets)
    @jupyterlab/git v0.30.1 enabled OK (python, jupyterlab-git)

Git version: git version 2.31.1
Operating System and its version: This is running as a docker container from the base image: jupyter/datascience-notebook

Command Line Output

Executing the command: jupyter lab
/opt/conda/lib/python3.9/site-packages/jupyter_server/transutils.py:13: FutureWarning: The alias `_()` will be deprecated. Use `_i18n()` instead.
  warnings.warn(warn_msg, FutureWarning)
[I 2021-05-25 22:14:49.051 ServerApp] jupyter_server_mathjax | extension was successfully linked.
[I 2021-05-25 22:14:49.065 ServerApp] jupyterlab | extension was successfully linked.
[I 2021-05-25 22:14:49.065 ServerApp] jupyterlab_git | extension was successfully linked.
[W 2021-05-25 22:14:49.069 NotebookApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.  
[W 2021-05-25 22:14:49.069 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2021-05-25 22:14:49.069 NotebookApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[I 2021-05-25 22:14:49.078 ServerApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/jupyter_cookie_secret
[I 2021-05-25 22:14:49.315 ServerApp] nbclassic | extension was successfully linked.
[I 2021-05-25 22:14:49.315 ServerApp] nbdime | extension was successfully linked.
[I 2021-05-25 22:14:49.340 ServerApp] nbclassic | extension was successfully loaded.
[I 2021-05-25 22:14:49.341 ServerApp] jupyter_server_mathjax | extension was successfully loaded.
[I 2021-05-25 22:14:49.342 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab
[I 2021-05-25 22:14:49.342 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2021-05-25 22:14:49.345 ServerApp] jupyterlab | extension was successfully loaded.
[I 2021-05-25 22:14:49.348 ServerApp] jupyterlab_git | extension was successfully loaded.
[I 2021-05-25 22:14:49.423 ServerApp] nbdime | extension was successfully loaded.
[I 2021-05-25 22:14:49.426 ServerApp] Serving notebooks from local directory: /home/jovyan
[I 2021-05-25 22:14:49.426 ServerApp] Jupyter Server 1.8.0 is running at:
[I 2021-05-25 22:14:49.426 ServerApp] http://020c042a1ccd:8888/lab?token=...
[I 2021-05-25 22:14:49.426 ServerApp]     http://127.0.0.1:8888/lab?token=...
[I 2021-05-25 22:14:49.426 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-05-25 22:20:04.054 ServerApp] 302 GET / (172.17.0.1) 1.02ms
[W 2021-05-25 22:20:04.073 ServerApp] Clearing invalid/expired login cookie username-localhost-8888
[W 2021-05-25 22:20:04.073 LabApp] Clearing invalid/expired login cookie username-localhost-8888
[I 2021-05-25 22:20:04.074 LabApp] 302 GET /lab? (172.17.0.1) 2.51ms
[I 2021-05-25 22:20:08.835 ServerApp] 302 POST /login?next=%2Flab%3F (172.17.0.1) 0.98ms
[I 2021-05-25 22:20:12.174 LabApp] Build is up to date
[E 2021-05-25 22:20:42.977 ServerApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Content-Length": "138",
      "Sec-Ch-Ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"",
      "Authorization": "token password",
      "X-Xsrftoken": "2|6f9104cc|dba8ca3e7a173e12b695365f1e8ea722|1621968277",
      "Sec-Ch-Ua-Mobile": "?0",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66",
      "Content-Type": "text/plain;charset=UTF-8",
      "Accept": "*/*",
      "Origin": "http://localhost:8888",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "cors",
      "Sec-Fetch-Dest": "empty",
      "Referer": "http://localhost:8888/lab/workspaces/auto-e",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "_xsrf=2|6f9104cc|dba8ca3e7a173e12b695365f1e8ea722|1621968277; username-localhost-8889=\"2|1:0|10:1621980254|23:username-localhost-8889|44:Mzk1NWMyMWYzNjNmNDY5Njg3YTQwMzQ4YzhkMDkwNzc=|1add14bb562585e1f15a230a2961ac4626219af371de6d7db290fde9abd8f296\"; username-localhost-8888=\"2|1:0|10:1621981241|23:username-localhost-8888|44:MDBkY2U4YjNkZjFlNDBiNDk2YzBkNGZjNDUxMmM5ZTE=|6c6782ac09827374ec86d196f79a24aff58cc3d5b47042e7bc15654c8a6df059\""
    }
[E 2021-05-25 22:20:42.977 ServerApp] 500 POST /git/clone?1621981241646 (172.17.0.1) 1018.72ms referer=http://localhost:8888/lab/workspaces/auto-e
[E 2021-05-25 22:20:51.969 ServerApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Content-Length": "235",
      "Sec-Ch-Ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"",
      "Authorization": "token password",
      "X-Xsrftoken": "2|6f9104cc|dba8ca3e7a173e12b695365f1e8ea722|1621968277",
      "Sec-Ch-Ua-Mobile": "?0",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66",
      "Content-Type": "text/plain;charset=UTF-8",
      "Accept": "*/*",
      "Origin": "http://localhost:8888",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "cors",
      "Sec-Fetch-Dest": "empty",
      "Referer": "http://localhost:8888/lab/workspaces/auto-e",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "_xsrf=2|6f9104cc|dba8ca3e7a173e12b695365f1e8ea722|1621968277; username-localhost-8889=\"2|1:0|10:1621980254|23:username-localhost-8889|44:Mzk1NWMyMWYzNjNmNDY5Njg3YTQwMzQ4YzhkMDkwNzc=|1add14bb562585e1f15a230a2961ac4626219af371de6d7db290fde9abd8f296\"; username-localhost-8888=\"2|1:0|10:1621981241|23:username-localhost-8888|44:MDE3ZDNjMTQyZTQ5NGY0ZWE2YmIzNzdlN2UxNWQ2Yjg=|8e8a227540dd187e38a4075ffc15a3efc12c786020141a2dd91135837d881e65\""
    }
[E 2021-05-25 22:20:51.969 ServerApp] 500 POST /git/clone?1621981249628 (172.17.0.1) 2028.99ms referer=http://localhost:8888/lab/workspaces/auto-e
[E 2021-05-25 22:21:04.844 ServerApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Content-Length": "138",
      "Sec-Ch-Ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"",
      "Authorization": "token password",
      "X-Xsrftoken": "2|6f9104cc|dba8ca3e7a173e12b695365f1e8ea722|1621968277",
      "Sec-Ch-Ua-Mobile": "?0",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66",
      "Content-Type": "text/plain;charset=UTF-8",
      "Accept": "*/*",
      "Origin": "http://localhost:8888",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "cors",
      "Sec-Fetch-Dest": "empty",
      "Referer": "http://localhost:8888/lab/workspaces/auto-e",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "_xsrf=2|6f9104cc|dba8ca3e7a173e12b695365f1e8ea722|1621968277; username-localhost-8889=\"2|1:0|10:1621980254|23:username-localhost-8889|44:Mzk1NWMyMWYzNjNmNDY5Njg3YTQwMzQ4YzhkMDkwNzc=|1add14bb562585e1f15a230a2961ac4626219af371de6d7db290fde9abd8f296\"; username-localhost-8888=\"2|1:0|10:1621981262|23:username-localhost-8888|44:MzExZjE0MzIwODI5NGFkZDg3Y2RmNWY0YTBiMzQxM2E=|8294bb88029ed9f126584f26726e89a9ca997c71439258477088f4349d43042b\""
    }
[E 2021-05-25 22:21:04.844 ServerApp] 500 POST /git/clone?1621981263467 (172.17.0.1) 1065.01ms referer=http://localhost:8888/lab/workspaces/auto-e
[E 2021-05-26 15:37:47.519 ServerApp] {
      "Host": "localhost:8888",
      "Connection": "keep-alive",
      "Content-Length": "27",
      "Sec-Ch-Ua": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"",
      "Authorization": "token password",
      "X-Xsrftoken": "2|6f9104cc|dba8ca3e7a173e12b695365f1e8ea722|1621968277",
      "Sec-Ch-Ua-Mobile": "?0",
      "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66",
      "Content-Type": "text/plain;charset=UTF-8",
      "Accept": "*/*",
      "Origin": "http://localhost:8888",
      "Sec-Fetch-Site": "same-origin",
      "Sec-Fetch-Mode": "cors",
      "Sec-Fetch-Dest": "empty",
      "Referer": "http://localhost:8888/lab/workspaces/auto-e",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept-Language": "en-US,en;q=0.9",
      "Cookie": "_xsrf=2|6f9104cc|dba8ca3e7a173e12b695365f1e8ea722|1621968277; username-localhost-8889=\"2|1:0|10:1621980254|23:username-localhost-8889|44:Mzk1NWMyMWYzNjNmNDY5Njg3YTQwMzQ4YzhkMDkwNzc=|1add14bb562585e1f15a230a2961ac4626219af371de6d7db290fde9abd8f296\"; username-localhost-8888=\"2|1:0|10:1622043447|23:username-localhost-8888|44:MmQ0NzkwY2E5MTVjNDcwOGFmYWIyNjIyMDc1NzA2OWE=|e9b29028fb2ae4dbbcf699d22870373b100eec6ca0b14a43799cd5bd84dead18\""
    }
[E 2021-05-26 15:37:47.532 ServerApp] 500 POST /git/show_top_level?1622043447335 (172.17.0.1) 20009.47ms referer=http://localhost:8888/lab/workspaces/auto-e

Web Browser Output

[DOM] Password field is not contained in a form: (More info: https://www.chromium.org/developers/design-documents/create-amazing-password-forms) 
:8888/git/show_top_level?1622043447335:1 Failed to load resource: the server responded with a status of 500 (Internal Server Error)
664.1e7f16b3afb000957a27.js?v=1e7f16b3afb000957a27:1 Fail to find Git top level for path Caboodle.
Error: Unable to get the lock on the directory

welcome[bot] commented 3 years ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

brandonmcclure commented 3 years ago

I am tried this again today disconnected from our VPN. Sometimes our security appliances can cause issues communicating over the web/to our Azure Devops repos, especially when traffic is coming from inside a docker container. I get the same behavior and results though which would rule out interference from those security appliances.

fcollonval commented 3 years ago

Thanks for reporting the issue. Did you try to run the git clone command via the terminal? If so does it works there?

Dev note: a work around is gonna be tricky as the extension assumes git command to be quick. We may need to change the logic to ping server for task progression on certain git commands.

brandonmcclure commented 3 years ago

Running clone from terminal hangs after some time. If I hit enter then I get prompted to enter my password again.

I set git config --global credential.helper 'cache --timeout=2400' in my docker image and running from the command line and from the UI it works as expected. Not sure how the extension could detect/alert the user to this since it is related to how the host/git credential manager is configured. I am happy to help troubleshoot this further if you would like to pursue more, or we can close.

fcollonval commented 3 years ago

Thanks @brandonmcclure

There gonna be some work on dealing with credentials this summer. So let's keep this case open. It could justify setting the cache credential helper if none is provided.

jupyterlab / jupyterlab-git