databrickslabs / dbx

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.
https://dbx.readthedocs.io
Other
443 stars 121 forks source link

Installing dbx on Windows 10 Enterprise fails: "The filename or extension is too long" #778

Closed 4kbnick closed 1 year ago

4kbnick commented 1 year ago

Hello - I am attempting to install dbx on a Windows 10 Enterprise desktop (running as a virtual desktop on Microsoft Devbox). However, during the installation, I am getting an error from the Python installer that the filename or extension being written is too long.

Expected Behavior

pip3 install dbx successfully installs the dbx package

Current Behavior

pip3 install dbx fails with the following error:

ERROR: Could not install packages due to an OSError: [WinError 206] The filename or extension is too long: 'C:\Users\\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\dbx\templates\projects\python_basic\render\{{cookiecutter.project_name}}\{{cookiecutter.project_slug}}\tasks'

Steps to Reproduce (for bugs)

Download and install VSCode Download and install Python 3.10 from Microsoft Store In VSCode, open terminal > type pip3 install dbx

Context

There is a Windows registry key that can override the max_path setting of the operating system. However, I am running in an enterprise environment and don't have direct access to modify the registry. I can probably ask IT to do it on my behalf, but it would be nice if the package didn't bring up this error in the first place.

Your Environment

Windows 10 Enterprise Version 10.0.19045 Build 19045 VSCode 1.75.1 Python 3.10 from Microsoft Store

Extended Log

PS C:\Users\redacted\bl-lakehouse\bl-datalakehouse> pip3 install dbx Collecting dbx Downloading dbx-0.8.14-py3-none-any.whl (136 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 136.2/136.2 kB 1.6 MB/s eta 0:00:00 Collecting mlflow-skinny<3.0.0,>=1.28.0 Downloading mlflow_skinny-2.3.1-py3-none-any.whl (4.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 32.5 MB/s eta 0:00:00 Collecting rich==12.6.0 Downloading rich-12.6.0-py3-none-any.whl (237 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 237.5/237.5 kB 7.3 MB/s eta 0:00:00 Collecting pathspec>=0.9.0 Downloading pathspec-0.11.1-py3-none-any.whl (29 kB) Collecting databricks-cli<0.18,>=0.17 Downloading databricks-cli-0.17.7.tar.gz (83 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.5/83.5 kB ? eta 0:00:00 Preparing metadata (setup.py) ... done Collecting click<9.0.0,>=8.1.0 Downloading click-8.1.3-py3-none-any.whl (96 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 kB 5.4 MB/s eta 0:00:00 Collecting typer[all]==0.7.0 Downloading typer-0.7.0-py3-none-any.whl (38 kB) Collecting tenacity<=9.0.0,>=8.2.2 Downloading tenacity-8.2.2-py3-none-any.whl (24 kB) Collecting Jinja2>=2.11.2 Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB ? eta 0:00:00 Collecting cookiecutter<3.0.0,>=1.7.2 Downloading cookiecutter-2.1.1-py2.py3-none-any.whl (36 kB) Collecting cryptography<41.0.0,>=3.3.1 Downloading cryptography-40.0.2-cp36-abi3-win_amd64.whl (2.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 41.3 MB/s eta 0:00:00 Collecting pyyaml>=6.0 Downloading PyYAML-6.0-cp310-cp310-win_amd64.whl (151 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 8.8 MB/s eta 0:00:00 Collecting aiohttp>=3.8.2 Downloading aiohttp-3.8.4-cp310-cp310-win_amd64.whl (319 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 319.8/319.8 kB 20.6 MB/s eta 0:00:00 Collecting watchdog>=2.1.0 Downloading watchdog-3.0.0-py3-none-win_amd64.whl (82 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.0/82.0 kB 4.5 MB/s eta 0:00:00 Collecting pydantic<=2.0.0,>=1.9.1 Downloading pydantic-1.10.7-cp310-cp310-win_amd64.whl (2.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 44.8 MB/s eta 0:00:00 Collecting requests<2.30.0,>=2.24.0 Downloading requests-2.29.0-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.5/62.5 kB 3.3 MB/s eta 0:00:00 Collecting pygments<3.0.0,>=2.6.0 Downloading Pygments-2.15.1-py3-none-any.whl (1.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 36.7 MB/s eta 0:00:00 Collecting commonmark<0.10.0,>=0.9.0 Downloading commonmark-0.9.1-py2.py3-none-any.whl (51 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.1/51.1 kB 2.6 MB/s eta 0:00:00 Collecting shellingham<2.0.0,>=1.3.0 Downloading shellingham-1.5.0.post1-py2.py3-none-any.whl (9.4 kB) Collecting colorama<0.5.0,>=0.4.3 Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB) Collecting charset-normalizer<4.0,>=2.0 Downloading charset_normalizer-3.1.0-cp310-cp310-win_amd64.whl (97 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.1/97.1 kB ? eta 0:00:00 Collecting async-timeout<5.0,>=4.0.0a3 Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB) Collecting attrs>=17.3.0 Downloading attrs-23.1.0-py3-none-any.whl (61 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 kB 3.2 MB/s eta 0:00:00 Collecting aiosignal>=1.1.2 Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB) Collecting yarl<2.0,>=1.0 Downloading yarl-1.9.2-cp310-cp310-win_amd64.whl (61 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 3.4 MB/s eta 0:00:00 Collecting frozenlist>=1.1.1 Downloading frozenlist-1.3.3-cp310-cp310-win_amd64.whl (33 kB) Collecting multidict<7.0,>=4.5 Downloading multidict-6.0.4-cp310-cp310-win_amd64.whl (28 kB) Collecting jinja2-time>=0.2.0 Downloading jinja2_time-0.2.0-py2.py3-none-any.whl (6.4 kB) Collecting binaryornot>=0.4.4 Downloading binaryornot-0.4.4-py2.py3-none-any.whl (9.0 kB) Collecting python-slugify>=4.0.0 Downloading python_slugify-8.0.1-py2.py3-none-any.whl (9.7 kB) Collecting cffi>=1.12 Downloading cffi-1.15.1-cp310-cp310-win_amd64.whl (179 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 179.1/179.1 kB 10.6 MB/s eta 0:00:00 Collecting pyjwt>=1.7.0 Downloading PyJWT-2.7.0-py3-none-any.whl (22 kB) Collecting oauthlib>=3.1.0 Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 9.4 MB/s eta 0:00:00 Collecting tabulate>=0.7.7 Downloading tabulate-0.9.0-py3-none-any.whl (35 kB) Collecting six>=1.10.0 Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting urllib3<2.0.0,>=1.26.7 Downloading urllib3-1.26.15-py2.py3-none-any.whl (140 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.9/140.9 kB 8.2 MB/s eta 0:00:00 Collecting MarkupSafe>=2.0 Downloading MarkupSafe-2.1.2-cp310-cp310-win_amd64.whl (16 kB) Collecting entrypoints<1 Downloading entrypoints-0.4-py3-none-any.whl (5.3 kB) Collecting cloudpickle<3 Downloading cloudpickle-2.2.1-py3-none-any.whl (25 kB) Collecting importlib-metadata!=4.7.0,<7,>=3.7.0 Downloading importlib_metadata-6.6.0-py3-none-any.whl (22 kB) Collecting sqlparse<1,>=0.4.0 Downloading sqlparse-0.4.4-py3-none-any.whl (41 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.2/41.2 kB ? eta 0:00:00 Collecting pytz<2024 Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 502.3/502.3 kB 30.8 MB/s eta 0:00:00 Collecting protobuf<5,>=3.12.0 Downloading protobuf-4.23.0-cp310-abi3-win_amd64.whl (422 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 422.5/422.5 kB 25.8 MB/s eta 0:00:00 Collecting gitpython<4,>=2.1.0 Downloading GitPython-3.1.31-py3-none-any.whl (184 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 184.3/184.3 kB 10.9 MB/s eta 0:00:00 Collecting packaging<24 Downloading packaging-23.1-py3-none-any.whl (48 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 2.4 MB/s eta 0:00:00 Collecting typing-extensions>=4.2.0 Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB) Collecting certifi>=2017.4.17 Downloading certifi-2023.5.7-py3-none-any.whl (156 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB ? eta 0:00:00 Collecting idna<4,>=2.5 Downloading idna-3.4-py3-none-any.whl (61 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 3.4 MB/s eta 0:00:00 Collecting chardet>=3.0.2 Downloading chardet-5.1.0-py3-none-any.whl (199 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.1/199.1 kB 4.1 MB/s eta 0:00:00 Collecting pycparser Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 3.5 MB/s eta 0:00:00 Collecting gitdb<5,>=4.0.1 Downloading gitdb-4.0.10-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.7/62.7 kB 1.7 MB/s eta 0:00:00 Collecting zipp>=0.5 Downloading zipp-3.15.0-py3-none-any.whl (6.8 kB) Collecting arrow Downloading arrow-1.2.3-py3-none-any.whl (66 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.4/66.4 kB 3.5 MB/s eta 0:00:00 Collecting text-unidecode>=1.3 Downloading text_unidecode-1.3-py2.py3-none-any.whl (78 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.2/78.2 kB ? eta 0:00:00 Collecting smmap<6,>=3.0.1 Downloading smmap-5.0.0-py3-none-any.whl (24 kB) Collecting python-dateutil>=2.7.0 Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 14.8 MB/s eta 0:00:00 Installing collected packages: text-unidecode, pytz, commonmark, zipp, watchdog, urllib3, typing-extensions, tenacity, tabulate, sqlparse, smmap, six, shellingham, pyyaml, python-slugify, pyjwt, pygments, pycparser, protobuf, pathspec, packaging, oauthlib, multidict, MarkupSafe, idna, frozenlist, entrypoints, colorama, cloudpickle, charset-normalizer, chardet, certifi, attrs, async-timeout, yarl, rich, requests, python-dateutil, pydantic, Jinja2, importlib-metadata, gitdb, click, cffi, binaryornot, aiosignal, typer, gitpython, databricks-cli, cryptography, arrow, aiohttp, mlflow-skinny, jinja2-time, cookiecutter, dbx WARNING: The script cmark.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script watchmedo.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script tabulate.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script sqlformat.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script slugify.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script pygmentize.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script normalizer.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script chardetect.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. DEPRECATION: databricks-cli is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559 Running setup.py install for databricks-cli ... done WARNING: The script mlflow.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script cookiecutter.exe is installed in 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. ERROR: Could not install packages due to an OSError: [WinError 206] The filename or extension is too long: 'C:\Users\redacted\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0\LocalCache\local-packages\Python310\site-packages\dbx\templates\projects\python_basic\render\{{cookiecutter.project_name}}\{{cookiecutter.project_slug}}\tasks'

renardeinside commented 1 year ago

hi @4kbnick ,

unfortunately this error is unrelated to dbx, it's related to the way you setup your python path.

I would recommend using a shorter path for Python env, i.e.:

C:\Users\<uname>\python\...

Please note that you'll run into the same error with pretty much any package with long names in it's structure.