search

microsoft / azdo-databricks

A set of Build and Release tasks for Building, Deploying and Testing Databricks notebooks
MIT License
25 stars 31 forks source link

Configure Databricks task does not fail even if "pip install databricks-cli" fails #14

Closed neerajanil closed 1 year ago

neerajanil commented 5 years ago

Right now "pip install databricks-cli" is failing because databricks-cli depends on tabulate>=0.7.7 and there is an open issue with tabulate 0.8.4 install failing on windows (link). But despite the failure to install databricks-cli, the "Configure Databricks" task still succeeds.

2019-09-25T01:58:16.2362991Z ##[section]Starting: Configure Databricks CLI 2019-09-25T01:58:16.2493382Z ============================================================================== 2019-09-25T01:58:16.2493505Z Task : Configure Databricks 2019-09-25T01:58:16.2493592Z Description : Configure Databricks CLI 2019-09-25T01:58:16.2493659Z Version : 0.5.2 2019-09-25T01:58:16.2493736Z Author : Microsoft DevLabs 2019-09-25T01:58:16.2493983Z Help : 2019-09-25T01:58:16.2494079Z ============================================================================== 2019-09-25T01:58:16.5084259Z [command]C:\hostedtoolcache\windows\Python\3.7.4\x64\python.exe -V 2019-09-25T01:58:16.5261893Z Python 3.7.4 2019-09-25T01:58:16.5279102Z Version: 3.7.4 2019-09-25T01:58:16.5279763Z 2019-09-25T01:58:16.5280285Z Python3 selected. Running... 2019-09-25T01:58:16.5323535Z [command]C:\Users\VssAdministrator\AppData\Roaming\Python\Python37\Scripts\pip.exe install databricks-cli 2019-09-25T01:58:20.0684365Z Collecting databricks-cli 2019-09-25T01:58:20.0685136Z Downloading https://files.pythonhosted.org/packages/51/0b/75dac581d98c493be74df97f3ea515c678da2e4be8cafbaf9cba9f01c309/databricks-cli-0.9.0.tar.gz (45kB) 2019-09-25T01:58:20.0685674Z Collecting click>=6.7 (from databricks-cli) 2019-09-25T01:58:20.0686018Z Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB) 2019-09-25T01:58:20.0686283Z Collecting requests>=2.17.3 (from databricks-cli) 2019-09-25T01:58:20.0686507Z Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB) 2019-09-25T01:58:20.0686750Z Collecting tabulate>=0.7.7 (from databricks-cli) 2019-09-25T01:58:20.0686992Z Downloading https://files.pythonhosted.org/packages/76/35/ae65ed1268d6e2a1be141723e5fffdf4a28e4f4e7c1e083709b308998f90/tabulate-0.8.4.tar.gz (45kB) 2019-09-25T01:58:20.0688048Z ERROR: Command errored out with exit status 1: 2019-09-25T01:58:20.0688475Z command: 'c:\hostedtoolcache\windows\python\3.7.4\x64\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\pip-install-bih6hjoh\\tabulate\\setup.py'"'"'; __file__='"'"'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\pip-install-bih6hjoh\\tabulate\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info 2019-09-25T01:58:20.0688826Z cwd: C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-bih6hjoh\tabulate\ 2019-09-25T01:58:20.0689034Z Complete output (7 lines): 2019-09-25T01:58:20.0689234Z Traceback (most recent call last): 2019-09-25T01:58:20.0689416Z File "<string>", line 1, in <module> 2019-09-25T01:58:20.0689642Z File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-bih6hjoh\tabulate\setup.py", line 25, in <module> 2019-09-25T01:58:20.0689849Z f.write(LONG_DESCRIPTION) 2019-09-25T01:58:20.0690071Z File "c:\hostedtoolcache\windows\python\3.7.4\x64\lib\encodings\cp1252.py", line 19, in encode 2019-09-25T01:58:20.0690275Z return codecs.charmap_encode(input,self.errors,encoding_table)[0] 2019-09-25T01:58:20.0690507Z UnicodeEncodeError: 'charmap' codec can't encode characters in position 5907-5924: character maps to <undefined> 2019-09-25T01:58:20.0690761Z ---------------------------------------- 2019-09-25T01:58:20.0690964Z ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. 2019-09-25T01:58:20.0691190Z WARNING: You are using pip version 19.2.2, however version 19.2.3 is available. 2019-09-25T01:58:20.0691394Z You should consider upgrading via the 'python -m pip install --upgrade pip' command. 2019-09-25T01:58:20.0691681Z Error while installing databricks-cli: ERROR: Command errored out with exit status 1: 2019-09-25T01:58:20.0694017Z command: 'c:\hostedtoolcache\windows\python\3.7.4\x64\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\pip-install-bih6hjoh\\tabulate\\setup.py'"'"'; __file__='"'"'C:\\Users\\VSSADM~1\\AppData\\Local\\Temp\\pip-install-bih6hjoh\\tabulate\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base pip-egg-info 2019-09-25T01:58:20.0694925Z cwd: C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-bih6hjoh\tabulate\ 2019-09-25T01:58:20.0695130Z Complete output (7 lines): 2019-09-25T01:58:20.0695333Z Traceback (most recent call last): 2019-09-25T01:58:20.0696031Z File "<string>", line 1, in <module> 2019-09-25T01:58:20.0696248Z File "C:\Users\VSSADM~1\AppData\Local\Temp\pip-install-bih6hjoh\tabulate\setup.py", line 25, in <module> 2019-09-25T01:58:20.0696484Z f.write(LONG_DESCRIPTION) 2019-09-25T01:58:20.0696686Z File "c:\hostedtoolcache\windows\python\3.7.4\x64\lib\encodings\cp1252.py", line 19, in encode 2019-09-25T01:58:20.0696911Z return codecs.charmap_encode(input,self.errors,encoding_table)[0] 2019-09-25T01:58:20.0697122Z UnicodeEncodeError: 'charmap' codec can't encode characters in position 5907-5924: character maps to <undefined> 2019-09-25T01:58:20.0697350Z ---------------------------------------- 2019-09-25T01:58:20.0697904Z ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. 2019-09-25T01:58:20.0698145Z WARNING: You are using pip version 19.2.2, however version 19.2.3 is available. 2019-09-25T01:58:20.0698584Z You should consider upgrading via the 'python -m pip install --upgrade pip' command. 2019-09-25T01:58:20.0698773Z 2019-09-25T01:58:20.0713289Z Writing databricks-cli configuration to file: C:\Users\VssAdministrator\.databrickscfg 2019-09-25T01:58:20.0786544Z ##[section]Finishing: Configure Databricks CLI

bindsi commented 5 years ago

Hi,

we faced the same problem but I still had the logs of a recent successfully-running release. So, I noticed what packages to load that are included in the databricks-cli package.

As workaround you can now add a Command Line task in your pipeline prior to the "Configure Databricks" task with the following commands: pip install click pip install requests pip install tabulate==0.8.3 pip install six pip install configparser pip install chardet==3.0.4 pip install certifi pip install urllib3==1.21.1 pip install idna==2.8

Afterwards you have to run the "Configure Databricks" though. It will recognize now that tabulate is already installed with version 0.8.3 that fulfills the required version (>=0.7.7). So, it doesn´t fail at that step and continues execution with preparing the agent for Databricks.

Have fun!!!