Ubuntu 16.04 failure to provision using enova-python.

thomasdziedzic-pd commented 4 years ago

I have the following lines in my recipe to try to install python2:

node.default['poise-python']['options']['pip_version'] = '18.0'
include_recipe 'poise-python'

When I tried to provision a new host, I got the following error message:

[2020-02-25T23:10:12Z] 34.213.177.81   * python_runtime[2] action install[2020-02-25T23:10:12+00:00] INFO: Processing python_runtime[2] action install (poise-python::default line 26)

[2020-02-25T23:10:13Z] 34.213.177.81 

[2020-02-25T23:10:13Z] 34.213.177.81     * poise_languages_system[python2.7] action install[2020-02-25T23:10:13+00:00] INFO: Processing poise_languages_system[python2.7] action install (/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/mixin.rb line 32)

[2020-02-25T23:10:19Z] 34.213.177.81 [2020-02-25T23:10:19+00:00] INFO: apt_package[python2.7, python2.7-dev] installed ["python2.7-dev"] at ["2.7.12-1ubuntu0~16.04.9"]

[2020-02-25T23:10:19Z] 34.213.177.81 

[2020-02-25T23:10:19Z] 34.213.177.81       [32m- install version 2.7.12-1ubuntu0~16.04.9 of package python2.7-dev[0m

[2020-02-25T23:10:19Z] 34.213.177.81 [0m    * apt_package[python2.7, python2.7-dev] action nothing[2020-02-25T23:10:19+00:00] INFO: Processing apt_package[python2.7, python2.7-dev] action nothing (/var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/system/resource.rb line 144)

[2020-02-25T23:10:19Z] 34.213.177.81  (skipped due to action :nothing)

[2020-02-25T23:10:19Z] 34.213.177.81   

[2020-02-25T23:10:19Z] 34.213.177.81 [0m[2020-02-25T23:10:19+00:00] INFO: [python_runtime[2]] distutils installer - only valid on ubuntu for python 3

[2020-02-25T23:10:19Z] 34.213.177.81   * python_runtime_pip[2] action install[2020-02-25T23:10:19+00:00] INFO: Processing python_runtime_pip[2] action install (/var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb line 140)

[2020-02-25T23:10:26Z] 34.213.177.81 

[2020-02-25T23:10:26Z] 34.213.177.81     [32m- Bootstrapping pip latest from https://bootstrap.pypa.io/get-pip.py[0m

[2020-02-25T23:10:26Z] 34.213.177.81 [0m  * python_package[setuptools] action install[2020-02-25T23:10:26+00:00] INFO: Processing python_package[setuptools] action install (/var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb line 160)

[2020-02-25T23:10:27Z] 34.213.177.81 checking for ["setuptools"]

[2020-02-25T23:10:28Z] 34.213.177.81 

[2020-02-25T23:10:28Z] 34.213.177.81     [0m

[2020-02-25T23:10:28Z] 34.213.177.81     ================================================================================[0m

[2020-02-25T23:10:28Z] 34.213.177.81     [31mError executing action `install` on resource 'python_package[setuptools]'[0m

[2020-02-25T23:10:28Z] 34.213.177.81     ================================================================================[0m

[2020-02-25T23:10:28Z] 34.213.177.81     

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    Mixlib::ShellOut::ShellCommandFailed[0m

[2020-02-25T23:10:28Z] 34.213.177.81     ------------------------------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81     Expected process to exit with [0], but received '1'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    ---- Begin output of ["/usr/bin/python2.7", "-", "setuptools"] ----

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    STDOUT: 

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    STDERR: Traceback (most recent call last):

[2020-02-25T23:10:28Z] 34.213.177.81 [0m      File "<stdin>", line 22, in <module>

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    ImportError: cannot import name InstallCommand

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    ---- End output of ["/usr/bin/python2.7", "-", "setuptools"] ----

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    Ran ["/usr/bin/python2.7", "-", "setuptools"] returned 1[0m

[2020-02-25T23:10:28Z] 34.213.177.81     

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    Cookbook Trace:[0m

[2020-02-25T23:10:28Z] 34.213.177.81     ---------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81     /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:207:in `tap'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:207:in `language_command_shell_out!'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:222:in `block in language_command_mixin'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:343:in `pip_command'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:368:in `pip_outdated'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:235:in `check_package_versions'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:214:in `load_current_resource'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:50:in `action_install'[0m

[2020-02-25T23:10:28Z] 34.213.177.81     

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    Resource Declaration:[0m

[2020-02-25T23:10:28Z] 34.213.177.81     ---------------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81     # In /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    160:         python_package 'setuptools' do

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    161:           parent_python new_resource

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    162:           version setuptools_version if setuptools_version.is_a?(String)

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    163:         end

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    164:       end

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    Compiled Resource:[0m

[2020-02-25T23:10:28Z] 34.213.177.81     ------------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81     # Declared in /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:160:in `install_setuptools'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    python_package("setuptools") do

[2020-02-25T23:10:28Z] 34.213.177.81 [0m      package_name "setuptools"

[2020-02-25T23:10:28Z] 34.213.177.81 [0m      action [:install]

[2020-02-25T23:10:28Z] 34.213.177.81 [0m      default_guard_interpreter :default

[2020-02-25T23:10:28Z] 34.213.177.81 [0m      declared_type :python_package

[2020-02-25T23:10:28Z] 34.213.177.81 [0m      cookbook_name "poise-python"

[2020-02-25T23:10:28Z] 34.213.177.81 [0m      parent_python python_runtime[2]

[2020-02-25T23:10:28Z] 34.213.177.81 [0m      timeout 900

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    end

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    System Info:[0m

[2020-02-25T23:10:28Z] 34.213.177.81     ------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81     chef_version=14.13.11

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    platform=ubuntu

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    platform_version=16.04

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    ruby=ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux]

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    program_name=/opt/chef/bin/chef-client

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    executable=/opt/chef/bin/chef-client[0m

[2020-02-25T23:10:28Z] 34.213.177.81     

[2020-02-25T23:10:28Z] 34.213.177.81 [0m[2020-02-25T23:10:28+00:00] INFO: Running queued delayed notifications before re-raising exception

[2020-02-25T23:10:28Z] 34.213.177.81   [0m

[2020-02-25T23:10:28Z] 34.213.177.81   ================================================================================[0m

[2020-02-25T23:10:28Z] 34.213.177.81   [31mError executing action `install` on resource 'python_runtime[2]'[0m

[2020-02-25T23:10:28Z] 34.213.177.81   ================================================================================[0m

[2020-02-25T23:10:28Z] 34.213.177.81   

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  Mixlib::ShellOut::ShellCommandFailed[0m

[2020-02-25T23:10:28Z] 34.213.177.81   ------------------------------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81   python_package[setuptools] (/var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb line 160) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  ---- Begin output of ["/usr/bin/python2.7", "-", "setuptools"] ----

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  STDOUT: 

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  STDERR: Traceback (most recent call last):

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    File "<stdin>", line 22, in <module>

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  ImportError: cannot import name InstallCommand

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  ---- End output of ["/usr/bin/python2.7", "-", "setuptools"] ----

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  Ran ["/usr/bin/python2.7", "-", "setuptools"] returned 1[0m

[2020-02-25T23:10:28Z] 34.213.177.81   

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  Cookbook Trace:[0m

[2020-02-25T23:10:28Z] 34.213.177.81   ---------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81   /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:207:in `tap'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:207:in `language_command_shell_out!'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  /var/chef/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:222:in `block in language_command_mixin'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:343:in `pip_command'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:368:in `pip_outdated'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:235:in `check_package_versions'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:214:in `load_current_resource'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  /var/chef/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  /var/chef/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:50:in `action_install'[0m

[2020-02-25T23:10:28Z] 34.213.177.81   

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  Resource Declaration:[0m

[2020-02-25T23:10:28Z] 34.213.177.81   ---------------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81   # In /var/chef/cache/cookbooks/poise-python/recipes/default.rb

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  

[2020-02-25T23:10:28Z] 34.213.177.81 [0m   26: python_runtime '2' if node['poise-python']['install_python2']

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  Compiled Resource:[0m

[2020-02-25T23:10:28Z] 34.213.177.81   ------------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81   # Declared in /var/chef/cache/cookbooks/poise-python/recipes/default.rb:26:in `from_file'

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  python_runtime("2") do

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    action [:install]

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    updated true

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    updated_by_last_action true

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    default_guard_interpreter :default

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    subresources [python_virtualenv[/opt/pd-analytics/shared/python-env-2], python_runtime_pip[2], python_package[setuptools], python_package[wheel], python_package[virtualenv]]

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    declared_type :python_runtime

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    cookbook_name "poise-python"

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    recipe_name "default"

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    get_pip_url "https://bootstrap.pypa.io/get-pip.py"

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    pip_version true

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    setuptools_version true

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    version "2"

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    virtualenv_version true

[2020-02-25T23:10:28Z] 34.213.177.81 [0m    wheel_version true

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  end

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  System Info:[0m

[2020-02-25T23:10:28Z] 34.213.177.81   ------------[0m

[2020-02-25T23:10:28Z] 34.213.177.81   chef_version=14.13.11

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  platform=ubuntu

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  platform_version=16.04

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  ruby=ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux]

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  program_name=/opt/chef/bin/chef-client

[2020-02-25T23:10:28Z] 34.213.177.81 [0m  executable=/opt/chef/bin/chef-client[0m

This might be related to https://github.com/enova/enova-python/pull/3#issuecomment-520536318 since they had an error in the CI during setuptools install as well.

Thanks for your time!

thomasdziedzic-pd commented 4 years ago

I can reproduce the issue with the local kitchen tests using kitchen test Python3and2-pip18-ubuntu-1604, I will attach the full log, Python3and2-pip18-ubuntu-1604.log

The relevant message from the log is at:

       [2020-02-26T15:30:52+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: python_runtime[3] (poise-python::default line 25) had an error: Mixlib::ShellOut::ShellCommandFailed: python_package[setuptools] (/tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb line 160) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
       ---- Begin output of ["/usr/bin/python3.5", "-", "setuptools"] ----
       STDOUT:
       STDERR: Traceback (most recent call last):
         File "<stdin>", line 17, in <module>
       ImportError: No module named 'pip.commands'

       During handling of the above exception, another exception occurred:

       Traceback (most recent call last):
         File "<stdin>", line 22, in <module>
       ImportError: cannot import name 'InstallCommand'
       ---- End output of ["/usr/bin/python3.5", "-", "setuptools"] ----
       Ran ["/usr/bin/python3.5", "-", "setuptools"] returned 1

which is the same error message that I get when trying to use the cookbook.

thomasdziedzic-pd commented 4 years ago

I applied the patch located at https://github.com/shareaholic/poise-python/commit/929195a7381afd50c458edb4bfe5f86b1d965dad and it seems to have made the job succeed.

mpescetto commented 4 years ago

Hi @thomasdziedzic-pd thanks for reporting the issue. If you have time to submit a PR that would be amazing, otherwise I can work on integrating your changes to this cookbook sometime next week. Also, you might want to remove your public IP from the logs in the description if you care about such things.

thomasdziedzic-pd commented 4 years ago

. If you have time to submit a PR that would be amazing

I'm trying to figure out some permissions issue with forking cookbooks, if I manage to figure it out I will try and submit one!

Also, you might want to remove your public IP from the logs in the description if you care about such things.

The machine was already decommissioned by the time I posted the logs.

thomasdziedzic-pd commented 4 years ago

As a follow up to:

I'm trying to figure out some permissions issue with forking cookbooks, if I manage to figure it out I will try and submit one!

I wasn't able to resolve this so I wont be submitting a PR unfortunately.

pescetto commented 4 years ago

@thomasdziedzic-pd super sorry for the delay on this. I'll follow up by end of month. My apologies.

ju2wheels commented 4 years ago

Here is my first round fix for this:

https://github.com/tvision-insights/enova-python/blob/1.8.11/files/halite_gem/poise_python/resources/python_package.rb

I overlayed ndpgroup/poise-python changes over your generated cookbook in addition to fixing this bug, but the hack script changes in that should be able to be grabbed without also pulling in the other changes.

ju2wheels commented 3 years ago

I pushed another fix for the hack script as 1.8.12 on our fork. I have manually tested this version of the hack script against all pip versions from 8.1.1 to 20.3.3 using Python 2.7 to address a few more bugs on pip 19.3.x and higher.

enova / enova-python

Ubuntu 16.04 failure to provision using enova-python. #4