search

poise / poise-python

A Chef cookbook to provide a unified interface for installing Python, managing Python packages, and creating virtualenvs.
Apache License 2.0
124 stars 108 forks source link

Error executing action `install` on resource 'python_package[setuptools]' #133

Open f9n opened 6 years ago

f9n commented 6 years ago 
$ chef generate cookbook install-python
$ cd install-python
$ vim recipes/default.rb
python_runtime '3'
$ vim .kitchen.yml
...
platforms:
  - name: ubuntu-16.04
...
$ vim metadata.rb   # Add poise python
...
depends 'poise-python'
...
$ kitchen converge ubuntu
.....
Installing chef 
       installing with dpkg...
       Selecting previously unselected package chef.
(Reading database ... 65145 files and directories currently installed.)
       Preparing to unpack .../cache/chef_14.5.33-1_amd64.deb ...
       Unpacking chef (14.5.33-1) ...
       Setting up chef (14.5.33-1) ...
       Thank you for installing Chef!
       Transferring files to <default-ubuntu-1604>
       Starting Chef Client, version 14.5.33
       Creating a new client identity for default-ubuntu-1604 using the validator key.
       resolving cookbooks for run list: ["install-python::default"]
       Synchronizing Cookbooks:
         - poise-python (1.7.0)
         - install-python (0.1.0)
         - poise-languages (2.1.2)
         - poise (2.8.2)
         - poise-archive (1.5.0)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       Converging 1 resources
       Recipe: install-python::default
         * python_runtime[3] action install
           * poise_languages_system[python3.5] action install
             - install version 3.5.2-2ubuntu0~16.04.4 of package python3.5-dev
           * apt_package[python3.5, python3.5-dev] action nothing (skipped due to action :nothing)

         * python_runtime_pip[3] action install
           - Bootstrapping pip latest from https://bootstrap.pypa.io/get-pip.py
         * python_package[setuptools] action install

           ================================================================================
           Error executing action `install` on resource 'python_package[setuptools]'
           ================================================================================

           Mixlib::ShellOut::ShellCommandFailed
           ------------------------------------
           Expected process to exit with [0], but received '1'
           ---- Begin output of ["/usr/bin/python3.5", "-", "setuptools"] ----
           STDOUT: 
           STDERR: The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
           Traceback (most recent call last):
             File "<stdin>", line 42, in <module>
           AttributeError: type object 'InstallRequirement' has no attribute 'from_line'
           ---- End output of ["/usr/bin/python3.5", "-", "setuptools"] ----
           Ran ["/usr/bin/python3.5", "-", "setuptools"] returned 1

           Cookbook Trace:
           ---------------
           /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:207:in `tap'
           /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:207:in `language_command_shell_out!'
           /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:222:in `block in language_command_mixin'
           /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:329:in `pip_command'
           /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:353:in `pip_outdated'
           /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:221:in `check_package_versions'
           /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:200:in `load_current_resource'
           /tmp/kitchen/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
           /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:50:in `action_install'

           Resource Declaration:
           ---------------------
           # In /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb

           138:         python_package 'setuptools' do
           139:           parent_python new_resource
           140:           version setuptools_version if setuptools_version.is_a?(String)
           141:         end
           142:       end

           Compiled Resource:
           ------------------
           # Declared in /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:138:in `install_setuptools'

           python_package("setuptools") do
             package_name "setuptools"
             action [:install]
             default_guard_interpreter :default
             declared_type :python_package
             cookbook_name "install-python"
             parent_python python_runtime[3]
             timeout 900
           end

           System Info:
           ------------
           chef_version=14.5.33
           platform=ubuntu
           platform_version=16.04
           ruby=ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
           program_name=/opt/chef/bin/chef-client
           executable=/opt/chef/bin/chef-client

         ================================================================================
         Error executing action `install` on resource 'python_runtime[3]'
         ================================================================================

         Mixlib::ShellOut::ShellCommandFailed
         ------------------------------------
         python_package[setuptools] (/tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb line 138) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
         ---- Begin output of ["/usr/bin/python3.5", "-", "setuptools"] ----
         STDOUT: 
         STDERR: The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
         Traceback (most recent call last):
           File "<stdin>", line 42, in <module>
         AttributeError: type object 'InstallRequirement' has no attribute 'from_line'
         ---- End output of ["/usr/bin/python3.5", "-", "setuptools"] ----
         Ran ["/usr/bin/python3.5", "-", "setuptools"] returned 1

         Cookbook Trace:
         ---------------
         /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:207:in `tap'
         /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:207:in `language_command_shell_out!'
         /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:222:in `block in language_command_mixin'
         /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:329:in `pip_command'
         /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:353:in `pip_outdated'
         /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:221:in `check_package_versions'
         /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:200:in `load_current_resource'
         /tmp/kitchen/cache/cookbooks/poise/files/halite_gem/poise/helpers/notifying_block.rb:69:in `notifying_block'
         /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb:50:in `action_install'

         Resource Declaration:
         ---------------------
         # In /tmp/kitchen/cache/cookbooks/install-python/recipes/default.rb

           7: python_runtime '3'

         Compiled Resource:
         ------------------
         # Declared in /tmp/kitchen/cache/cookbooks/install-python/recipes/default.rb:7:in `from_file'

         python_runtime("3") do
           action [:install]
           updated true
           updated_by_last_action true
           default_guard_interpreter :default
           subresources [python_runtime_pip[3], python_package[setuptools], python_package[wheel]]
           declared_type :python_runtime
           cookbook_name "install-python"
           recipe_name "default"
           get_pip_url "https://bootstrap.pypa.io/get-pip.py"
           pip_version true
           setuptools_version true
           version "3"
           virtualenv_version true
           wheel_version true
         end

         System Info:
         ------------
         chef_version=14.5.33
         platform=ubuntu
         platform_version=16.04
         ruby=ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]
         program_name=/opt/chef/bin/chef-client
         executable=/opt/chef/bin/chef-client

       Running handlers:
       [2018-10-05T12:52:36+00:00] ERROR: Running exception handlers
       [2018-10-05T12:52:36+00:00] ERROR: Running exception handlers
       Running handlers complete
       [2018-10-05T12:52:36+00:00] ERROR: Exception handlers complete
       [2018-10-05T12:52:36+00:00] ERROR: Exception handlers complete
       Chef Client failed. 3 resources updated in 22 seconds
       [2018-10-05T12:52:36+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2018-10-05T12:52:36+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2018-10-05T12:52:36+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
       [2018-10-05T12:52:36+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
       [2018-10-05T12:52:36+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: python_runtime[3] (install-python::default line 7) had an error: Mixlib::ShellOut::ShellCommandFailed: python_package[setuptools] (/tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb line 138) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
       ---- Begin output of ["/usr/bin/python3.5", "-", "setuptools"] ----
       STDOUT: 
       STDERR: The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
       Traceback (most recent call last):
         File "<stdin>", line 42, in <module>
       AttributeError: type object 'InstallRequirement' has no attribute 'from_line'
       ---- End output of ["/usr/bin/python3.5", "-", "setuptools"] ----
       Ran ["/usr/bin/python3.5", "-", "setuptools"] returned 1
       [2018-10-05T12:52:36+00:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: python_runtime[3] (install-python::default line 7) had an error: Mixlib::ShellOut::ShellCommandFailed: python_package[setuptools] (/tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/python_providers/base.rb line 138) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
       ---- Begin output of ["/usr/bin/python3.5", "-", "setuptools"] ----
       STDOUT: 
       STDERR: The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
       Traceback (most recent call last):
         File "<stdin>", line 42, in <module>
       AttributeError: type object 'InstallRequirement' has no attribute 'from_line'
       ---- End output of ["/usr/bin/python3.5", "-", "setuptools"] ----
       Ran ["/usr/bin/python3.5", "-", "setuptools"] returned 1
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Converge failed on instance <default-ubuntu-1604>.  Please see .kitchen/logs/default-ubuntu-1604.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

I don't understand, it worked in the past.

devsibwarra commented 6 years ago

https://pypi.org/project/pip/#history shows 18.1 was released a couple hours ago. Rolling back to 18.0 via pip install pip==18.0 resolved this error for me

knightorc commented 6 years ago

Interestingly enough. Does not impact CentOS 6.x but does impact CentOS 7.x

fozboz commented 6 years ago

Doing

python_runtime '3' do
  pip_version '18.0'
end

does not work because of https://github.com/pypa/pip/issues/5857 which is probably the cause of original issue, given that they occurred at the same time.

nxf5025 commented 6 years ago

Any workaround we can implement into the actual cookbook?

coderanger commented 6 years ago

You can try setting the get_pip_url property to an older version of the script, either one you host yourself or one of the legacy scripts like https://bootstrap.pypa.io/3.3/get-pip.py.

jeffbyrnes commented 6 years ago

This works for me:

python_runtime '3' do
  get_pip_url 'https://github.com/pypa/get-pip/raw/f88ab195ecdf2f0001ed21443e247fb32265cabb/get-pip.py'
  pip_version '18.0'
end
coderanger commented 6 years ago

Hopefully this will be fixed upstream shortly. I'll monitor the upstream ticket and see how it goes.

aungger commented 6 years ago

I am getting the same thing for python_runtime '2', any tips for this? plz and ty!

coderanger commented 6 years ago

@aungger Same fix, the current default get-pip.py script is broken.

eriepasquare commented 6 years ago

if your dependency on poise-python is through a wrapped cookbook and you can't change the pip_version or get_pip_url parameters of the resource... we had success implementing the following workaround in our wrapper cookbook:

edit_resource(:python_runtime, '2') do
  pip_version '18.0'
  get_pip_url 'https://github.com/pypa/get-pip/raw/f88ab195ecdf2f0001ed21443e247fb32265cabb/get-pip.py'
end
nkovacne commented 6 years ago

If you're using a virtualenv resource, you also needed to set the pip_version and get_pip_url in the python_virtualenv resource:

python_virtualenv '/opt/venv' do
  pip_version '18.0'
  get_pip_url 'https://github.com/pypa/get-pip/raw/f88ab195ecdf2f0001ed21443e247fb32265cabb/get-pip.py'
  action :create
end
hartmantis commented 6 years ago

Update: They merged a fix to get-pip but it seems to only resolve the part where get_pip_url had to be specified. So

python_virtualenv '/opt/venv' do
  pip_version '18.0'
end

works now, but blows up without the pip_version when it tries to install setuptools.

lopezm1 commented 6 years ago

confirmed, overriding the attribute will force poise-python to use pip 18.0 and our cookbooks now work as expected


"poise-python": {
      "options": {
        "pip_version": "18.0"
      }       
}
nikbhadane commented 6 years ago

This issue still appears in wrapper cookbook. Can someone please help?

jmd9019 commented 6 years ago

python_runtime '3' failing on ubuntu 18 `

mike10010100 commented 6 years ago

@lopezm1 Good find! This was the quickest and easiest way to get around this error for me.

lopezm1 commented 6 years ago

This issue still appears in wrapper cookbook. Can someone please help?

I was having this issue if a previous chef run was cached. Have you tried clearing the cache or running the cookbook on a fresh server?

nikbhadane commented 6 years ago

I was having this issue if a previous chef run was cached. Have you tried clearing the cache or running the cookbook on a fresh server?

Right, By overiding the pip_version in attribute file of wrapper cookbook resolving the issue now. like

override['poise-python']['options']['pip_version'] = '9.0.3'

thanks for the help.

jharbott commented 6 years ago

Just a note to some folks for whom the workarounds above might show no effect: If pip==18.1 is already installed (and your resource is using the provider: system option), the broken script is executed with that version and fails before pip can be downgraded. So in that case you may need to run pip install pip=18.0 manually once as additional workaround.

Let's hope the patch in #134 gets merged soon. A long-term solution that does not depend on any pip internals would be even nicer.

adamfeuer commented 6 years ago

As a workaround, since we need this to work to deploy our software using poise-python, I forked into our own copy of this repo, used Rake to build a version from the tip of master that has the fix for using pip 18.1, and saved it to a branch (just a convenient place to put the release). I put this line in my Berksfile:

cookbook "poise-python", "1.7.1", git:'https://github.com/twistle/poise-python.git', branch: 'fix/pip-18-1-problems'

Then when 1.7.1 is really released, I will switch back to using the version from the Chef Supermarket.

Posting this here in case others might find it helpful.

You're welcome to use this or fork it and use it from your own repo.

sunilkumarmohanty commented 6 years ago

when is the 1.7.1 version getting released?

kpmueller commented 6 years ago

Why is 1.7.1 not out yet? :(

coderanger commented 6 years ago

I explained this on another issue, but my CI system is pretty much busted so I have no safe way to vet a release. No one seems to be willing to pay for the time it would take to fix and I’m no longer doing pro-bono Chef work when I can avoid it.

adamfeuer commented 6 years ago

@coderanger How could we collaborate to help...? At Twistle we're using this code as part of our chef build for a production project, so maybe there's a way to help. It seems like others are in similar positions. Would you be willing to point me at the other issue where you explained the problem?

adamfeuer commented 6 years ago

@coderanger P.S. I get not wanting to do volunteer work using chef. I feel the same way! 🙂

coderanger commented 6 years ago

I’m on my phone so easier to just summarize here again: the first issue is that since my last release, Travis CI has cut the maximum time for builds from two hours to one, which means my tests get killed. But I’m not inclined to fix that because the second issue is my (very generous) infrastructure sponsor will sadly be terminating their open source sponsorship program so the infra that my tests actually run on will go away in a few more weeks. So to get my CI back into a healthy state would require a pretty major rebuild, probably switching providers as well as reworking things to not use the dedicated infra (which will make the tests even slower).

ramereth commented 6 years ago

@coderanger I might be able to help with this via the OSUOSL. PM me on slack if you're interested.

adamfeuer commented 6 years ago

@ramereth Cool!

@coderanger I might be able to help with moving things over to a new provider if you need more help.

Maybe there are others that could pitch in too?