search

bahchis / airflow-cookbook

Airflow workflow management platform chef cookbook.
Apache License 2.0
68 stars 49 forks source link

Unable to install airflow_package via chef in kitchen environment #24

Closed Idun0 closed 6 years ago

Idun0 commented 6 years ago

Ive been trying to bring up a clean kitchen instance for a couple days now, i was getting a setuptools issue until i was able to get past that with attribute assigning the pip_version to 18.0. But i can not seem to get past this error:

`* python_package[apache-airflow] action install[2018-10-18T15:19:20+00:00] INFO: Processing python_package[apache-airflow] action install (airflow::packages line 67)

     ================================================================================
     Error executing action `install` on resource 'python_package[apache-airflow]'
     ================================================================================

     Mixlib::ShellOut::ShellCommandFailed
     ------------------------------------
     Expected process to exit with [0], but received '1'
     ---- Begin output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
     STDOUT: Collecting apache-airflow==1.10.0
       Downloading https://files.pythonhosted.org/packages/da/2a/6e9efcd40193850e2f636c7306eede2ff5607aa9f81ff9f7a151d9b13ff8/apache-airflow-1.10.0.tar.gz (4.3MB)
         Complete output from command python setup.py egg_info:
         Traceback (most recent call last):
    File "<string>", line 20, in <module>
    File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 393, in <module>
      do_setup()
    File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 258, in do_setup
      verify_gpl_dependency()
    File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 49, in verify_gpl_dependency
      raise RuntimeError("By default one of Airflow's dependencies installs a GPL "
         RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE

         ----------------------------------------
     STDERR: The directory '/home/vagrant/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
     The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
     You are using pip version 6.1.0, however version 18.1 is available.
     You should consider upgrading via the 'pip install --upgrade pip' command.
     The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
     /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
       InsecurePlatformWarning
         Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-jtxQR7/apache-airflow
     ---- End output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
     Ran ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] returned 1

     Cookbook Trace:
     ---------------
     /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:201:in `tap'
     /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:201:in `language_command_shell_out!'
     /tmp/kitchen/cache/cookbooks/poise-languages/files/halite_gem/poise_languages/command/mixin.rb:216:in `block in language_command_mixin'
     /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:282:in `pip_command'
     /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:296:in `pip_install'
     /tmp/kitchen/cache/cookbooks/poise-python/files/halite_gem/poise_python/resources/python_package.rb:209:in `install_package'

     Resource Declaration:
     ---------------------
     # In /tmp/kitchen/cache/cookbooks/airflow/recipes/packages.rb

      67: python_package node['airflow']['airflow_package'] do
      68:   version node['airflow']['version']
      69: end
      70:

     Compiled Resource:
     ------------------
     # Declared in /tmp/kitchen/cache/cookbooks/airflow/recipes/packages.rb:67:in `from_file'

     python_package("apache-airflow") do
       package_name "apache-airflow"
       action [:install]
       retries 0
       retry_delay 2
       default_guard_interpreter :default
       declared_type :python_package
       cookbook_name "airflow"
       recipe_name "packages"
       parent_python python_runtime[2]
       timeout 900
     end

     Platform:
     ---------
     x86_64-linux

   [2018-10-18T15:19:25+00:00] INFO: Running queued delayed notifications before re-raising exception

   Running handlers:
   [2018-10-18T15:19:25+00:00] ERROR: Running exception handlers
   Running handlers complete
   [2018-10-18T15:19:25+00:00] ERROR: Exception handlers complete
   Chef Client failed. 38 resources updated in 02 minutes 31 seconds
   [2018-10-18T15:19:25+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
   [2018-10-18T15:19:25+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
   [2018-10-18T15:19:25+00:00] ERROR: python_package[apache-airflow] (airflow::packages line 67) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
   ---- Begin output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
   STDOUT: Collecting apache-airflow==1.10.0
     Downloading https://files.pythonhosted.org/packages/da/2a/6e9efcd40193850e2f636c7306eede2ff5607aa9f81ff9f7a151d9b13ff8/apache-airflow-1.10.0.tar.gz (4.3MB)
       Complete output from command python setup.py egg_info:
       Traceback (most recent call last):
         File "<string>", line 20, in <module>
         File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 393, in <module>
    do_setup()
         File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 258, in do_setup
    verify_gpl_dependency()
         File "/tmp/pip-build-jtxQR7/apache-airflow/setup.py", line 49, in verify_gpl_dependency
    raise RuntimeError("By default one of Airflow's dependencies installs a GPL "
       RuntimeError: By default one of Airflow's dependencies installs a GPL dependency (unidecode). To avoid this dependency set SLUGIFY_USES_TEXT_UNIDECODE=yes in your environment when you install or upgrade Airflow. To force installing the GPL version set AIRFLOW_GPL_UNIDECODE

       ----------------------------------------
   STDERR: The directory '/home/vagrant/.cache/pip/log' or its parent directory is not owned by the current user and the debug log has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
   The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
   You are using pip version 6.1.0, however version 18.1 is available.
   You should consider upgrading via the 'pip install --upgrade pip' command.
   The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
   /usr/local/lib/python2.7/dist-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:79: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
     InsecurePlatformWarning
       Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-jtxQR7/apache-airflow
   ---- End output of ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] ----
   Ran ["/usr/bin/python2.7", "-m", "pip.__main__", "install", "apache-airflow==1.10.0"] returned 1
   [2018-10-18T15:19:25+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)`
cloneluke commented 6 years ago

I have not had time to test this cookbook with airflow 1.10 yet.

I do know that is a known issue with airflow 1.10

https://issues.apache.org/jira/browse/AIRFLOW-3117

also discussed at length here: apache-airflow.slack.com

I will try to get a fix for this, it should be pretty simple, just add magic_shell_environment 'SLUGIFY_USES_TEXT_UNIDECODE' do value 'yes' end

Idun0 commented 6 years ago

@cloneluke Thank you very much, I really appreciate it!!!

cloneluke commented 6 years ago

add magic_shell_environment 'SLUGIFY_USES_TEXT_UNIDECODE' do value 'yes' end

as a resource before you call airflow cookbook, or manually set the env var to test it before you run the recipe/cookbook let me know if that works

Idun0 commented 6 years ago

This worked thank you

cloneluke commented 5 years ago

https://github.com/bahchis/airflow-cookbook/pull/26