search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.15k stars 5.47k forks source link

Unable to manage file: 'handle' in 2015.2.0 #23103

Closed h0jeZvgoxFepBQ2C closed 9 years ago

h0jeZvgoxFepBQ2C commented 9 years ago

I want to download the couchdb sources and extract them, but always fail with:

Unable to manage file: 'handle'

See the reproducable problem below:

# salt-ssh 'myserver' state.highstate -i  test=False

myserver:
  Name: /tmp/couchdb - Function: file.directory - Result: Clean
----------
          ID: test_source
    Function: archive.extracted
        Name: /tmp/couchdb
      Result: False
     Comment: Unable to manage file: 'handle'
     Started: 20:11:54.891882
    Duration: 469.074 ms
     Changes:

Summary
------------
Succeeded: 1
Failed:    1
------------
Total states run:     2
OSX: 10.9.5

# salt --versions-report
           Salt: 2015.2.0
         Python: 2.7.5 (default, Mar  9 2014, 22:15:05)
         Jinja2: 2.7.3
       M2Crypto: 0.22
 msgpack-python: 0.4.2
   msgpack-pure: Not Installed
       pycrypto: 2.6
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.3.1
           RAET: Not Installed
            ZMQ: 4.0.5
           Mako: Not Installed

Here is a example:

#test.sls

test_tmp:
  file.directory:
    - name: /tmp/couchdb
    - makedirs: True

test_source:
  archive.extracted:
    - name: /tmp/couchdb
    - source: 'https://www.apache.org/dist/couchdb/source/1.5.1/apache-couchdb-1.5.1.tar.gz'
    - source_hash: 'https://www.apache.org/dist/couchdb/source/1.5.1/apache-couchdb-1.5.1.tar.gz.md5'
    - archive_format: tar
    - if_missing: /tmp/couchdb/apache-couchdb-1.5.1
    - tar_options: zv
    - require:
      - file: test_tmp
jfindlay commented 9 years ago

@lichtamberg, is there anything relevant in the master or salt-ssh debug logs? You can find the master log at /var/log/salt/master and salt-ssh logs when you append -l debug. Thanks.

h0jeZvgoxFepBQ2C commented 9 years ago

I tried now the git version (branch 2015.2), and I'm getting now this error, but I'm not sure if this is because i don't have all dependencies installed correctly:

I can't find anything usefull in the logs/debug.. but I'm still investigating..

myserver:
  Name: /tmp/couchdb - Function: file.directory - Result: Clean
----------
          ID: test_source
    Function: archive.extracted
        Name: /tmp/couchdb
      Result: False
     Comment: Unable to manage file: 'Module_six_moves_urllib' object has no attribute 'Request'
     Started: 16:41:25.648379
    Duration: 982.9 ms
     Changes:

Summary
------------
Succeeded: 1
Failed:    1
------------
Total states run:     2
jfindlay commented 9 years ago

That comment looks like the final statement in a python stack trace which shouldn't happen, obviously, but if it did should at least appear in the log if it doesn't already.

h0jeZvgoxFepBQ2C commented 9 years ago

Ok, i found now this line in the debug messages (but I'm still looking for the source of this exception):

[CRITICAL] Unable to import msgpack or msgpack_pure python modules
jfindlay commented 9 years ago

That looks like a missing dependency. What is the output of salt --versions-report?

h0jeZvgoxFepBQ2C commented 9 years ago 
# salt-ssh 'myserver' state.highstate -i  test=False

myserver:
  Name: /tmp/couchdb - Function: file.directory - Result: Clean
----------
          ID: test_source
    Function: archive.extracted
        Name: /tmp/couchdb
      Result: False
     Comment: Unable to manage file: 'Module_six_moves_urllib' object has no attribute 'Request'
     Started: 17:18:00.362519
    Duration: 993.738 ms
     Changes:

Summary
------------
Succeeded: 1
Failed:    1
------------
Total states run:     2
 # salt --versions-report
           Salt: 2015.2.0rc2-185-gcdbb20b
         Python: 2.7.9 (default, Jan  7 2015, 11:50:42)
         Jinja2: 2.7.3
       M2Crypto: 0.22
 msgpack-python: 0.4.6
   msgpack-pure: 0.1.3
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: 1.2.1
          PyZMQ: 14.6.0
           RAET: Not Installed
            ZMQ: 4.0.5
           Mako: 1.0.1

And in the debug logs i still can see: 

[CRITICAL] Unable to import msgpack or msgpack_pure python modules

But i have these packages installed?

# pip search msgpack
msgpack-python                  - MessagePack (de)serializer.
  INSTALLED: 0.4.6 (latest)
msgpack-rpc-python              - MessagePack RPC
  INSTALLED: 0.4 (latest)
u-msgpack-python                - A portable, lightweight MessagePack serializer and deserializer written in pure Python.
greenrpc                        - TCP & HTTP RPC Servers written with msgpack and gevent
msgpack-numpy                   - Numpy data serialization using msgpack
wzmsgpackrpc                    - MessagePack-RPC implementation using Whizzer
msgpack-pure                    - MessagePack (de)serializer written in pure Python.
  INSTALLED: 0.1.3 (latest)
tornado_msgpack                 - ore ore MessagePack RPC
django-msgpack-serializer       - A MsgPack serializer for Django.
tastypie-msgpack                - MsgPack support for Django Tastypie.
gpsd_format                     - A library and command line tool to read, write and validate AIS and GPS messages in the GPSD JSON format (or the same
                                  format in a msgpack container).
txmsgpackrpc                    - txmsgpackrpc is a Twisted library to support msgpack-rpc
djangorestframework-msgpack     - MessagePack support for Django REST framework
msgpack-pypy                    - A msgpack extension with special optimizations for PyPy
pyramid_msgpack                 - Pyramid view renderer to output in msgpack (as opposed to JSON).
msgpack-tool                    - a cli utility for msgpack/json conversion
  INSTALLED: 0.0.1 (latest)
h0jeZvgoxFepBQ2C commented 9 years ago

It seems that here is somewhere the problem.. I try now to execute the command on the server directly:

myserveruser@myserver:~$ sudo /usr/bin/python2.7 /tmp/.myserveruser_85176d__salt/salt-call --local --metadata --out json -l quiet -c /tmp/.myserveruser_85176d__salt -- test.opts_pkg
[CRITICAL] Unable to import msgpack or msgpack_pure python modules
{
    "local": {
        "fun_args": [],
        "jid": "20150428172641327840",
        "return": {
            "output_file_append": false,
            "ioflo_realtime": true,
            "acceptance_wait_time": 10,
            "recon_default": 1000,
            "whitelist_modules": [],
            "ioflo_console_logdir": "",
            "utils_dirs": [
                "/var/cache/salt/minion/extmods/utils"
            ],
            "states_dirs": [],
            "fileserver_backend": [
                "roots"
            ],
            "outputter_dirs": [],
            "sls_list": [],
            "module_dirs": [],
            "tcp_keepalive": true,
            "extension_modules": "/tmp/.myserveruser_85176d__salt/running_data/var/cache/salt/minion/extmods",
            "state_auto_order": true,
            "master_alive_interval": 0,
            "__role": "minion",
            "disable_modules": [],
            "backup_mode": "",
            "recon_randomize": true,
            "return": "",
            "file_ignore_glob": null,
            "auto_accept": true,
            "cache_jobs": false,
            "state_verbose": true,
            "verify_master_pubkey_sign": false,
            "password": null,
            "startup_states": "",
            "grains": {
                "biosversion": "Bochs",
                "kernel": "Linux",
                "domain": "",
                "zmqversion": "4.0.5",
                "kernelrelease": "3.8.0-29-generic",
                "pythonpath": [
                    "/tmp/.myserveruser_85176d__salt",
                    "/usr/lib/python2.7",
                    "/usr/lib/python2.7/plat-linux2",
                    "/usr/lib/python2.7/lib-tk",
                    "/usr/lib/python2.7/lib-old",
                    "/usr/lib/python2.7/lib-dynload",
                    "/usr/local/lib/python2.7/dist-packages",
                    "/usr/lib/python2.7/dist-packages",
                    "/usr/lib/pymodules/python2.7"
                ],
                "serialnumber": "Not Specified",
                "ip_interfaces": {
                    "lo": [
                        "127.0.0.1",
                        "::1"
                    ],
                    "eth0": [
                        "146.185.136.84",
                        "fe80::601:2cff:fee4:f901"
                    ]
                },
                "shell": "/bin/bash",
                "mem_total": 2002,
                "osfinger": "Ubuntu-12.04",
                "SSDs": [],
                "mdadm": [],
                "id": "myserver",
                "osrelease": "12.04",
                "ps": "ps -efHww",
                "server_id": 1874993583,
                "lsb_distrib_description": "Ubuntu 12.04.5 LTS",
                "ip6_interfaces": {
                    "lo": [
                        "::1"
                    ],
                    "eth0": [
                        "fe80::601:2cff:fee4:f901"
                    ]
                },
                "num_cpus": 2,
                "hwaddr_interfaces": {
                    "lo": "00:00:00:00:00:00",
                    "eth0": "04:01:2c:e4:f9:01"
                },
                "ip4_interfaces": {
                    "lo": [
                        "127.0.0.1"
                    ],
                    "eth0": [
                        "146.185.136.84"
                    ]
                },
                "osfullname": "Ubuntu",
                "master": "salt",
                "ipv4": [
                    "127.0.0.1",
                    "146.185.136.84"
                ],
                "ipv6": [
                    "::1",
                    "fe80::601:2cff:fee4:f901"
                ],
                "cpu_flags": [
                    "fpu",
                    "de",
                    "pse",
                    "tsc",
                    "msr",
                    "pae",
                    "mce",
                    "cx8",
                    "apic",
                    "sep",
                    "mtrr",
                    "pge",
                    "mca",
                    "cmov",
                    "pse36",
                    "clflush",
                    "mmx",
                    "fxsr",
                    "sse",
                    "sse2",
                    "syscall",
                    "nx",
                    "lm",
                    "rep_good",
                    "nopl",
                    "pni",
                    "vmx",
                    "cx16",
                    "popcnt",
                    "hypervisor",
                    "lahf_lm"
                ],
                "localhost": "myserver",
                "lsb_distrib_id": "Ubuntu",
                "fqdn_ip4": [
                    "127.0.1.1"
                ],
                "fqdn_ip6": [],
                "nodename": "myserver",
                "saltversion": "2015.2.0rc2-185-gcdbb20b",
                "lsb_distrib_release": "12.04",
                "saltpath": "/tmp/.myserveruser_85176d__salt/salt",
                "pythonversion": [
                    2,
                    7,
                    3,
                    "final",
                    0
                ],
                "saltversioninfo": [
                    2015,
                    2,
                    0,
                    0
                ],
                "host": "myserver",
                "os_family": "Debian",
                "oscodename": "precise",
                "path": "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "biosreleasedate": "01/01/2011",
                "manufacturer": "Bochs",
                "num_gpus": 1,
                "virtual": "kvm",
                "cpu_model": "QEMU Virtual CPU version 1.4.0",
                "fqdn": "myserver",
                "pythonexecutable": "/usr/bin/python2.7",
                "productname": "Bochs",
                "osarch": "amd64",
                "cpuarch": "x86_64",
                "lsb_distrib_codename": "precise",
                "osrelease_info": [
                    12,
                    4
                ],
                "locale_info": {
                    "detectedencoding": "UTF-8",
                    "defaultlanguage": "en_US",
                    "defaultencoding": "UTF-8"
                },
                "gpus": [
                    {
                        "model": "Device 1111",
                        "vendor": "unknown"
                    }
                ],
                "machine_id": "7420189b8d22d0c035bbc7d40000019b",
                "os": "Ubuntu"
            },
            "always_verify_signature": false,
            "file_recv_max_size": 100,
            "pillar": {
                "master": {
                    "output_file_append": false,
                    "ioflo_realtime": true,
                    "master_alive_interval": 0,
                    "recon_default": 1000,
                    "whitelist_modules": [],
                    "ioflo_console_logdir": "",
                    "utils_dirs": [
                        "/var/cache/salt/minion/extmods/utils"
                    ],
                    "states_dirs": [],
                    "fileserver_backend": [
                        "roots"
                    ],
                    "outputter_dirs": [],
                    "sls_list": [],
                    "module_dirs": [],
                    "tcp_keepalive": true,
                    "extension_modules": "/tmp/.myserveruser_85176d__salt/running_data/var/cache/salt/minion/extmods",
                    "state_auto_order": true,
                    "acceptance_wait_time": 10,
                    "__role": "minion",
                    "disable_modules": [],
                    "backup_mode": "",
                    "recon_randomize": true,
                    "return": "",
                    "file_ignore_glob": null,
                    "auto_accept": true,
                    "cache_jobs": false,
                    "state_verbose": true,
                    "verify_master_pubkey_sign": false,
                    "password": null,
                    "startup_states": "",
                    "auth_timeout": 60,
                    "always_verify_signature": false,
                    "tcp_pull_port": 4511,
                    "gitfs_pubkey": "",
                    "fileserver_ignoresymlinks": false,
                    "retry_dns": 30,
                    "file_ignore_regex": null,
                    "output": "json",
                    "master_shuffle": false,
                    "metadata": true,
                    "multiprocessing": true,
                    "file_roots": {
                        "base": [
                            "/srv/salt"
                        ]
                    },
                    "caller": true,
                    "root_dir": "/tmp/.myserveruser_85176d__salt/running_data",
                    "log_granular_levels": {},
                    "returner_dirs": [],
                    "gitfs_privkey": "",
                    "skip_grains": false,
                    "arg": [],
                    "log_datefmt_logfile": "%Y-%m-%d %H:%M:%S",
                    "config_dir": "/tmp/.myserveruser_85176d__salt",
                    "random_reauth_delay": 10,
                    "autosign_timeout": 120,
                    "gitfs_base": "master",
                    "render_dirs": [],
                    "gitfs_user": "",
                    "fileserver_limit_traversal": false,
                    "tcp_keepalive_intvl": -1,
                    "conf_file": "/tmp/.myserveruser_85176d__salt/minion",
                    "top_file": "",
                    "zmq_monitor": false,
                    "file_recv_max_size": 100,
                    "pidfile": "/tmp/.myserveruser_85176d__salt/running_data/var/run/salt-minion.pid",
                    "range_server": "range:80",
                    "raet_mutable": false,
                    "grains_dirs": [],
                    "pillar_roots": {
                        "base": [
                            "/srv/pillar"
                        ]
                    },
                    "schedule": {},
                    "raet_main": false,
                    "fun": "test.opts_pkg",
                    "cachedir": "/tmp/.myserveruser_85176d__salt/running_data/var/cache/salt/minion",
                    "interface": "0.0.0.0",
                    "update_restart_services": [],
                    "recon_max": 10000,
                    "default_include": "minion.d/*.conf",
                    "hard_crash": false,
                    "rejected_retry": false,
                    "state_events": false,
                    "environment": null,
                    "win_repo_cachefile": "salt://win/repo/winrepo.p",
                    "ipc_mode": "ipc",
                    "keysize": 2048,
                    "master_sign_key_name": "master_sign",
                    "cython_enable": false,
                    "raet_port": 4510,
                    "ext_job_cache": "",
                    "hash_type": "md5",
                    "state_output": "full",
                    "force_color": false,
                    "modules_max_memory": -1,
                    "renderer": "yaml_jinja",
                    "state_top": "salt://top.sls",
                    "gitfs_env_whitelist": [],
                    "auth_tries": 7,
                    "gitfs_insecure_auth": false,
                    "mine_interval": 60,
                    "grains_cache": false,
                    "file_recv": false,
                    "no_color": false,
                    "ipv6": false,
                    "verify_env": true,
                    "username": null,
                    "master_finger": "",
                    "saltversion": "2015.2.0rc2-185-gcdbb20b",
                    "syndic_finger": "",
                    "tcp_keepalive_idle": 300,
                    "gitfs_passphrase": "",
                    "fileserver_followsymlinks": true,
                    "master_port": "4506",
                    "ioflo_period": 0.1,
                    "ping_interval": 0,
                    "local": true,
                    "tcp_keepalive_cnt": -1,
                    "raet_alt_port": 4511,
                    "retcode_passthrough": false,
                    "doc": false,
                    "state_aggregate": false,
                    "update_url": false,
                    "grains_refresh_every": 0,
                    "transport": "zeromq",
                    "providers": {},
                    "autoload_dynamic_modules": true,
                    "file_buffer_size": 262144,
                    "log_fmt_console": "[%(levelname)-8s] %(message)s",
                    "random_master": false,
                    "log_datefmt": "%H:%M:%S",
                    "grains_cache_expiration": 300,
                    "minion_floscript": "/tmp/.myserveruser_85176d__salt/salt/daemons/flo/minion.flo",
                    "id": "myserver",
                    "caller_floscript": "/tmp/.myserveruser_85176d__salt/salt/daemons/flo/caller.flo",
                    "loop_interval": 1,
                    "log_level": "quiet",
                    "gitfs_env_blacklist": [],
                    "auth_safemode": false,
                    "clean_dynamic_modules": true,
                    "disable_returners": [],
                    "cache_sreqs": true,
                    "minion_id_caching": true,
                    "gitfs_root": "",
                    "test": false,
                    "gitfs_password": "",
                    "failhard": false,
                    "raet_clear_remotes": true,
                    "file_client": "local",
                    "user": "root",
                    "use_master_when_local": false,
                    "acceptance_wait_time_max": 0,
                    "open_mode": false,
                    "permissive_pki_access": false,
                    "cmd_safe": true,
                    "zmq_filtering": false,
                    "refresh_grains_cache": false,
                    "selected_output_option": "output_indent",
                    "master_type": "str",
                    "pki_dir": "/tmp/.myserveruser_85176d__salt/running_data/etc/salt/pki/minion",
                    "grains_run": false,
                    "max_event_size": 1048576,
                    "ioflo_verbose": 0,
                    "sock_dir": "/tmp/.myserveruser_85176d__salt/running_data",
                    "tcp_pub_port": 4510,
                    "master": "salt",
                    "log_fmt_logfile": "%(asctime)s,%(msecs)03.0f [%(name)-17s][%(levelname)-8s][%(process)d] %(message)s",
                    "log_file": "/tmp/.myserveruser_85176d__salt/running_data/var/log/salt/minion",
                    "gitfs_remotes": [],
                    "gitfs_mountpoint": ""
                }
            },
            "fileserver_ignoresymlinks": false,
            "retry_dns": 30,
            "pillar_roots": {
                "base": [
                    "/srv/pillar"
                ]
            },
            "output": "json",
            "master_shuffle": false,
            "range_server": "range:80",
            "multiprocessing": true,
            "file_roots": {
                "base": [
                    "/srv/salt"
                ]
            },
            "caller": true,
            "root_dir": "/tmp/.myserveruser_85176d__salt/running_data",
            "log_granular_levels": {},
            "returner_dirs": [],
            "gitfs_privkey": "",
            "skip_grains": false,
            "arg": [],
            "grains_dirs": [],
            "config_dir": "/tmp/.myserveruser_85176d__salt",
            "random_reauth_delay": 10,
            "autosign_timeout": 120,
            "gitfs_base": "master",
            "render_dirs": [],
            "gitfs_user": "",
            "fileserver_limit_traversal": false,
            "tcp_keepalive_intvl": -1,
            "conf_file": "/tmp/.myserveruser_85176d__salt/minion",
            "top_file": "",
            "zmq_monitor": false,
            "tcp_pull_port": 4511,
            "pidfile": "/tmp/.myserveruser_85176d__salt/running_data/var/run/salt-minion.pid",
            "metadata": true,
            "raet_mutable": false,
            "log_datefmt_logfile": "%Y-%m-%d %H:%M:%S",
            "file_ignore_regex": null,
            "schedule": {},
            "raet_main": false,
            "fun": "test.opts_pkg",
            "cachedir": "/tmp/.myserveruser_85176d__salt/running_data/var/cache/salt/minion",
            "interface": "0.0.0.0",
            "update_restart_services": [],
            "recon_max": 10000,
            "default_include": "minion.d/*.conf",
            "hard_crash": false,
            "rejected_retry": false,
            "state_events": false,
            "environment": null,
            "win_repo_cachefile": "salt://win/repo/winrepo.p",
            "ipc_mode": "ipc",
            "keysize": 2048,
            "master_sign_key_name": "master_sign",
            "cython_enable": false,
            "raet_port": 4510,
            "ext_job_cache": "",
            "hash_type": "md5",
            "state_output": "full",
            "force_color": false,
            "modules_max_memory": -1,
            "renderer": "yaml_jinja",
            "state_top": "top.sls",
            "gitfs_env_whitelist": [],
            "auth_tries": 7,
            "gitfs_insecure_auth": false,
            "mine_interval": 60,
            "grains_cache": false,
            "file_recv": false,
            "no_color": false,
            "ipv6": false,
            "verify_env": true,
            "username": null,
            "master_finger": "",
            "syndic_finger": "",
            "tcp_keepalive_idle": 300,
            "gitfs_passphrase": "",
            "fileserver_followsymlinks": true,
            "master_port": "4506",
            "ioflo_period": 0.1,
            "ping_interval": 0,
            "auth_timeout": 60,
            "local": true,
            "tcp_keepalive_cnt": -1,
            "raet_alt_port": 4511,
            "retcode_passthrough": false,
            "doc": false,
            "state_aggregate": false,
            "update_url": false,
            "grains_refresh_every": 0,
            "transport": "zeromq",
            "providers": {},
            "autoload_dynamic_modules": true,
            "file_buffer_size": 262144,
            "log_fmt_console": "[%(levelname)-8s] %(message)s",
            "random_master": false,
            "log_datefmt": "%H:%M:%S",
            "grains_cache_expiration": 300,
            "minion_floscript": "/tmp/.myserveruser_85176d__salt/salt/daemons/flo/minion.flo",
            "id": "myserver",
            "caller_floscript": "/tmp/.myserveruser_85176d__salt/salt/daemons/flo/caller.flo",
            "loop_interval": 1,
            "log_level": "quiet",
            "gitfs_env_blacklist": [],
            "auth_safemode": false,
            "clean_dynamic_modules": true,
            "disable_returners": [],
            "cache_sreqs": true,
            "minion_id_caching": true,
            "gitfs_root": "",
            "test": false,
            "gitfs_password": "",
            "failhard": false,
            "raet_clear_remotes": true,
            "file_client": "local",
            "user": "root",
            "use_master_when_local": false,
            "acceptance_wait_time_max": 0,
            "open_mode": false,
            "permissive_pki_access": false,
            "cmd_safe": true,
            "gitfs_pubkey": "",
            "zmq_filtering": false,
            "refresh_grains_cache": false,
            "selected_output_option": "output_indent",
            "master_type": "str",
            "pki_dir": "/tmp/.myserveruser_85176d__salt/running_data/etc/salt/pki/minion",
            "grains_run": false,
            "max_event_size": 1048576,
            "ioflo_verbose": 0,
            "sock_dir": "/tmp/.myserveruser_85176d__salt/running_data",
            "tcp_pub_port": 4510,
            "master": "salt",
            "log_fmt_logfile": "%(asctime)s,%(msecs)03.0f [%(name)-17s][%(levelname)-8s][%(process)d] %(message)s",
            "log_file": "/tmp/.myserveruser_85176d__salt/running_data/var/log/salt/minion",
            "gitfs_remotes": [],
            "gitfs_mountpoint": ""
        },
        "retcode": 0,
        "fun": "test.opts_pkg",
        "id": "myserver"
    }
}
h0jeZvgoxFepBQ2C commented 9 years ago

And it seems that here is already something wrong with salt itself on the client:

myserveruser@myserver:~$ sudo /usr/bin/python2.7 /tmp/.myserveruser_85176d__salt/salt-call --versions-report
[CRITICAL] Unable to import msgpack or msgpack_pure python modules
           Salt: 2015.2.0rc2-185-gcdbb20b
         Python: 2.7.3 (default, Dec 18 2014, 19:10:20)
         Jinja2: 2.7.3
       M2Crypto: Not Installed
 msgpack-python: Not Installed
   msgpack-pure: Not Installed
       pycrypto: 2.4.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: Not Installed
          PyZMQ: 14.5.0
           RAET: Not Installed
            ZMQ: 4.0.5
           Mako: Not Installed
jfindlay commented 9 years ago

What is the OS version of the client?

h0jeZvgoxFepBQ2C commented 9 years ago 
# uname -a
Linux myserver 3.8.0-29-generic #42~precise1-Ubuntu SMP Wed Aug 14 16:19:23 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Ubuntu 12.04.5 LTS (GNU/Linux 3.8.0-29-generic x86_64) on a DigitalOcean droplet
jfindlay commented 9 years ago

@lichtamberg, thanks for working on this.

h0jeZvgoxFepBQ2C commented 9 years ago

No problem, thanks for the great framework! :)

EDIT: I'm offline for today. Please let me know if I can provide some other details, i will continue my investigation tomorrow... Thanks for your responses!

h0jeZvgoxFepBQ2C commented 9 years ago

Strange.. I tried it now via salt-master mode in a vagrant machine (also ubuntu 12.4.5)... The file download of archive.extract works only with http -> with https i get the following error:

# Vagrant Ubuntu precise64, works with http, but not with https!

# salt '*' state.highstate test=False
minion1:
  Name: curl - Function: pkg.installed - Result: Clean
  Name: /tmp/couchdb - Function: file.directory - Result: Clean
----------
          ID: test_source
    Function: archive.extracted
        Name: /tmp/couchdb
      Result: False
     Comment: Unable to manage file: hostname 'www.apache.org' doesn't match either of '*.openoffice.org', 'openoffice.org'
     Started: 09:53:35.171724
    Duration: 1133.245 ms
     Changes:

Summary
------------
Succeeded: 2
Failed:    1
------------
Total states run:     3

But when I do the same, masterless via salt-ssh (same salt version), i get following error message:

# salt-ssh 'minion1' state.highstate -i  test=False --passwd=vagrant
minion1:
  Name: curl - Function: pkg.installed - Result: Clean
  Name: /tmp/couchdb - Function: file.directory - Result: Clean
----------
          ID: test_source
    Function: archive.extracted
        Name: /tmp/couchdb
      Result: False
     Comment: Unable to manage file: 'Module_six_moves_urllib' object has no attribute 'Request'
     Started: 09:52:23.009688
    Duration: 449.688 ms
     Changes:

Summary
------------
Succeeded: 2
Failed:    1
------------
Total states run:     3

Both runs with following (latest git version of branch 2015.02):

           Salt: 2015.2.0rc2-203-g0407892
         Python: 2.7.9 (default, Jan  7 2015, 11:50:42)
         Jinja2: 2.7.3
       M2Crypto: 0.22
 msgpack-python: 0.4.6
   msgpack-pure: 0.1.3
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.11
          ioflo: 1.2.1
          PyZMQ: 14.6.0
           RAET: Not Installed
            ZMQ: 4.0.5
           Mako: 1.0.1

I created a gist with my vagrant & salt configs: https://gist.github.com/lichtamberg/fa38328d1d550d53432e

basepi commented 9 years ago

Appears not to be a salt-ssh problem. I'm reproducing the original Unable to manage file: 'handle' with 2015.5.2. Still investigating.

Edit: and here's the traceback that's happening on the minion:

[DEBUG   ] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/salt/states/file.py", line 1459, in managed
    **kwargs
  File "/usr/local/lib/python2.7/dist-packages/salt/modules/file.py", line 2819, in get_managed
    hash_fn = __salt__['cp.cache_file'](source_hash, saltenv)
  File "/usr/local/lib/python2.7/dist-packages/salt/modules/cp.py", line 344, in cache_file
    result = __context__['cp.fileclient'].cache_file(path, saltenv)
  File "/usr/local/lib/python2.7/dist-packages/salt/fileclient.py", line 152, in cache_file
    return self.get_url(path, '', True, saltenv)
  File "/usr/local/lib/python2.7/dist-packages/salt/fileclient.py", line 598, in get_url
    response = query['handle']
KeyError: 'handle'
basepi commented 9 years ago

Another update -- if I change both the source, and the source_hash to http instead of https, it works, on both salt and salt-ssh.

h0jeZvgoxFepBQ2C commented 9 years ago

Yeah, I experienced this too.. I think it has something to do with the ssl certificates..

basepi commented 9 years ago

Yep. Still investigating, the https link should work (works in browser for me)

basepi commented 9 years ago

Alright, I think I've traced this back to a bug in the implementation of our urllib2 certificate checking in salt.utils.http in 2015.5. Interestingly enough, requests also returns the same error. But as far as I can tell the site is configured correctly, as I can't reproduce the error with curl.

Does this mean that we just mark it as "upstream bug" and call it a day? Not quite.

I tried this with the develop version, and it works perfectly! The develop version of our salt.utils.http actually uses tornado as the default backend for the queries. And that appears to be working perfectly. So my recommendation is to just use http for the moment, and when the next feature release of salt comes out in a couple of months this should work perfectly.

basepi commented 9 years ago

Based on the above, I am going to go ahead and close this issue. If you can reproduce this bug using the develop version of salt I'd be happy to revisit it.