search

ansible / ansible

Ansible is a radically simple IT automation platform that makes your applications and systems easier to deploy and maintain. Automate everything from code deployment to network configuration to cloud management, in a language that approaches plain English, using SSH, with no agents to install on remote systems. https://docs.ansible.com.
https://www.ansible.com/
GNU General Public License v3.0
62.55k stars 23.85k forks source link

fetch results in checksum mismatch #69383

Open queglay opened 4 years ago

queglay commented 4 years ago
SUMMARY

Fetch results in checksum mismatch. Become must be used to acquire it from the location due to permissions.

ISSUE TYPE
COMPONENT NAME

fetch

ANSIBLE VERSION
2.9.7
CONFIGURATION
ACTION_WARNINGS(default) = True
AGNOSTIC_BECOME_PROMPT(default) = True
ALLOW_WORLD_READABLE_TMPFILES(default) = False
ANSIBLE_CONNECTION_PATH(default) = None
ANSIBLE_COW_PATH(default) = None
ANSIBLE_COW_SELECTION(default) = default
ANSIBLE_COW_WHITELIST(default) = ['bud-frogs', 'bunny', 'cheese', 'daemon', 'default', 'dragon', 'elephant-in-snake', 'elephant', 'eyes', 'hellokitty', 'kitty', 'luke-koala', 'meow', 'milk', 'moofasa', 'moose', 
ANSIBLE_FORCE_COLOR(default) = False
ANSIBLE_NOCOLOR(default) = False
ANSIBLE_NOCOWS(default) = False
ANSIBLE_PIPELINING(/deployuser/ansible.cfg) = False
ANSIBLE_SSH_ARGS(default) = -C -o ControlMaster=auto -o ControlPersist=60s
ANSIBLE_SSH_CONTROL_PATH(default) = None
ANSIBLE_SSH_CONTROL_PATH_DIR(default) = ~/.ansible/cp
ANSIBLE_SSH_EXECUTABLE(default) = ssh
ANSIBLE_SSH_RETRIES(/deployuser/ansible.cfg) = 10
ANY_ERRORS_FATAL(default) = False
BECOME_ALLOW_SAME_USER(default) = False
BECOME_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/become', u'/usr/share/ansible/plugins/become']
CACHE_PLUGIN(default) = memory
CACHE_PLUGIN_CONNECTION(default) = None
CACHE_PLUGIN_PREFIX(default) = ansible_facts
CACHE_PLUGIN_TIMEOUT(default) = 86400
COLLECTIONS_PATHS(default) = [u'/home/deployuser/.ansible/collections', u'/usr/share/ansible/collections']
COLOR_CHANGED(default) = yellow
COLOR_CONSOLE_PROMPT(default) = white
COLOR_DEBUG(default) = dark gray
COLOR_DEPRECATE(default) = purple
COLOR_DIFF_ADD(default) = green
COLOR_DIFF_LINES(default) = cyan
COLOR_DIFF_REMOVE(default) = red
COLOR_ERROR(default) = red
COLOR_HIGHLIGHT(default) = white
COLOR_OK(default) = green
COLOR_SKIP(default) = cyan
COLOR_UNREACHABLE(default) = bright red
COLOR_VERBOSE(default) = blue
COLOR_WARN(default) = bright purple
COMMAND_WARNINGS(default) = True
CONDITIONAL_BARE_VARS(default) = True
CONNECTION_FACTS_MODULES(default) = {'junos': 'junos_facts', 'eos': 'eos_facts', 'frr': 'frr_facts', 'iosxr': 'iosxr_facts', 'nxos': 'nxos_facts', 'ios': 'ios_facts', 'vyos': 'vyos_facts'}
COVERAGE_REMOTE_OUTPUT(default) = None
COVERAGE_REMOTE_WHITELIST(default) = *
DEFAULT_ACTION_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/action', u'/usr/share/ansible/plugins/action']
DEFAULT_ALLOW_UNSAFE_LOOKUPS(default) = False
DEFAULT_ASK_PASS(default) = False
DEFAULT_ASK_VAULT_PASS(default) = False
DEFAULT_BECOME(default) = False
DEFAULT_BECOME_ASK_PASS(default) = False
DEFAULT_BECOME_EXE(default) = None
DEFAULT_BECOME_FLAGS(default) = 
DEFAULT_BECOME_METHOD(default) = sudo
DEFAULT_BECOME_USER(default) = root
DEFAULT_CACHE_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/cache', u'/usr/share/ansible/plugins/cache']
DEFAULT_CALLABLE_WHITELIST(default) = []
DEFAULT_CALLBACK_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/callback', u'/usr/share/ansible/plugins/callback']
DEFAULT_CALLBACK_WHITELIST(/deployuser/ansible.cfg) = [u'profile_tasks']
DEFAULT_CLICONF_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/cliconf', u'/usr/share/ansible/plugins/cliconf']
DEFAULT_CONNECTION_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/connection', u'/usr/share/ansible/plugins/connection']
DEFAULT_DEBUG(default) = False
DEFAULT_EXECUTABLE(default) = /bin/sh
DEFAULT_FACT_PATH(default) = None
DEFAULT_FILTER_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/filter', u'/usr/share/ansible/plugins/filter']
DEFAULT_FORCE_HANDLERS(default) = False
DEFAULT_FORKS(default) = 5
DEFAULT_GATHERING(default) = implicit
DEFAULT_GATHER_SUBSET(/deployuser/ansible.cfg) = [u'!hardware # this line may help deal with an issue where a bad nfs mount will prevent ansible from connecting to the host.']
DEFAULT_GATHER_TIMEOUT(default) = 10
DEFAULT_HANDLER_INCLUDES_STATIC(default) = False
DEFAULT_HASH_BEHAVIOUR(default) = replace
DEFAULT_HOST_LIST(env: ANSIBLE_INVENTORY) = [u'/vagrant/ansible/hosts']
DEFAULT_HTTPAPI_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/httpapi', u'/usr/share/ansible/plugins/httpapi']
DEFAULT_INTERNAL_POLL_INTERVAL(default) = 0.001
DEFAULT_INVENTORY_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/inventory', u'/usr/share/ansible/plugins/inventory']
DEFAULT_JINJA2_EXTENSIONS(default) = []
DEFAULT_JINJA2_NATIVE(default) = False
DEFAULT_KEEP_REMOTE_FILES(default) = False
DEFAULT_LIBVIRT_LXC_NOSECLABEL(default) = False
DEFAULT_LOAD_CALLBACK_PLUGINS(/deployuser/ansible.cfg) = True
DEFAULT_LOCAL_TMP(default) = /home/deployuser/.ansible/tmp/ansible-local-285749tfiB1
DEFAULT_LOG_FILTER(default) = []
DEFAULT_LOG_PATH(/deployuser/ansible.cfg) = /deployuser/tmp/ansible_log
DEFAULT_LOOKUP_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/lookup', u'/usr/share/ansible/plugins/lookup']
DEFAULT_MANAGED_STR(default) = Ansible managed
DEFAULT_MODULE_ARGS(default) = 
DEFAULT_MODULE_COMPRESSION(default) = ZIP_DEFLATED
DEFAULT_MODULE_NAME(default) = command
DEFAULT_MODULE_PATH(default) = [u'/home/deployuser/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
DEFAULT_MODULE_UTILS_PATH(default) = [u'/home/deployuser/.ansible/plugins/module_utils', u'/usr/share/ansible/plugins/module_utils']
DEFAULT_NETCONF_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/netconf', u'/usr/share/ansible/plugins/netconf']
DEFAULT_NO_LOG(default) = False
DEFAULT_NO_TARGET_SYSLOG(default) = False
DEFAULT_NULL_REPRESENTATION(default) = None
DEFAULT_POLL_INTERVAL(default) = 15
DEFAULT_PRIVATE_KEY_FILE(default) = None
DEFAULT_PRIVATE_ROLE_VARS(default) = False
DEFAULT_REMOTE_PORT(default) = None
DEFAULT_REMOTE_USER(default) = None
DEFAULT_ROLES_PATH(default) = [u'/home/deployuser/.ansible/roles', u'/usr/share/ansible/roles', u'/etc/ansible/roles']
DEFAULT_SCP_IF_SSH(default) = smart
DEFAULT_SELINUX_SPECIAL_FS(default) = ['fuse', 'nfs', 'vboxsf', 'ramfs', '9p', 'vfat']
DEFAULT_SFTP_BATCH_MODE(default) = True
DEFAULT_SQUASH_ACTIONS(default) = ['apk', 'apt', 'dnf', 'homebrew', 'openbsd_pkg', 'pacman', 'pip', 'pkgng', 'yum', 'zypper']
DEFAULT_SSH_TRANSFER_METHOD(default) = None
DEFAULT_STDOUT_CALLBACK(env: ANSIBLE_STDOUT_CALLBACK) = debug
DEFAULT_STRATEGY(/deployuser/ansible.cfg) = mitogen_linear
DEFAULT_STRATEGY_PLUGIN_PATH(/deployuser/ansible.cfg) = [u'/deployuser/ansible/plugins/mitogen-0.2.9/ansible_mitogen/plugins/strategy']
DEFAULT_SU(default) = False
DEFAULT_SYSLOG_FACILITY(default) = LOG_USER
DEFAULT_TASK_INCLUDES_STATIC(default) = False
DEFAULT_TERMINAL_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/terminal', u'/usr/share/ansible/plugins/terminal']
DEFAULT_TEST_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/test', u'/usr/share/ansible/plugins/test']
DEFAULT_TIMEOUT(default) = 10
DEFAULT_TRANSPORT(default) = smart
DEFAULT_UNDEFINED_VAR_BEHAVIOR(default) = True
DEFAULT_VARS_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/vars', u'/usr/share/ansible/plugins/vars']
DEFAULT_VAULT_ENCRYPT_IDENTITY(default) = None
DEFAULT_VAULT_IDENTITY(default) = default
DEFAULT_VAULT_IDENTITY_LIST(default) = []
DEFAULT_VAULT_ID_MATCH(default) = False
DEFAULT_VAULT_PASSWORD_FILE(default) = None
DEFAULT_VERBOSITY(default) = 0
DEPRECATION_WARNINGS(default) = True
DIFF_ALWAYS(default) = False
DIFF_CONTEXT(default) = 3
DISPLAY_ARGS_TO_STDOUT(default) = False
DISPLAY_SKIPPED_HOSTS(default) = True
DOCSITE_ROOT_URL(default) = https://docs.ansible.com/ansible/
DOC_FRAGMENT_PLUGIN_PATH(default) = [u'/home/deployuser/.ansible/plugins/doc_fragments', u'/usr/share/ansible/plugins/doc_fragments']
DUPLICATE_YAML_DICT_KEY(default) = warn
ENABLE_TASK_DEBUGGER(default) = False
ERROR_ON_MISSING_HANDLER(default) = True
FACTS_MODULES(default) = ['smart']
GALAXY_IGNORE_CERTS(default) = False
GALAXY_ROLE_SKELETON(default) = None
GALAXY_ROLE_SKELETON_IGNORE(default) = ['^.git$', '^.*/.git_keep$']
GALAXY_SERVER(default) = https://galaxy.ansible.com
GALAXY_SERVER_LIST(default) = None
GALAXY_TOKEN(default) = None
GALAXY_TOKEN_PATH(default) = /home/deployuser/.ansible/galaxy_token
HOST_KEY_CHECKING(default) = True
HOST_PATTERN_MISMATCH(default) = warning
INJECT_FACTS_AS_VARS(default) = True
INTERPRETER_PYTHON(default) = auto_legacy
INTERPRETER_PYTHON_DISTRO_MAP(default) = {'centos': {'8': '/usr/libexec/platform-python', '6': '/usr/bin/python'}, 'rhel': {'8': '/usr/libexec/platform-python', '6': '/usr/bin/python'}, 'fedora': {'23': '/usr/bi
INTERPRETER_PYTHON_FALLBACK(default) = ['/usr/bin/python', 'python3.7', 'python3.6', 'python3.5', 'python2.7', 'python2.6', '/usr/libexec/platform-python', '/usr/bin/python3', 'python']
INVALID_TASK_ATTRIBUTE_FAILED(default) = True
INVENTORY_ANY_UNPARSED_IS_FAILED(default) = False
INVENTORY_CACHE_ENABLED(default) = False
INVENTORY_CACHE_PLUGIN(default) = None
INVENTORY_CACHE_PLUGIN_CONNECTION(default) = None
INVENTORY_CACHE_PLUGIN_PREFIX(default) = ansible_facts
INVENTORY_CACHE_TIMEOUT(default) = 3600
INVENTORY_ENABLED(default) = ['host_list', 'script', 'auto', 'yaml', 'ini', 'toml']
INVENTORY_EXPORT(default) = False
INVENTORY_IGNORE_EXTS(default) = {{(BLACKLIST_EXTS + ( '.orig', '.ini', '.cfg', '.retry'))}}
INVENTORY_IGNORE_PATTERNS(default) = []
INVENTORY_UNPARSED_IS_FAILED(default) = False
LOCALHOST_WARNING(default) = True
MAX_FILE_SIZE_FOR_DIFF(default) = 104448
NETCONF_SSH_CONFIG(default) = None
NETWORK_GROUP_MODULES(default) = ['eos', 'nxos', 'ios', 'iosxr', 'junos', 'enos', 'ce', 'vyos', 'sros', 'dellos9', 'dellos10', 'dellos6', 'asa', 'aruba', 'aireos', 'bigip', 'ironware', 'onyx', 'netconf']
OLD_PLUGIN_CACHE_CLEARING(default) = False
PARAMIKO_HOST_KEY_AUTO_ADD(default) = False
PARAMIKO_LOOK_FOR_KEYS(default) = True
PERSISTENT_COMMAND_TIMEOUT(default) = 30
PERSISTENT_CONNECT_RETRY_TIMEOUT(default) = 15
PERSISTENT_CONNECT_TIMEOUT(default) = 30
PERSISTENT_CONTROL_PATH_DIR(default) = /home/deployuser/.ansible/pc
PLAYBOOK_DIR(default) = None
PLAYBOOK_VARS_ROOT(default) = top
PLUGIN_FILTERS_CFG(default) = None
PYTHON_MODULE_RLIMIT_NOFILE(default) = 0
RETRY_FILES_ENABLED(default) = False
RETRY_FILES_SAVE_PATH(default) = None
SHOW_CUSTOM_STATS(default) = False
STRING_CONVERSION_ACTION(default) = warn
STRING_TYPE_FILTERS(default) = ['string', 'to_json', 'to_nice_json', 'to_yaml', 'ppretty', 'json']
SYSTEM_WARNINGS(default) = True
TAGS_RUN(default) = []
TAGS_SKIP(default) = []
TASK_DEBUGGER_IGNORE_ERRORS(default) = True
TRANSFORM_INVALID_GROUP_CHARS(default) = never
USE_PERSISTENT_CONNECTIONS(default) = False
VARIABLE_PRECEDENCE(default) = ['all_inventory', 'groups_inventory', 'all_plugins_inventory', 'all_plugins_play', 'groups_plugins_inventory', 'groups_plugins_play']
VERBOSE_TO_STDERR(default) = False
YAML_FILENAME_EXTENSIONS(default) = ['.yml', '.yaml', '.json']
OS / ENVIRONMENT

ubuntu 16.04

STEPS TO REPRODUCE
- name: Get files in a folder
  find:
    paths: "/opt/Thinkbox/DeadlineDatabase10/mongo/data/logs/"
  register: found_files
  no_log: true

- name: Get latest file
  set_fact:
    latest_file: "{{ found_files.files | sort(attribute='mtime',reverse=true) | first }}"

- name: "Get file {{ latest_file.path }} copy to {{ firehawk_path }}/tmp/deadline_db.log"
  fetch:
    src: "{{ latest_file.path }}"
    dest: "{{ firehawk_path }}/tmp/deadline_db.log"
    flat: yes
  become: true
EXPECTED RESULTS

It should fetch the file without any error.

ACTUAL RESULTS
 fatal: [firehawkgateway]: FAILED! => {
     "changed": false, 
     "checksum": "3ed68ded03e48466d7ccab9abdb174fd8d6d8dac", 
     "dest": "/deployuser/tmp/deadline_db.log", 
     "file": "/opt/Thinkbox/DeadlineDatabase10/mongo/data/logs/log.txt", 
     "md5sum": "cfcc74c8732a2c5318d891f141afa3f1", 
     "remote_checksum": "603c321f1e631188401d082be08a83f67515f84c", 
     "remote_md5sum": null
 }

 MSG:

 checksum mismatch
ansibot commented 4 years ago

Files identified in the description:

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

queglay commented 4 years ago

I've used fetch a lot in my AWS infrastructure, and I just saw another occurance with this related to another log file. I suspect that since those logs are being updated at the time of the fetch, the checksum changes between the check time and the copy operation.

If the theory is correct, fetch could perhaps copy the file to another location first, then perform the copy and checksum from this copied file, which shouldn't change any further.

bcoca commented 4 years ago

So if this is a race condition for files that are being updated during the fetch operation, I do not think we should copy it by default (file might still require more data to be written). But an option to force the fetch anyways might be a solution to your issue. (--ignore-checksum-changes? .. bikeshed away).

ansibot commented 4 years ago

Files identified in the description:

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

queglay commented 4 years ago

I figured a file copy first and checksum on the copy would be better than no checksum at all.