search

hitachienergy / epiphany

Cloud and on-premises automation for Kubernetes centered industrial grade solutions.
Apache License 2.0
138 stars 107 forks source link

[BUG] Failed `epicli upgrade` may lead to broken repository machine (disabled system repos) #2718

Closed przemyslavic closed 2 years ago

przemyslavic commented 2 years ago

Describe the bug Similar to #2324 but for upgrade mode. Failed epicli upgrade may lead to broken repository machine.

How to reproduce Steps to reproduce the behavior:

  1. execute epicli upgrade command
  2. let's assume that upgrade of any component fails
  3. the current status on the repository machine is that epirepo is enabled and system repositories are disabled
  4. re-run upgrade after about 2 hours (download_done_flag_expire_minutes is 120 by default) or manually remove download-requirements-done.flag file and re-run upgrade
  5. it fails on task TASK [repository : Run download-requirements script, this can take a long time because all packages are removed from epirepo and no system repositories are available.
Downloading packages:
http://10.1.11.52/epirepo/packages/wget-1.14-18.el7_6.1.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article 

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.

Error downloading packages:
  wget-1.14-18.el7_6.1.x86_64: [Errno 256] No more mirrors to try.

Full output:

2021-10-20T11:23:51.6752845Z11:23:51 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Check if flag file exists] **********************************
2021-10-20T11:23:53.3749257Z11:23:53 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:53.9511762Z11:23:53 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:23:53.9512849Z11:23:53 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Remove download-requirements-done.flag file if expired] *****
2021-10-20T11:23:55.3052185Z11:23:55 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:55.6872357Z11:23:55 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:23:55.6877932Z11:23:55 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Check whether to run download script] ***********************
2021-10-20T11:23:56.7045834Z11:23:56 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:56.7992855Z11:23:56 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:23:56.7998737Z11:23:56 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : include_tasks] **********************************************
2021-10-20T11:23:57.4211641Z11:23:57 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:57.6496915Z11:23:57 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:57.8248110Z11:23:57 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:58.0728224Z11:23:58 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:58.2406036Z11:23:58 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:58.2502871Z11:23:58 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:23:58.6031522Z11:23:58 INFO cli.engine.ansible.AnsibleCommand - included: /shared/build/10todevawrhcalico/ansible/roles/repository/tasks/clean-up-epirepo.yml for ec2-x-x-x-x.eu-west-3.compute.amazonaws.com
2021-10-20T11:23:59.1468367Z11:23:59 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:23:59.1475151Z11:23:59 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Load vars from old manifest] ********************************
2021-10-20T11:23:59.5609101Z11:23:59 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:00.1428641Z11:24:00 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:00.1429711Z11:24:00 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Set fact on epirepo directory path] *************************
2021-10-20T11:24:00.5539548Z11:24:00 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:00.9188694Z11:24:00 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:00.9189806Z11:24:00 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Get list of files in epirepo] *******************************
2021-10-20T11:24:03.5179008Z11:24:03 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=/var/www/html/epirepo/files)
2021-10-20T11:24:04.4136533Z11:24:04 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=/var/www/html/epirepo/images)
2021-10-20T11:24:04.9000904Z11:24:04 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:04.9001933Z11:24:04 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Remove old files from epirepo] ******************************
2021-10-20T11:24:05.9252449Z11:24:05 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=erlang-23.1.5-1.el7.x86_64.rpm.1)
2021-10-20T11:24:06.6530781Z11:24:06 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=haproxy_exporter-0.10.0.linux-amd64.tar.gz.1)
2021-10-20T11:24:07.4233549Z11:24:07 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=jmx_prometheus_javaagent-0.14.0.jar.1)
2021-10-20T11:24:08.1382537Z11:24:08 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=kafka_exporter-1.2.0.linux-amd64.tar.gz.1)
2021-10-20T11:24:08.9162020Z11:24:08 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=node_exporter-1.0.1.linux-amd64.tar.gz.1)
2021-10-20T11:24:09.6737502Z11:24:09 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=kafka_2.12-2.6.0.tgz.1)
2021-10-20T11:24:10.5943377Z11:24:10 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=prometheus-2.10.0.linux-amd64.tar.gz.1)
2021-10-20T11:24:11.3669825Z11:24:11 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=alertmanager-0.17.0.linux-amd64.tar.gz.1)
2021-10-20T11:24:12.1199090Z11:24:12 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=apache-zookeeper-3.5.8-bin.tar.gz.1)
2021-10-20T11:24:12.8965086Z11:24:12 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=apache-ignite-2.9.1-bin.zip.1)
2021-10-20T11:24:13.6180906Z11:24:13 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=helm-v3.2.0-linux-amd64.tar.gz.1)
2021-10-20T11:24:14.4009220Z11:24:14 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=filebeat-7.9.2.tgz.1)
2021-10-20T11:24:15.3439970Z11:24:15 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=node-exporter-1.1.2.tgz.1)
2021-10-20T11:24:15.6362399Z11:24:15 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:15.6374756Z11:24:15 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Load variables from "image_registry" role] ******************
2021-10-20T11:24:15.7118387Z11:24:15 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:15.9927643Z11:24:15 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:15.9940625Z11:24:15 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Remove old images from epirepo] *****************************
2021-10-20T11:24:16.9429394Z11:24:16 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=keycloak-14.0.0.tar)
2021-10-20T11:24:17.6519485Z11:24:17 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com] => (item=ignite-2.9.1.tar)
2021-10-20T11:24:17.8522107Z11:24:17 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:17.8527693Z11:24:17 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Remove all packages from epirepo] ***************************
2021-10-20T11:24:18.7296798Z11:24:18 INFO cli.engine.ansible.AnsibleCommand - changed: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:18.9887044Z11:24:18 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:18.9894150Z11:24:18 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Find leftovers in /tmp] *************************************
2021-10-20T11:24:19.7561743Z11:24:19 INFO cli.engine.ansible.AnsibleCommand - ok: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:20.1329898Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:20.1339387Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Remove leftovers from /tmp] *********************************
2021-10-20T11:24:20.4187329Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:20.4283503Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Copy requirements for offline installation to repository host, this can take a long time
2021-10-20T11:24:20.4287061Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - Destination: /var/www/html/epirepo] ***
2021-10-20T11:24:20.5071704Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:20.6525039Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:20.8132181Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:20.8487784Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:20.9206402Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:20.9252742Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-3x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:20.9813525Z11:24:20 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:21.0218593Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:21.0219757Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Download Epiphany requirements] *****************************
2021-10-20T11:24:21.1548431Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:21.2618454Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:21.3154956Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:21.4315862Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:21.4330118Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:21.4571771Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - skipping: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]
2021-10-20T11:24:21.6093528Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - included: /shared/build/10todevawrhcalico/ansible/roles/repository/tasks/download-requirements.yml for ec2-x-x-x-x.eu-west-3.compute.amazonaws.com
2021-10-20T11:24:21.8429730Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - 
2021-10-20T11:24:21.8430772Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Run download-requirements script, this can take a long time
2021-10-20T11:24:21.8432245Z11:24:21 INFO cli.engine.ansible.AnsibleCommand - You can check progress on repository host with: journalctl -f -t download-requirements.sh] ***
2021-10-20T11:24:24.9334287Z11:24:24 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ec2-x-x-x-x.eu-west-3.compute.amazonaws.com]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /tmp/epi-download-requirements/download-requirements.sh /var/www/html/epirepo --no-logfile |& tee >(systemd-cat --identifier=download-requirements.sh)", "delta": "0:00:01.987291", "end": "2021-10-20 11:24:24.838161", "msg": "non-zero return code", "rc": 1, "start": "2021-10-20 11:24:22.850870", "stderr": "", "stderr_lines": [], "stdout": "Detected arch: x86_64
Docker platform: linux/amd64
PID is: 1782, creating file: /var/run/download-requirements.sh.pid
Testing yum connection:
- epirepo...
Connection successful.
Backuping /etc/yum.repos.d/ to /tmp/epi-download-requirements/download-requirements.sh-yum-repos-backup-tmp-do-not-remove.tar
tar: Removing leading `/' from member names
/etc/yum.repos.d/
/etc/yum.repos.d/redhat-rhui-beta.repo.disabled
/etc/yum.repos.d/redhat-rhui-client-config.repo
/etc/yum.repos.d/redhat-rhui.repo
/etc/yum.repos.d/epirepo.repo
tar: Archive contains file names with leading prefixes removed.
tar: Verification may fail to locate original files.
Verify etc/yum.repos.d/
Verify etc/yum.repos.d/redhat-rhui-beta.repo.disabled
Verify etc/yum.repos.d/redhat-rhui-client-config.repo
Verify etc/yum.repos.d/redhat-rhui.repo
Verify etc/yum.repos.d/epirepo.repo
Backup done
Package yum-utils already installed
Installing package: wget
Loaded plugins: amazon-id, search-disabled-repos, versionlock
Resolving Dependencies
--> Running transaction check
---> Package wget.x86_64 0:1.14-18.el7_6.1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package       Arch            Version                   Repository        Size
================================================================================
Installing:
 wget          x86_64          1.14-18.el7_6.1           epirepo          547 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 547 k
Installed size: 2.0 M
Downloading packages:
http://10.1.11.52/epirepo/packages/wget-1.14-18.el7_6.1.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror.
To address this issue please refer to the below knowledge base article 

https://access.redhat.com/articles/1320623

If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.

Error downloading packages:
  wget-1.14-18.el7_6.1.x86_64: [Errno 256] No more mirrors to try.

Command failed: yum install -y \"wget\"", "stdout_lines": ["Detected arch: x86_64", "Docker platform: linux/amd64", "PID is: 1782, creating file: /var/run/download-requirements.sh.pid", "Testing yum connection:", "- epirepo...", "Connection successful.", "Backuping /etc/yum.repos.d/ to /tmp/epi-download-requirements/download-requirements.sh-yum-repos-backup-tmp-do-not-remove.tar", "tar: Removing leading `/' from member names", "/etc/yum.repos.d/", "/etc/yum.repos.d/redhat-rhui-beta.repo.disabled", "/etc/yum.repos.d/redhat-rhui-client-config.repo", "/etc/yum.repos.d/redhat-rhui.repo", "/etc/yum.repos.d/epirepo.repo", "tar: Archive contains file names with leading prefixes removed.", "tar: Verification may fail to locate original files.", "Verify etc/yum.repos.d/", "Verify etc/yum.repos.d/redhat-rhui-beta.repo.disabled", "Verify etc/yum.repos.d/redhat-rhui-client-config.repo", "Verify etc/yum.repos.d/redhat-rhui.repo", "Verify etc/yum.repos.d/epirepo.repo", "Backup done", "Package yum-utils already installed", "Installing package: wget", "Loaded plugins: amazon-id, search-disabled-repos, versionlock", "Resolving Dependencies", "--> Running transaction check", "---> Package wget.x86_64 0:1.14-18.el7_6.1 will be installed", "--> Finished Dependency Resolution", "", "Dependencies Resolved", "", "================================================================================", " Package       Arch            Version                   Repository        Size", "================================================================================", "Installing:", " wget          x86_64          1.14-18.el7_6.1           epirepo          547 k", "", "Transaction Summary", "================================================================================", "Install  1 Package", "", "Total download size: 547 k", "Installed size: 2.0 M", "Downloading packages:", "http://10.1.11.52/epirepo/packages/wget-1.14-18.el7_6.1.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found", "Trying other mirror.", "To address this issue please refer to the below knowledge base article ", "", "https://access.redhat.com/articles/1320623", "", "If above article doesn't help to resolve this issue please open a ticket with Red Hat Support.", "", "", "", "Error downloading packages:", "  wget-1.14-18.el7_6.1.x86_64: [Errno 256] No more mirrors to try.", "", "Command failed: yum install -y \"wget\""]}

To solve this I had to login to the repository vm and execute there: sudo /tmp/epi-repository-setup-scripts/disable-epirepo-client.sh && sudo /tmp/epi-repository-setup-scripts/enable-system-repos.sh.

Expected behavior Re-run should succeed at least when it comes to configuring the repository vm.

Config files If applicable, add config files to help explain your problem.

Environment

epicli version: [1.3.0dev]

Additional context Add any other context about the problem here.

DoD checklist

to-bar commented 2 years ago

Worth to check whether Ubuntu is affected as well, since there is different logic there, see comment

to-bar commented 2 years ago

We may also change paths from /tmp/epi-repository-setup-scripts to /var/tmp/epi-repository-setup-scripts and from /tmp/epi-download-requirements to /var/tmp/epi-download-requirements in order to preserve files after OS restart. They are removed in teardown playbook (task Clean up temporary files).

przemyslavic commented 2 years ago

A similar problem also occurs on Ubuntu:

2021-11-25T12:45:18.9960903Z 12:45:18 INFO cli.engine.ansible.AnsibleCommand - TASK [repository : Run download-requirements script, this can take a long time
2021-11-25T12:45:18.9962164Z 12:45:18 INFO cli.engine.ansible.AnsibleCommand - You can check progress on repository host with: journalctl -f -t download-requirements.sh] ***
2021-11-25T12:45:25.8500190Z 12:45:25 ERROR cli.engine.ansible.AnsibleCommand - fatal: [ec2-1-1-1-1.eu-west-1.compute.amazonaws.com]: FAILED! => {"changed": true, "cmd": "set -o pipefail && /tmp/epi-download-requirements/download-requirements.sh /var/www/html/epirepo --no-logfile |& tee >(systemd-cat --identifier=download-requirements.sh)", "delta": "0:00:06.139469", "end": "2021-11-25 12:45:25.784458", "msg": "non-zero return code", "rc": 1, "start": "2021-11-25 12:45:19.644989", "stderr": "", "stderr_lines": [], "stdout": "Detected arch: x86_64\nDocker platform: linux/amd64\nOS repositories seems missing, restoring...\n/etc/apt/sources.list\n/etc/apt/sources.list.d/\nTesting apt connection:\n- /etc/apt/sources.list...\nConnection failure, reason: [ WARNING: apt does not have a stable CLI interface. Use with caution in scripts. E: Failed to fetch http://10.1.11.207/epirepo/packages/./Packages 404 Not Found [IP: 10.1.11.207 80] E: Some index files failed to download. They have been ignored, or old ones used instead.]", "stdout_lines": ["Detected arch: x86_64", "Docker platform: linux/amd64", "OS repositories seems missing, restoring...", "/etc/apt/sources.list", "/etc/apt/sources.list.d/", "Testing apt connection:", "- /etc/apt/sources.list...", "Connection failure, reason: [ WARNING: apt does not have a stable CLI interface. Use with caution in scripts. E: Failed to fetch http://10.1.11.207/epirepo/packages/./Packages 404 Not Found [IP: 10.1.11.207 80] E: Some index files failed to download. They have been ignored, or old ones used instead.]"]}
przemyslavic commented 2 years ago

:heavy_check_mark: The apply and upgrade commands have been tested many times after the previous build was successfully completed and also on failure. No issues were noticed with the repository on re-run. Tested on RHEL, Ubuntu, CentOS.