Saltstack repository - status page/public mirror possibility? And causing stucked install states?

Reiner030 commented 6 years ago

Hello,

It would be great to have

a public status page to easily recognize down/routing/ddos/... issues
the possibilty for public mirrors like most OSS offers so it's easy
- to lower the impact of data transfers for you / make traffic more "localized"
- switch to (other) mirrors if such failure occurs

Actual (over 3-4 hours yet) the saltstack repository repo.saltstack.com is mainly not available in Germany - different locations tested for the case it's geoip distributed... so

fresh setup is not possible

but also other states which tries to setup packages get stucked for hours... (I killed 1st run av e.g.:

[INFO    ] Executing state pkg.installed for [python-redis]
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[DEBUG   ] Could not LazyLoad pkg.check_db: 'pkg.check_db' is not available.
[DEBUG   ] Could not LazyLoad pkg.normalize_name: 'pkg.normalize_name' is not available.
[INFO    ] Executing command ['dpkg', '--get-selections', '*'] in directory '/root'
[INFO    ] Executing command ['systemd-run', '--scope', 'apt-get', '-q', '-y', '-o', 'DPkg::Options::=--force-confold', '-o', 'DPkg::Options::=--force-confdef', 'install', 'python-redis'] in dire[...]

which is very weird...

I was checking the cause of this stucked states (running salt-call 2017.7.2 (Nitrogen)) which keeps apt locked (latest Debian Stretch Cloud image).

As state the installs stopped working; it seems after the package install routine itself but apt/history log has no finished entry written. And apt-get is so busy that even a kill -9 $(pidof apt-get) is not cancelling the call which is extremely unknown behavior for such tasks ... => Perhaps it comes because of running it as child of systemd-run ? When running the install manually which I tested for comparison / check if there is an apt problem it's all fine there.

Reiner030 commented 6 years ago

Now it seems up again.

BTW: There is a mirror offer but the rsync would include all distributions (OS) and versions included which may results in some hundreds GBs / some TBs of (mostly unwanted) files which is not very useful for "local mirror" setups - only for public ones (can you setup some info files about the sizes, too?): https://repo.saltstack.com/#mirror

For Debian/Ubuntu I know about apt-cacher-ng which would fit much better. For Redhad based repositories it seems best to use the combination reposync/createrepo (short-checked - not using this distro's)

evarghese commented 6 years ago

It seems we are running into an ssl error:

curl https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/dists/trusty/main/binary-i386/Packages
curl: (35) Unknown SSL protocol error in connection to repo.saltstack.com:443

This is only happening intermittently, but it is hanging apt as Reiner030 reported.

deepakhj commented 6 years ago

I'm having issues with server certificate verification. Some time today, I am unable to bootstrap new hosts.


  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   264  100   264    0     0   3435      0 --:--:-- --:--:-- --:--:--  3473
100  244k  100  244k    0     0  1314k      0 --:--:-- --:--:-- --:--:-- 1314k
root@ip-xxx-xx-xx-xx:~# sudo sh install_salt.sh -P
sudo: unable to resolve host ip-xxx-xx-xx-xx
 *  INFO: Running version: 2017.12.13
 *  INFO: Executed by: shell pipe
 *  INFO: Command line: 'install_salt.sh -P'

 *  INFO: System Information:
 *  INFO:   CPU:          GenuineIntel
 *  INFO:   CPU Arch:     x86_64
 *  INFO:   OS Name:      Linux
 *  INFO:   OS Version:   3.13.0-119-generic
 *  INFO:   Distribution: Ubuntu 14.04

 *  INFO: Installing minion
 *  INFO: Found function install_ubuntu_stable_deps
 *  INFO: Found function config_salt
 *  INFO: Found function preseed_master
 *  INFO: Found function install_ubuntu_stable
 *  INFO: Found function install_ubuntu_stable_post
 *  INFO: Found function install_ubuntu_restart_daemons
 *  INFO: Found function daemons_running
 *  INFO: Found function install_ubuntu_check_services
 *  INFO: Running install_ubuntu_stable_deps()
Ign http://us-east-1.ec2.archive.ubuntu.com trusty InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty Release.gpg
Hit http://us-east-1.ec2.archive.ubuntu.com trusty Release
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/restricted Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/multiverse Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/multiverse amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/multiverse Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/restricted Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/main Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/universe Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/main Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/multiverse Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/restricted Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty-backports/universe Translation-en
Hit http://security.ubuntu.com trusty-security InRelease
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/restricted Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/multiverse Sources
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/multiverse amd64 Packages
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/main Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/multiverse Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/restricted Translation-en
Hit http://us-east-1.ec2.archive.ubuntu.com trusty/universe Translation-en
Hit http://security.ubuntu.com trusty-security/main Sources
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/main Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/multiverse Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/restricted Translation-en_US
Ign http://us-east-1.ec2.archive.ubuntu.com trusty/universe Translation-en_US
Hit http://security.ubuntu.com trusty-security/universe Sources
Hit http://security.ubuntu.com trusty-security/main amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign https://repo.saltstack.com trusty InRelease
Ign https://repo.saltstack.com trusty Release.gpg
Ign https://repo.saltstack.com trusty Release
Err https://repo.saltstack.com trusty/main amd64 Packages
  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Ign https://repo.saltstack.com trusty/main Translation-en_US
Ign https://repo.saltstack.com trusty/main Translation-en
W: Failed to fetch https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/dists/trusty/main/binary-amd64/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

E: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
apt-transport-https is already the newest version.
ca-certificates is already the newest version.
gnupg-curl is already the newest version.
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'apt-get autoremove' to remove it.
0 upgraded, 0 newly installed, 0 to remove and 104 not upgraded.
Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.EEw1Qz5MYk --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver-options ca-cert-file=/etc/ssl/certs/ca-certificates.crt --fetch-keys https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub
gpgkeys: https fetch error 60: server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: keyserver internal error
gpg: WARNING: unable to fetch URI https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/SALTSTACK-GPG-KEY.pub: keyserver error
 * ERROR: Failed to run install_ubuntu_stable_deps()!!!```

lsh-0 commented 6 years ago

similar error here:

# apt-get -q update
[...]
Ign https://repo.saltstack.com trusty/main amd64 Packages/DiffIndex
Ign https://repo.saltstack.com trusty/main Translation-en
Err https://repo.saltstack.com trusty/main amd64 Packages
  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
Fetched 1198 kB in 3min 36s (5544 B/s)
[ERROR   ] stderr: W: Failed to fetch https://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2016.3.6/dists/trusty/main/binary-amd64/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

E: Some index files failed to download. They have been ignored, or old ones used instead.
[ERROR   ] retcode: 100
[ERROR   ] An error was encountered while installing package(s): W: Failed to fetch https://repo.saltstack.com/apt/ubuntu/14.04/amd64/archive/2016.3.6/dists/trusty/main/binary-amd64/Packages  server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

E: Some index files failed to download. They have been ignored, or old ones used instead.

lsh-0 commented 6 years ago

I can replicate @evarghese case:

# curl -vvv https://repo.saltstack.com/apt/ubuntu/14.04/amd64/latest/dists/trusty/main/binary-i386/Packages
* Hostname was NOT found in DNS cache
*   Trying 138.197.226.47...
*   Trying 2604:a880:400:d0::2:e001...
* Immediate connect fail for 2604:a880:400:d0::2:e001: Network is unreachable
* Connected to repo.saltstack.com (138.197.226.47) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*    subject: C=US; ST=UT; L=Lehi; O=Salt Stack, Inc.; CN=*.saltstack.com
*    start date: 2017-05-08 00:00:00 GMT
*    expire date: 2019-05-13 12:00:00 GMT
*    subjectAltName: repo.saltstack.com matched
*    issuer: C=US; O=DigiCert Inc; CN=DigiCert SHA2 Secure Server CA
*    SSL certificate verify ok.
> GET /apt/ubuntu/14.04/amd64/latest/dists/trusty/main/binary-i386/Packages HTTP/1.1
> User-Agent: curl/7.35.0
> Host: repo.saltstack.com
> Accept: */*
> 
* SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
* Closing connection 0
curl: (56) SSL read: error:00000000:lib(0):func(0):reason(0), errno 104

update: curl case is now working, apt-get isn't.

update2: apt-get is now working too

+1 for a status page

garethgreenaway commented 6 years ago

@dubb-b FYI.

xavieryao commented 6 years ago

Would it be possible to maintain a "official mirror site" or "alternative download" list? We are hosting one at https://mirrors.tuna.tsinghua.edu.cn/saltstack/ and wish to be displayed one the official webpage.