search

saltstack-formulas / opensds-formula

Integration with Open Software Defined Storage - Capri release - SuSE/CentOS/Ubuntu
https://opensds.io
Other
4 stars 2 forks source link

[CentOS] curl#6 - "Could not resolve host: mirrorlist.centos.org; #72

Closed noelmcloughlin closed 5 years ago

noelmcloughlin commented 5 years ago

Make blockbox fails. My internet connection is okay; name resolution works?

          ID: opensds backend block daemon cinder software build make blockbox
    Function: cmd.run
        Name: make blockbox
      Result: False
     Comment: Command "make blockbox" run
     Started: 09:37:46.534821
    Duration: 129760.935 ms
     Changes:   
              ----------
              pid:
                  25817
              retcode:
                  2
              stderr:
                  The command '/bin/sh -c /opt/loci/scripts/install.sh' returned a non-zero code: 1
                  make: *** [base] Error 1
              stdout:
                  docker build https://git.openstack.org/openstack/loci.git \
                    --build-arg PROJECT=cinder \
                    --build-arg FROM=centos:7 \
                    --build-arg PROJECT_REF=master \
                    --tag debian-cinder:latest
                  Sending build context to Docker daemon  250.9kB

                  Step 1/26 : ARG FROM=ubuntu:xenial
                  Step 2/26 : FROM ${FROM}
                  7: Pulling from library/centos
                  a02a4930cb5d: Pulling fs layer
                  a02a4930cb5d: Download complete
                  a02a4930cb5d: Pull complete
                  Digest: sha256:184e5f35598e333bfa7de10d8fb1cebb5ee4df5bc0f970bf2b1e7c7345136426
                  Status: Downloaded newer image for centos:7
                   ---> 1e1148e4cc2c
                  Step 3/26 : ENV PATH=/var/lib/openstack/bin:$PATH
                   ---> Running in f251a479b1f4
                  Removing intermediate container f251a479b1f4
                   ---> ea5dd140fc6e
                  Step 4/26 : ARG PROJECT
                   ---> Running in c9b5f5bcb6d2
                  Removing intermediate container c9b5f5bcb6d2
                   ---> e645f35066e8
                  Step 5/26 : ARG WHEELS=loci/requirements:master-ubuntu
                   ---> Running in c5591a391c16
                  Removing intermediate container c5591a391c16
                   ---> 7eefb2db8cd6
                  Step 6/26 : ARG PROJECT_REPO=https://git.openstack.org/openstack/${PROJECT}
                   ---> Running in 1c4ffb8b68a6
                  Removing intermediate container 1c4ffb8b68a6
                   ---> 79303fa7bc48
                  Step 7/26 : ARG PROJECT_REF=master
                   ---> Running in d72d7e855495
                  Removing intermediate container d72d7e855495
                   ---> b5246ca44fba
                  Step 8/26 : ARG DISTRO
                   ---> Running in 2b0152dff469
                  Removing intermediate container 2b0152dff469
                   ---> 61a9404f23c8
                  Step 9/26 : ARG PROFILES
                   ---> Running in 68bf940f2aa0
                  Removing intermediate container 68bf940f2aa0
                   ---> dbb1f3e7b0cc
                  Step 10/26 : ARG PIP_PACKAGES=""
                   ---> Running in 7cafb9d76843
                  Removing intermediate container 7cafb9d76843
                   ---> 7b9e11f53f2f
                  Step 11/26 : ARG PIP_ARGS=""
                   ---> Running in e46625a239bc
                  Removing intermediate container e46625a239bc
                   ---> dce293f86882
                  Step 12/26 : ARG PIP_WHEEL_ARGS=$PIP_ARGS
                   ---> Running in beb16116ba7d
                  Removing intermediate container beb16116ba7d
                   ---> 1855d7a232a9
                  Step 13/26 : ARG DIST_PACKAGES=""
                   ---> Running in eb8f572573cc
                  Removing intermediate container eb8f572573cc
                   ---> de684274d720
                  Step 14/26 : ARG PLUGIN=no
                   ---> Running in 0da837edccb1
                  Removing intermediate container 0da837edccb1
                   ---> c56634d7e0f4
                  Step 15/26 : ARG PYTHON3=no
                   ---> Running in 9f27713d4576
                  Removing intermediate container 9f27713d4576
                   ---> 1e7873b9544c
                  Step 16/26 : ARG EXTRA_BINDEP=""
                   ---> Running in 1643051e45a6
                  Removing intermediate container 1643051e45a6
                   ---> a92b0c4694d2
                  Step 17/26 : ARG EXTRA_PYDEP=""
                   ---> Running in a106c729f3d5
                  Removing intermediate container a106c729f3d5
                   ---> 0c2726ce6efc
                  Step 18/26 : ARG UID=42424
                   ---> Running in b5d69f7c78fc
                  Removing intermediate container b5d69f7c78fc
                   ---> 5c004ec16d9c
                  Step 19/26 : ARG GID=42424
                   ---> Running in a3e3b14fe58d
                  Removing intermediate container a3e3b14fe58d
                   ---> 10bfe58e93c7
                  Step 20/26 : ARG NOVNC_REPO=https://github.com/novnc/novnc
                   ---> Running in 13386469d2d4
                  Removing intermediate container 13386469d2d4
                   ---> c50c1b95d134
                  Step 21/26 : ARG NOVNC_REF=v1.0.0
                   ---> Running in a5b09a2647c4
                  Removing intermediate container a5b09a2647c4
                   ---> 5005e3b8632b
                  Step 22/26 : ARG SPICE_REPO=https://gitlab.freedesktop.org/spice/spice-html5.git
                   ---> Running in ffd93b26b8fb
                  Removing intermediate container ffd93b26b8fb
                   ---> 099ac0eff250
                  Step 23/26 : ARG SPICE_REF=spice-html5-0.1.6
                   ---> Running in d360f460eff6
                  Removing intermediate container d360f460eff6
                   ---> 915d2777c532
                  Step 24/26 : COPY scripts /opt/loci/scripts
                   ---> 72be9c3646e4
                  Step 25/26 : ADD bindep.txt pydep.txt $EXTRA_BINDEP $EXTRA_PYDEP /opt/loci/
                   ---> 56eb751d3f64
                  Step 26/26 : RUN /opt/loci/scripts/install.sh
                   ---> Running in 5cc0bb7ac0e8
                  ?[91m++ awk -F= '/^ID=/ {gsub(/\"/, "", $2); print $2}' /etc/centos-release /etc/os-release /etc/redhat-release /etc/system-release
                  ?[0m?[91m+ distro=centos
                  + export distro=centos
                  + distro=centos
                  + [[ no == \n\o ]]
                  + dpkg_python_packages=("python" "virtualenv")
                  + rpm_python_packages=("python" "python-virtualenv")
                  + case ${distro} in
                  + yum upgrade -y
                  ?[0mLoaded plugins: fastestmirror, ovl
                  Determining fastest mirrors
                  Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
                  14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"
                  ?[91m

                   One of the configured repositories failed (Unknown),
                   and yum doesn't have enough cached data to continue. At this point the only
                   safe thing yum can do is fail. There are a few ways to work "fix" this:

                       1. Contact the upstream for the repository and get them to fix the problem.

                       2. Reconfigure the baseurl/etc. for the repository, to point to a working
                          upstream. This is most often useful if you are using a newer
                          distribution release than is supported by the repository (and the
                          packages for the previous distribution release still work).

                       3. Run the command with the repository temporarily disabled
                              yum --disablerepo=<repoid> ...

                       4. Disable the repository permanently, so yum won't use it by default. Yum
                          will then just ignore the repository until you permanently enable it
                          again or use --enablerepo for temporary usage:

                              yum-config-manager --disable <repoid>
                          or
                              subscription-manager repos --disable=<repoid>

                       5. Configure the failing repository to be skipped, if it is unavailable.
                          Note that yum will try to contact the repo. when it runs most commands,
                          so will have to try and fail each time (and thus. yum will be be much
                          slower). If it is a very temporary problem though, this is often a nice
                          compromise:

                              yum-config-manager --save --setopt=<repoid>.skip_if_unavailable=true

                  Cannot find a valid baseurl for repo: base/7/x86_64
                  ?[0m
mostafahussein commented 5 years ago

Can you try to check the dns settings on the minion ? resolv.conf in specfic

noelmcloughlin commented 5 years ago

I think your hunch is good - feels like name resolution issue.

The vagrant Centos:7 image has this-

[vagrant@localhost ~]$ cat /etc/resolv.conf 
# Generated by NetworkManager
search home
nameserver 10.0.2.3
nameserver 192.168.1.1

Then resolver-formula updates to this-

[vagrant@localhost ~]$ cat /etc/resolv.conf 
# THIS FILE IS MANAGED BY SALTSTACK
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
domain example.com
nameserver 8.8.8.8
nameserver 64.6.64.6
options attempts:5 rotate timeout:1

[vagrant@localhost ~]$ ping mirrorlist.centos.org
PING mirrorlist.centos.org (85.236.43.108) 56(84) bytes of data.
64 bytes from 108-43-236-85.rev.customer-net.de (85.236.43.108): icmp_seq=1 ttl=63 time=67.6 ms

Maybe I need to add 192.168.1.1 (home router) as name server too?

[vagrant@localhost ~]$ nslookup mirrorlist.centos.org
Server:     8.8.8.8
Address:    8.8.8.8#53

Non-authoritative answer:
Name:   mirrorlist.centos.org
Address: 212.69.166.138
Name:   mirrorlist.centos.org
Address: 216.176.179.218
Name:   mirrorlist.centos.org
Address: 67.219.148.138
Name:   mirrorlist.centos.org
Address: 85.236.43.108

[vagrant@localhost ~]$ dig mirrorlist.centos.org

; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> mirrorlist.centos.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42174
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;mirrorlist.centos.org.     IN  A

;; ANSWER SECTION:
mirrorlist.centos.org.  55  IN  A   216.176.179.218
mirrorlist.centos.org.  55  IN  A   67.219.148.138
mirrorlist.centos.org.  55  IN  A   85.236.43.108
mirrorlist.centos.org.  55  IN  A   212.69.166.138

;; Query time: 78 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun Feb 17 11:34:28 MST 2019
;; MSG SIZE  rcvd: 114
noelmcloughlin commented 5 years ago

Hmm works if I do this manually.

[vagrant@localhost ~]$ curl 'http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container'

http://ftp.heanet.ie/pub/centos/7.6.1810/os/x86_64/
http://mirror.strencom.net/centos/7.6.1810/os/x86_64/
http://mirror.bytemark.co.uk/centos/7.6.1810/os/x86_64/
http://mirrors.melbourne.co.uk/sites/ftp.centos.org/centos/7.6.1810/os/x86_64/
http://mirror.sax.uk.as61049.net/centos/7.6.1810/os/x86_64/
http://mirror.mhd.uk.as44574.net/mirror.centos.org/7.6.1810/os/x86_64/
http://mozart.ee.ic.ac.uk/CentOS/7.6.1810/os/x86_64/
http://mirror.econdc.com/centos/7.6.1810/os/x86_64/
http://www.mirrorservice.org/sites/mirror.centos.org/7.6.1810/os/x86_64/
http://mirrors.ukfast.co.uk/sites/ftp.centos.org/7.6.1810/os/x86_64/
mostafahussein commented 5 years ago

What about from within the container ? check the following issue from docker's repo. I would suggest you to create a docker container then ensure that the issue is exist and try to fix it or just try the suggested solution in that thread

noelmcloughlin commented 5 years ago

Hi @mostafahussein thanks for suggestions. I connected to a few docker containers and name resolution worked. Timeout is not set in yum repos - default to 30seconds.

Anyway I found a solution: I added following stanza to blockbox docker-compose.yml. 

services:
    dns:
        - 8.8.8.8
        - 192.168.1.1

Now the docker compose is working.

+ yum install -y --setopt=skip_missing_names_on_install=False git patch redhat-lsb-core sudo python python-virtualenv
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: ftp.heanet.ie
 * extras: ftp.heanet.ie
 * updates: mozart.ee.ic.ac.uk

I'm wondering if blockbox compose file should have this by default?

mostafahussein commented 5 years ago

If this solves the issue for you then you might go with the solution where you modify DOCKER_OPTS and restart the daemon. By default docker-compose does not contain the dns settings and you don't need it unless there is a specific dns server you want to resolve from it or you have similar issues.

noelmcloughlin commented 5 years ago

I raised blockbox issue. I'll try your suggestion - maybe I can pass DOCKER_OPTS into the make blockbox environment.

mostafahussein commented 5 years ago

In this case I will close this issue, if you have any other questions you can update the thread!

noelmcloughlin commented 5 years ago

Updated investigation.

Solution is daemon.json file

[root@localhost vagrant]# cat /etc/docker/daemon.json 
{
  "dns": ["8.8.8.8", "8.8.4.4"],
  "dns-search": ["example.com"]
}
[root@localhost vagrant]# systemctl restart docker

I add commit to PR #73 to fix the formula.

Testing shows positive results (command takes long time) but error has gone away.

root     17907  0.0  0.6 420416 45144 pts/0    Sl+  17:04   0:00 docker build https://git.openstack.org/openstack/loci.git --build-arg PROJECT=cinder --build-arg FROM=centos:7 --build-arg PROJECT_REF=master --build-arg PROFILES=lvm --tag lvm-debian-cinder:latest
root     18035  0.0  0.0  10720  5668 ?        Sl   17:04   0:00 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/d52fec3b77d38231f500828950804c9303eb62cec6e36a8e79837c4d6eaf5062 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc
root     18052  0.0  0.0  11684  1512 ?        Ss   17:04   0:00 /bin/bash /opt/loci/scripts/install.sh
root     18854  0.0  0.0  11684  1396 ?        S    17:09   0:00 /bin/bash /opt/loci/scripts/clone_project.sh
root     18855  0.0  0.1  19380  7920 ?        S    17:09   0:00 git clone https://git.openstack.org/openstack/cinder /tmp/cinder
root     18856  2.3  0.2 182316 17668 ?        S    17:09   0:01 git-remote-https origin https://git.openstack.org/openstack/cinder
root     18922  0.5  0.2  34152 14228 ?        Sl   17:09   0:00 git fetch-pack --stateless-rpc --stdin --lock-pack --thin --no-progress https://git.openstack.org/openstack/cinder/
root     18928  1.7  0.1  30184  7780 ?        S    17:09   0:00 git index-pack --stdin --fix-thin --keep=fetch-pack 443 on d52fec3b77d3 --pack_header=2,180703
noelmcloughlin commented 5 years ago

Summary - Problem is resolved by /etc/docker/daemon.json.

                  Complete!
                  ?[91m+ yum clean all
                  ?[0mLoaded plugins: fastestmirror, ovl
                  Cleaning repos: base extras updates
                  Cleaning up list of fastest mirrors
                  ?[91m+ rm /var/lib/openstack/lib/python2.7/no-global-site-packages.txt
                  ?[0m?[91m+ rm -rf /tmp/cinder /tmp/ks-script-h2MyUP /tmp/wheels /tmp/wheels.tar.gz /tmp/yum.log /root/.cache /etc/machine-id
                  ?[0m?[91m+ find /usr/ /var/ '(' -name '*.pyc' -o -name __pycache__ ')' -delete
                  ?[0mRemoving intermediate container d52fec3b77d3
                   ---> b989ce5b1c62
                  Successfully built b989ce5b1c62
                  Successfully tagged lvm-debian-cinder:latest
noelmcloughlin commented 5 years ago

Closed by #73