Amazon Linux, pkg.installed, "Cannot retrieve metalink for repository: epel/x86_64"

[dezertol]

When trying to install the yum package memcached using

memcached: pkg:

• installed

It fails to install, the debug log on the client shows the following error

Install failed: Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again

Installing it manually via yum install memcached works correctly. This is a Linux Amazon AMI, with salt installed today using the following command

wget -O - http://bootstrap.saltstack.org | sudo sh

Is there a path or something that needs to be modified for this to work on an Amazon AMI?

Thanks (in advance)

[dezertol]

Also when the exact same thing is tested with a CentOS release 5.4 (Final) it works fine. I'm guessing there is some path or other setting that I need to adjust for the Amazon Linux AMI

[dezertol]

if I uncomment baseurl and comment out mirorlist in the epel.repo it seems to fix it.

I should point out that before adjusting the repo file, using yum install by it's self from the client works fine.

[basepi]

Hrm, I wonder if the Amazon Linux AMI is doing something weird with an alias or something. Have you tried running the exact yum command that salt runs? (You can see this command from the debug log, I think)

[dezertol]

Manually running it via yum on the box works fine with no errors. The issue is seems is that the salt uses the yum api, where as the yum command is simply the command.... at least that's my guess. I did not see any specific command that i could run from the debug log. But that fact that it works if you uncomment the baseurl and comment out the mirrorlist makes me think it's something to do with the way it resolves the mirror list. Unknown at this point and not sure how to test any of it other then say that what I've done so far is listed here in this case.

[basepi]

Ah, right, totally forgot that we don't shell out for yum, we use the API. You're right there, I was just being dumb.

@terminalmage any ideas? I know you're a super fan of the yum API. ;)

[shawnbutts]

I'm having the same issue. I've found that trigger highstate from the master gives this error. Triggering highstate from the minion with salt-call, does not give this error and installs the package.

[basepi]

Are any of you using SELinux?

[shawnbutts]

No SELinux here

[basepi]

Can you try forcing the pkg provider to yumpkg5 as shown here? http://docs.saltstack.com/ref/configuration/minion.html#providers

yumpkg5 is the predecessor to yumpkg (and is still used on RHEL < 6), and shells out to yum rather than using the API. Curious if that module will succeed.

[dezertol]

I can confirm that I'm not

[basepi]

@dezertol Please try forcing the pkg provider as outlined above. I want to know if it's a salt problem or just a yum API problem.

[dezertol]

I tried adding that perhaps I didn't put it int he right place.. cuz it blows up on the client with python errors

{% if grains['os'] == 'Amazon' %}

providers:
  pkg: yumpkg5
  service: systemd

{% endif %}

again this is "ONLY" an issue on amazon Linux ami's and it's not "all the time" I have had it work correctly once or twice but I can't reproduce it. At the moment I work around it by commenting out the mirror and uncommenting the base in the repo file.

[shawnbutts]

Adding the following to /etc/salt/minion config file and restarting the minion took care of this.

providers:
  pkg: yumpkg5

[terminalmage]

@shawnbutts @dezertol which AMI were you using, so I can test this? It seems to be restricted to the yum provider which uses the yum python API (yumpkg.py).

[shawnbutts]

ami-83e4bcea (us-east-1) & ami-be1c848e (us-west-2).

[terminalmage]

@shawnbutts thanks

[terminalmage]

@shawnbutts what version of the salt-minion is running on your ec2 instance?

[shawnbutts]

Currently it's 0.17.1. I did try 0.17.2 and got the same results.

[terminalmage]

OK, thank you very much, I've got the us-east-1 AMI you specified above up and running now and am looking into this.

[terminalmage]

Successfully reproduced. Posting the traceback here for reference.

root@erik-ubuntu-01:~# salt amazontest pkg.latest_version foo
amazontest:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/salt/minion.py", line 728, in _thread_return
        return_data = func(*args, **kwargs)
      File "/usr/lib/python2.6/site-packages/salt/modules/yumpkg.py", line 299, in latest_version
        pkglist = yumbase.doPackageLists(pkgtype)
      File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 2925, in doPackageLists
        avail = self.pkgSack.returnNewestByNameArch(patterns=patterns,
      File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 1063, in <lambda>
        pkgSack = property(fget=lambda self: self._getSacks(),
      File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 768, in _getSacks
        self.repos.populateSack(which=repos)
      File "/usr/lib/python2.6/site-packages/yum/repos.py", line 383, in populateSack
        sack.populate(repo, mdtype, callback, cacheonly)
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 250, in populate
        if self._check_db_version(repo, mydbtype):
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 342, in _check_db_version
        return repo._check_db_version(mdtype)
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1516, in _check_db_version
        repoXML = self.repoXML
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1702, in <lambda>
        repoXML = property(fget=lambda self: self._getRepoXML(),
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1698, in _getRepoXML
        self._loadRepoXML(text=self.ui_id)
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1689, in _loadRepoXML
        return self._groupLoadRepoXML(text, self._mdpolicy2mdtypes())
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1663, in _groupLoadRepoXML
        if self._commonLoadRepoXML(text):
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1488, in _commonLoadRepoXML
        result = self._getFileRepoXML(local, text)
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1266, in _getFileRepoXML
        size=102400) # setting max size as 100K
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 1049, in _getFile
        result = self.grab.urlgrab(misc.to_utf8(remote), local,
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 719, in <lambda>
        grab = property(lambda self: self._getgrab())
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 714, in _getgrab
        self._setupGrab()
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 653, in _setupGrab
        urls = self.urls
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 897, in <lambda>
        urls = property(fget=lambda self: self._geturls(),
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 894, in _geturls
        self._baseurlSetup()
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 840, in _baseurlSetup
        mirrorurls.extend(list(self.metalink_data.urls()))
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 937, in <lambda>
        metalink_data = property(fget=lambda self: self._getMetalink(),
      File "/usr/lib/python2.6/site-packages/yum/yumRepo.py", line 916, in _getMetalink
        raise Errors.RepoError, msg
    RepoError: Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again

Doing some more digging now...

[terminalmage]

I've tracked this down to an SSL error in a pycurl call that the yum API makes. When it attempts to download the metalink XML, there is a pycurl error when it attempts to verify the SSL certificate. The pycurl error code is 35, and the error is:

A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot..

[terminalmage]

I've found the issue. The bad news is that this is not a salt bug. It definitely appears to be the following:

http://lists.baseurl.org/pipermail/yum-devel/2013-August/010295.html

To confirm this, I wrote a script similar to the one that is referenced in the above message:

#!/usr/bin/env python

import os
import sys

import urlgrabber

def test():
    option = {
        'copy_local': 1,
        'ssl_verify_peer': False,
        'ssl_verify_host': False,
    }
    try:
        urlgrabber.urlgrab('https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=x86_64', '/tmp/metalink.xml.tmp', **option)
    except urlgrabber.grabber.URLGrabError:
        print("didn't work")
    else:
        print("worked")

def main():
    pid = os.fork()
    if pid:
        return
    test()

if __name__ == '__main__':
    if sys.argv[1] == '1':
        print('PASS #1 (without rpmUtils.arch import)')
    elif sys.argv[1] == '2':
        import rpmUtils.arch
        print('PASS #2 (with rpmUtils.arch import)')
    main()

If you save that and run: python test.py 1 it will test without importing rpmUtils.arch, and if you run python test.py 2 it will test with rpmUtils.arch imported. The rpmUtils.arch module imports rpm, which causes the error:

[root@amazontest ~]# python test.py 1
PASS #1 (without rpmUtils.arch import)
[root@amazontest ~]# worked

[root@amazontest ~]# python test.py 2
PASS #2 (with rpmUtils.arch import)
[root@amazontest ~]# didn't work

(Note that the prompt appears again because of the call to os.fork())

For now, the best solution is to continue to use the yumpkg5 provider for EC2 minions, but since this bug could eventually find its way into other Fedora/RHEL/CentOS versions, we might need to look into other ways of getting the functionality we get from rpmUtils.arch.

[terminalmage]

Actually, removing rpmUtils.arch doesn't work, something else in the yum API is likely also importing the rpm module.

[terminalmage]

I've modified the module detection so that Amazon now uses yumpkg5 while we figure out what we're going to do about this. This change should be available in 0.17.3 which we plan to release soon.

[shawnbutts]

Great! Thanks for all the work on this.

[terminalmage]

I'm going to go ahead and close this then, since I've tested this fix and confirmed it works. We may end up eventually getting rid of the yumpkg provider, since the yum/rpm python API has been more of a headache than it is worth.