Unable to setup AWS

[gnvalbuena]

I've been trying to set up bcbio_vm on a new computer running MacOSX (after having gotten it working on a different computer before), and have been unable to get past the iam step. I've followed the instructions using

export AWS_ACCESS_KEY_ID=***
export AWS_SECRET_ACCESS_KEY=***
bcbio_vm.py aws iam --region=eu-west-1

and have been getting the following error:

Traceback (most recent call last): File "/usr/local/bin/bcbio_vm.py", line 334, in args.func(args) File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbiovm/aws/iam.py", line 37, in bootstrap config = create_keypair(args.econfig, region=args.region) File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbiovm/aws/iam.py", line 88, in create_keypair key = ec2.get_key_pair(keyname) File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/ec2/connection.py", line 2838, in get_key_pair dry_run=dry_run File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/ec2/connection.py", line 2820, in get_all_key_pairs [('item', KeyPair)], verb='POST') File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/connection.py", line 1170, in get_list response = self.make_request(action, params, path, verb) File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/connection.py", line 1116, in make_request return self._mexe(http_request) File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe raise ex ssl.SSLError: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I've tried renewing access keys, setting up a new IAM user, etc., but continue to come up against the same error (even with region set to us-east-1). Have I just completely missed something basic here?

On a separate note, when I try to set region as eu-west-2, I get a different error.

Traceback (most recent call last): File "/usr/local/bin/bcbio_vm.py", line 334, in args.func(args) File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbiovm/aws/iam.py", line 37, in bootstrap config = create_keypair(args.econfig, region=args.region) File "/Users/gv510/install/bcbio-vm/anaconda/lib/python2.7/site-packages/bcbiovm/aws/iam.py", line 88, in create_keypair key = ec2.get_key_pair(keyname) AttributeError: 'NoneType' object has no attribute 'get_key_pair'

I was setting region to that initially, but I'm assuming that's to do with region settings that haven't been specified on bcbio-vm for eu-west-2?

Thanks!

Best, Gabriel

[gnvalbuena]

Figured the SSL certificate verification error out. The SSL issues seem to stem from older versions of Python 2.7, which the most recent miniconda install is still using (in this case, a fresh Miniconda install from the "latest" MacOSX script was installing Python 2.7.3)

resolved it by running

~/install/bcbio-vm/anaconda/bin/conda update python

which updated it to Python 2.7.13, which hasn't raised the SSL issues.

[chapmanb]

Gabriel; Thanks for the report and for following up the SSL solution. Apologies, I'm not sure why you got such an old python version with miniconda latest but glad an update fixes it for you.

Regarding the second error, I'm not exactly sure what is going on there. It looks like it's having trouble resolving the region you specified. Would you be able to share the exact command line you ran? Is it possible there was a typo of the region name in the command, which would give an error like this? Sorry to not know what is going on for sure but hope this helps some.

[gnvalbuena]

I've had a look at the output from the installation, and after entering

~/install/bcbio-vm/anaconda/bin/conda install --yes -c conda-forge -c bioconda bcbio-nextgen-vm

I get:

The following packages will be DOWNGRADED:

python:                             2.7.14-hde5916a_29                    --> 2.7.3-5                  
readline:                           7.0-hc1231fa_4                        --> 6.2-2 

So potentially something in the install instructions on bioconda?

As for the second error, all I'm entering is

bcbio_vm.py aws iam --region=eu-west-2

[chapmanb]

Thanks much for following up on both of these. I think we've resolved the python install issue, which appeared to be due to a change over in bioconda htslib versioning. This forced install of an older version of bcbio which led to the downgrades. I tried a clean install after resolving this and we no longer rever to the old python version.

For the second issue, the problem was that one of the toois pinned an older version of boto without support for the eu-west-2 region. I've updated the dependency tree for this so new installs should give you a boto version that supports it, and you can update an existing install with:

bcbiovm_conda install -y -c conda-forge -c bioconda toil boto

Hope this gets things running cleanly for you and please let us know if you run into any other issues.