search

deeptools / pyBigWig

A python extension for quick access to bigWig and bigBed files
MIT License
216 stars 48 forks source link

Accessing ENCODE files on Mac; issue with curl? #26

Closed lavenderca closed 7 years ago

lavenderca commented 7 years ago

In the README, an ENCODE bigBed file is opened. On Linux, I'm able to run the command as written:

>>> import pyBigWig
>>> bb = pyBigWig.open("https://www.encodeproject.org/files/ENCFF001JBR/@@download/ENCFF001JBR.bigBed")

On Mac, I'm receiving the following error:

>>> import pyBigWig
>>> bb = pyBigWig.open("https://www.encodeproject.org/files/ENCFF001JBR/@@download/ENCFF001JBR.bigBed")
[urlOpen] curl_easy_perform received an error: Peer certificate cannot be authenticated with given CA certificates
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Received an error during file opening!

I'm still able to access the file using my browser.

Trying to fix the issue, I updated my installation of curl (now 7.52.1) but am still getting the error. I ran curl with the file, and it seems like there is a redirect:

$ curl https://www.encodeproject.org/files/ENCFF001JBR/@@download/ENCFF001JBR.bigBed
<html>
 <head>
  <title>307 Temporary Redirect</title>
 </head>
 <body>
  <h1>307 Temporary Redirect</h1>
  The resource has been moved to https://download.encodeproject.org/http://encode-files.s3.amazonaws.com/2012/07/30/40ee5859-5afc-46b8-a4e5-9779669a5051/ENCFF001JBR.bedRnaElements.bigBed?Signature=HiyYAGlS1jZP2ffhK46Op%2B1hQXQ%3D&amp;Expires=1485926876&amp;AWSAccessKeyId=ASIAITRX7ADMF5IN73PA&amp;response-content-disposition=attachment%3B%20filename%3DENCFF001JBR.bigBed&amp;x-amz-security-token=FQoDYXdzEHkaDLO1lA56pGNBt180RyKcAymTGbyRfgD6/Dd/ACwRyDEoeSRkqrEwuOaOvo3SUg9re/zmv7XpMQowGhuHqtlbz1oLjj%2BBX4CZesx4DJzhs8nJep1APd9vvYmV5PekS2MKZM4/7GWlbOqAKYfJUtSQWq/NGjHtwaFJz1fhG2HKJQMyGuZxyW5goH%2Bi8MOLqYo8Md49BLJGhCJ89F12HRTUzsVDKyo1bbrx5YsT2/9hAmyP1qihgj2uFP9dBs/LUudwtx%2BWvdleLlJ1fbVLy3rWMzd1rjv%2B1%2B6/Vg146ukHraoKekz4c9lnztGoZPukA9Tjd3sXfz1Lb1/H8ZqHbZO/yjAR3RwILpdT%2BVG8DUNo9GuPxLhai1v9s6S1%2BYt3HnvRtL/rNYWtdGVIAdDW6QhjvgmaqO6qNAf0PkoE33EWl34TENBXxOfrZ8VOwEUDUxPLI5G6iebW8Kwknj1F/XR6hUJpoGmDGrDpAU54FG/pf4NmlMAqa2fE9/xcJYxAZPs36Arm2v%2BThSfYvLlHz/yvEWwq3G6YaxUk5P8vMjBRGTd41RgypvBA%2BSGk5scoyM%2B9xAU%3D; you should be redirected automatically.

 </body>
</html>

Could this be contributing to my issue?

dpryan79 commented 7 years ago

Interesting, this is likely unrelated to the redirect but instead related to a certificate error in OSX. Did you install pyBigWig with bioconda or pip?

lavenderca commented 7 years ago

bioconda. I used the following: conda install pybigwig -c bioconda

dpryan79 commented 7 years ago

What's the output of conda list | grep curl?

lavenderca commented 7 years ago 
$ conda list | grep curl
curl                      7.45.0                        0  
pycurl                    7.19.5.3                 py34_0
lavenderca commented 7 years ago

Currently trying an upgrade.

dpryan79 commented 7 years ago

What happens if you conda install -c bioconda curl (you might have to remove the one you currently have installed)? I made a version of curl for bioconda a year or so ago that fixed a few problems like this.

lavenderca commented 7 years ago

The upgrade seems to work! Thanks so much for your help.

Ran conda upgrade curl. Now:

$ conda list | grep curl
curl                      7.52.1                        0  
pycurl                    7.43.0                   py34_2
dpryan79 commented 7 years ago

Cool, conda finally fixed their package! Glad that worked!

kepbod commented 7 years ago

Hi! I came up with the same question even after I upgraded curl. How shall I solve this problem? Many thanks!

$ conda list | grep curl
curl                      7.52.1                        0    defaults
pycurl                    7.43.0                   py35_2    defaults
dpryan79 commented 7 years ago

What version of OSX are both of you using? Maybe that's the difference? Perhaps I should add an option to libBigWig (what pyBigWig uses) to ignore certificate errors.

kepbod commented 7 years ago

I download some bigwig files from ENCODE, and some of them works, but some are not. In addition, using url directly works well. For example:

pyBigWig.open('https://www.encodeproject.org/files/ENCFF555VCB/@@download/ENCFF555VCB.bigWig')

works well.

pyBigWig.open('ENCFF555VCB.bigwig')

could not work after I download the file to my server. The error message is:

[urlOpen] Couldn't open ENCFF555VCB.bigwig for reading

Sorry, this problem occurs in linux server.

dpryan79 commented 7 years ago

Are you in the same directory as ENCFF555VCB.bigwig?

kepbod commented 7 years ago

Yes. In addition, I am wondering whether it is because the version of my installed pybigwig is too low. I installed it via bioconda, and the latest version of pybigwig is 0.2.8 in bioconda.

dpryan79 commented 7 years ago

@kepbod: You'll get that error whenever there's an error opening a local file. This particular error will usually only happen if the file specified doesn't exist or you don't have rights to open it.

kepbod commented 7 years ago

Oh, it is my fault. I figured out what is wrong that I used the wrong file name. Thanks for your assistance.