search

dpryan79 / libBigWig

A C library for handling bigWig files
MIT License
73 stars 25 forks source link

URL with Temporary Redirect #24

Closed lidaof closed 7 years ago

lidaof commented 7 years ago

I am trying to fetch some bigWig file from ENCODE project, seems they hosted their files at Amazon S3, using testRemote gives me error. wondering is there a solution? Thanks.

$ ./testRemote https://www.encodeproject.org/files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig
[bwHdrRead] There was an error while reading in the header!
An error occured while opening https://www.encodeproject.org/files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig
$ curl -I https://www.encodeproject.org/files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig
HTTP/1.1 307 Temporary Redirect
Server: nginx/1.10.1
Date: Fri, 17 Mar 2017 21:08:37 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 1196
Connection: keep-alive
X-Request-URL: https://www.encodeproject.org/files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig
X-Stats: es_count=1&es_time=7226&queue_begin=1489784917370001&queue_time=830&rss_begin=492408832&rss_change=0&rss_end=492408832&wsgi_begin=1489784917370831&wsgi_end=1489784917389025&wsgi_time=18194
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, HEAD
Access-Control-Allow-Headers: Accept, Origin, Range, X-Requested-With
Access-Control-Expose-Headers: Content-Length, Content-Range, Content-Type
Location: https://download.encodeproject.org/http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=Tb43S%2BXE%2BOT0jVVRYn1E1amZZss%3D&Expires=1489914517&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D
Strict-Transport-Security: max-age=15768000
lidaof commented 7 years ago

Tried add this in the curl callback function, but it's not working

   /* example.com is redirected, so we tell libcurl to follow redirection */
   rv = curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
dpryan79 commented 7 years ago

Oddly, libBigWig isn't getting a redirect, it's getting a "401 Unauthorized" message and I haven't a clue why.

dpryan79 commented 7 years ago

BTW, you can add a rv = curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); and then you'll see the underlying curl queries and responses.

lidaof commented 7 years ago

wondering why testBigBed is working:

./testBigBed https://www.encodeproject.org/files/ENCFF001JBR/@@download/ENCFF001JBR.bigBed

dpryan79 commented 7 years ago

I suspect I'll have to use a packet sniffer to debug this when I have time, you're certainly correct that this seems odd.

lidaof commented 7 years ago

Thanks for looking into this.

dpryan79 commented 7 years ago

Nevermind, apparently it didn't like the default "basic" authentication method. Please try adding in the following:

rv = curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
lidaof commented 7 years ago

we are making progress.

got an error in bwStatsFromZoom in the range 27053199-27059606: Resource temporarily unavailable

dpryan79 commented 7 years ago

If you enable CURLOPT_VERBOSE, what does the remote server return just before this? I ask mostly since this solved the issue for me.

lidaof commented 7 years ago

yeah, got tons of output....

 ./querybw https://www.encodeproject.org/files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig chr7 27053199 27373567 50 bwOutput.txt 1
* About to connect() to www.encodeproject.org port 443 (#0)
*   Trying 171.67.205.70... * connected
* found 153 certificates in /etc/ssl/certs/ca-certificates.crt
*    server certificate verification OK
*    common name: *.encodeproject.org (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: OU=Domain Control Validated,CN=*.encodeproject.org
*    start date: Thu, 14 Aug 2014 14:12:02 GMT
*    expire date: Fri, 02 Aug 2019 08:32:08 GMT
*    issuer: C=BE,O=GlobalSign nv-sa,CN=AlphaSSL CA - SHA256 - G2
*    compression: NULL
*    cipher: AES-128-CBC
*    MAC: SHA256
> GET /files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig HTTP/1.1
Range: bytes=0-131071
Host: www.encodeproject.org
Accept: */*

< HTTP/1.1 307 Temporary Redirect
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:32 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 1192
< Connection: keep-alive
< X-Request-URL: https://www.encodeproject.org/files/ENCFF188HKC/@@download/ENCFF188HKC.bigWig
< X-Stats: es_count=1&es_time=6901&queue_begin=1489787732112648&queue_time=917&rss_begin=504201216&rss_change=0&rss_end=504201216&wsgi_begin=1489787732113565&wsgi_end=1489787732134395&wsgi_time=20830
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET, HEAD
< Access-Control-Allow-Headers: Accept, Origin, Range, X-Requested-With
< Access-Control-Expose-Headers: Content-Length, Content-Range, Content-Type
< Location: https://download.encodeproject.org/http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D
< Strict-Transport-Security: max-age=15768000
< 
* Ignoring the response-body
* Connection #0 to host www.encodeproject.org left intact
* Issue another request to this URL: 'https://download.encodeproject.org/http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D'
* About to connect() to download.encodeproject.org port 443 (#1)
*   Trying 171.67.205.70... * connected
* found 153 certificates in /etc/ssl/certs/ca-certificates.crt
*    server certificate verification OK
*    common name: *.encodeproject.org (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: RSA
*    certificate version: #3
*    subject: OU=Domain Control Validated,CN=*.encodeproject.org
*    start date: Thu, 14 Aug 2014 14:12:02 GMT
*    expire date: Fri, 02 Aug 2019 08:32:08 GMT
*    issuer: C=BE,O=GlobalSign nv-sa,CN=AlphaSSL CA - SHA256 - G2
*    compression: NULL
*    cipher: AES-128-CBC
*    MAC: SHA256
> GET /http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D HTTP/1.1
Range: bytes=0-131071
Host: download.encodeproject.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:32 GMT
< Content-Type: binary/octet-stream
< Content-Length: 131072
< Connection: keep-alive
< x-amz-id-2: JpA2cB854K6fVz+112Ovylal2+AirfoNifFZOCKt3I3ke/MQ61/SaT8PEOIEqaOfeHL7Um86PHM=
< x-amz-request-id: D1314CFF89768692
< Last-Modified: Sun, 05 Mar 2017 05:25:16 GMT
< ETag: "27a0a165fbaae852aafdb97860bce35a-141"
< x-amz-version-id: nJS91vm0UxKvoK9zaXN6by.IC9VgKm52
< Content-Disposition: attachment; filename=ENCFF188HKC.bigWig
< Accept-Ranges: bytes
< Content-Range: bytes 0-131071/1176156755
< 
* Connection #1 to host download.encodeproject.org left intact
* Re-using existing connection! (#1) with host download.encodeproject.org
* Connected to download.encodeproject.org (171.67.205.70) port 443 (#1)
> GET /http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D HTTP/1.1
Range: bytes=924251599-924382670
Host: download.encodeproject.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:33 GMT
< Content-Type: binary/octet-stream
< Content-Length: 131072
< Connection: keep-alive
< x-amz-id-2: cj+yFdyady4mEY6r6M/8yibNhijgyvRkvZXP39pP+P9z2pOCsgIiraBfjO9LiK6QchvgoWuQFdE=
< x-amz-request-id: 0C7711F4484A095C
< Last-Modified: Sun, 05 Mar 2017 05:25:16 GMT
< ETag: "27a0a165fbaae852aafdb97860bce35a-141"
< x-amz-version-id: nJS91vm0UxKvoK9zaXN6by.IC9VgKm52
< Content-Disposition: attachment; filename=ENCFF188HKC.bigWig
< Accept-Ranges: bytes
< Content-Range: bytes 924251599-924382670/1176156755
< 
* Connection #1 to host download.encodeproject.org left intact
* Re-using existing connection! (#1) with host download.encodeproject.org
* Connected to download.encodeproject.org (171.67.205.70) port 443 (#1)
> GET /http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D HTTP/1.1
Range: bytes=1160431630-1160562701
Host: download.encodeproject.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:33 GMT
< Content-Type: binary/octet-stream
< Content-Length: 131072
< Connection: keep-alive
< x-amz-id-2: KVfKcEFlAu9gJmS5O0e12KaEolZAFlsy3j/5tGjJhnwRXYIUhQf3EklfMK0H2B92jMvNv00MmwQ=
< x-amz-request-id: 06204334E134A86A
< Last-Modified: Sun, 05 Mar 2017 05:25:16 GMT
< ETag: "27a0a165fbaae852aafdb97860bce35a-141"
< x-amz-version-id: nJS91vm0UxKvoK9zaXN6by.IC9VgKm52
< Content-Disposition: attachment; filename=ENCFF188HKC.bigWig
< Accept-Ranges: bytes
< Content-Range: bytes 1160431630-1160562701/1176156755
< 
* Connection #1 to host download.encodeproject.org left intact
* Re-using existing connection! (#1) with host download.encodeproject.org
* Connected to download.encodeproject.org (171.67.205.70) port 443 (#1)
> GET /http://encode-files.s3.amazonaws.com/2017/03/05/d7be9e16-e742-4554-9e9b-347834665817/ENCFF188HKC.bigWig?Signature=h7TAvD1OvLohS282XhnUghbM2Ec%3D&Expires=1489917332&AWSAccessKeyId=ASIAIBGS2LKKZLYIOLVA&response-content-disposition=attachment%3B%20filename%3DENCFF188HKC.bigWig&x-amz-security-token=FQoDYXdzEB4aDKo3b1EtKQK0xzlrqCK3A8jMrctRMooXvbFhPZaBtN46iqYhdsIuZVnmCBYphXlMoRFfa%2B7dyVq1ICoFY7d6wrVj2sKHs4VfVMOYlRJOHonPlRj9BvF5DYR8EHZaItBq4ouDlkzOYcrCNbo36uR1IP%2BsDlX8vwqn7hw6ri/wtQYjReE35P8wyG7D3cN4cHZFm2bAmd4xfS6o7vsgh21LfSHjhKIg%2BoQqPoxZwdNB64qlUBrKYo%2BnhDQdKDceMc/0GB9NJqy1U1n0kaXitFHSwg88LzgXR/CY2Eyk/tQVcScceLERWAupB9nLyVpsVH1uSOumFhwcSf1FXEyqFCKWf4jgUqBHJ7T7kfUHmKcLP8VbJgQs0/TB7q8OY0fn7lzugK4kTXkF3GoGdI8aUwNBo2VuA7Z1S0ldUntTMHeh%2Bl9x9nLETbzmVSOB/qIlP%2BZimKVJTMszPoj57cqhxAUd/%2B6xfdiZbjjoh6NSY5rI%2BfMWfTqZS5my6aCIVXTjkg26Bfm4HGl04bNXhJp7uTiIeorwMex0yfSxFJnybunDaLv9fuFmqsOpY7tOXNzmbina6Tp38euVaLnHD/9rAKNpTu6C7pvyUWQohJ6xxgU%3D HTTP/1.1
Range: bytes=1141013476-1141144547
Host: download.encodeproject.org
Accept: */*

< HTTP/1.1 206 Partial Content
< Server: nginx/1.10.1
< Date: Fri, 17 Mar 2017 21:55:34 GMT
< Content-Type: binary/octet-stream
< Content-Length: 131072
< Connection: keep-alive
< x-amz-id-2: oNjXzR17jQCdvrrznQ4Mid4aObOokUVp3a/awpyCQFZiiP9esEMhauW6JYMA1d96VQqVZjhWZ+g=
< x-amz-request-id: AC8B8ABBCDBB6905
< Last-Modified: Sun, 05 Mar 2017 05:25:16 GMT
< ETag: "27a0a165fbaae852aafdb97860bce35a-141"
< x-amz-version-id: nJS91vm0UxKvoK9zaXN6by.IC9VgKm52
< Content-Disposition: attachment; filename=ENCFF188HKC.bigWig
< Accept-Ranges: bytes
< Content-Range: bytes 1141013476-1141144547/1176156755
< 
* Connection #1 to host download.encodeproject.org left intact
got an error in bwStatsFromZoom in the range 27053199-27059606: Resource temporarily unavailable
lidaof commented 7 years ago

the testRemote is working after adding: rv = curl_easy_setopt(curl, CURLOPT_HTTPAUTH, CURLAUTH_ANY); my problem is now with bwStatsFromZoom

dpryan79 commented 7 years ago

@lidaof My guess is that you're not setting the zoom level. I don't export bwStatsFromZoom in the header, so I imagine you're getting some warnings indicating that. If you really really want to directly use that function, then its syntax is bwStatsFromZoom(bw, level, tid, start, end, nBins, type), where bw is the bigWigFile_t pointer, level is an int32_t specifying which of the zoom levels to use, tid is the numeric index of the chromosome/contig, and start/end/nBins/type are as specified for bwStats(). I never expected anyone to directly use that function, so I never documented or exported it.

lidaof commented 7 years ago

Thanks @dpryan79 . is the level parameter be 1 or 0, and 1 for full chromosome data fetch? could the tid be chr1 or chr2?

dpryan79 commented 7 years ago

The level parameter is 0 through n-1, indicating the n zoom levels. There is no level to indicate full chromosome, since these zoom levels are intrinsic to how a given bigWig file was made. I'll just point you to the code for bwStats(), which is what I would recommend people use unless they explicitly need exact results and don't care about the extra overhead incurred by bwStatsFromFull().

As an aside, I mention some of the details about what zoom levels in the python wrapper.

lidaof commented 7 years ago

Thanks for the reply @dpryan79. If I understand you correct, are you suggesting me using bwStatsFromZoom instead of bwStats? I am confusing about how different they would be? I think I need also call determineZoomLevel and bwGetTid, basically like copy all lines from bwStats instead of call bwStats directly? Sorry...I am not quite understand bigWig or bigBed actually.

lidaof commented 7 years ago

Closed this as it's indeed working sometimes. Thanks.