Issues with browserSession()

[muqiaowhale]

Hi,

thanks for your awesome package! I’m trying to extract UCSC data using your package. I successfully ran the command below last week but got this error today. Is there anything wrong with the function or UCSC? Could you please help me to figure it out? Thank you so much. Look forward to your reply!

mySession <- rtracklayer::browserSession('UCSC') Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

Jingyu

[maximilianh]

UCSC here, we did a code release on Tuesday (we have one every three weeks). This could be a new rtracklayer bug or it could also related to something that we changed at UCSC, but I have no idea what it could be. We need more details from an rtracklayer developer to say more.

[YiweiNiu]

Same error. Here is the session info

R version 4.3.1 (2023-06-16)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.2 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: Europe/Berlin
tzcode source: system (glibc)

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] BSgenome.Hsapiens.UCSC.hg19_1.4.3 BSgenome_1.68.0                   Biostrings_2.68.1                
 [4] XVector_0.40.0                    rtracklayer_1.60.1                GenomicRanges_1.52.0             
 [7] GenomeInfoDb_1.36.4               IRanges_2.34.1                    S4Vectors_0.38.1                 
[10] BiocGenerics_0.46.0              

loaded via a namespace (and not attached):
 [1] Matrix_1.6-1                compiler_4.3.1              rjson_0.2.21                crayon_1.5.2               
 [5] SummarizedExperiment_1.30.2 Biobase_2.60.0              GenomicAlignments_1.36.0    Rsamtools_2.16.0           
 [9] bitops_1.0-7                parallel_4.3.1              BiocParallel_1.34.2         yaml_2.3.7                 
[13] fastmap_1.1.1               lattice_0.21-8              here_1.0.1                  S4Arrays_1.0.5             
[17] knitr_1.44                  XML_3.99-0.14               DelayedArray_0.26.7         MatrixGenerics_1.12.3      
[21] rprojroot_2.0.3             GenomeInfoDbData_1.2.10     rlang_1.1.1                 xfun_0.40                  
[25] cli_3.6.1                   magrittr_2.0.3              zlibbioc_1.46.0             digest_0.6.33              
[29] grid_4.3.1                  rstudioapi_0.15.0           lifecycle_1.0.3             vctrs_0.6.3                
[33] evaluate_0.21               codetools_0.2-19            abind_1.4-5                 RCurl_1.98-1.12            
[37] restfulr_0.0.15             rmarkdown_2.22              purrr_1.0.2                 matrixStats_1.0.0          
[41] tools_4.3.1                 BiocIO_1.10.0               htmltools_0.5.6

[ukesh-sys]

I am also getting similar error when I try to use makeTxDbFromUCSC("hg38"). The error I am getting is: Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

It was working fine couple of days back.

[hpages]

Same problem here. Breaks rtracklayer RELEASE_3_17 branch (BioC 3.17) and devel branch (BioC 3.18): https://bioconductor.org/checkResults/3.18/bioc-LATEST/rtracklayer/

This started a couple of days ago so could be related with your latest code release @maximilianh.

FWIW this also breaks the following rtracklayer's rev deps: CNVRanger, coMET, customProDB, DMRcate, ELMER, epigraHMM, epimutacions, GenomicFeatures, GenomicRanges, goseq, IdeoViz, MEAL, methylPipe, MethylSeekR, missMethyl, OGRE, segmenter.

[hpages]

Could it be that a redirect was recently implemented from HTTP://genome.ucsc.edu/cgi-bin/hgGateway to HTTPS://genome.ucsc.edu/cgi-bin/hgGateway @maximilianh? To my surprise browserSession() is still using HTTP by default, and is not able to handle the redirect. However, if I do:

session <- browserSession(url="https://genome.ucsc.edu/cgi-bin/")

everything seems to work as usual.

@lawremi @sanchit-saini Do you guys want me to submit a PR?

Thanks

[hpages]

Why are you closing the issue @muqiaowhale?

[hpages]

Thanks for reopening.

Also what error you got when doing browserSession(url="https://genome.ucsc.edu/cgi-bin/"). Is it still "Failed to obtain 'hguid' cookie"? If not, please show the new error.

[muqiaowhale]

@hpages yes, the same error session <- browserSession(url="https://genome.ucsc.edu/cgi-bin/") Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

[hpages]

Hmm.. I just tried this on 6 different machines (2 Ubuntu Linux, 2 Intel Mac, 2 arm64 Mac), in addition to my laptop, and it works on all of them. Maybe it's time for you guys to disclose your sessionInfo()? Here is mine:

library(rtracklayer)

session <- browserSession()
# Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

session <- browserSession(url="https://genome.ucsc.edu/cgi-bin/")
session
# A genome browser session of class 'UCSCSession' with 0 views and 144 tracks

sessionInfo():

R version 4.3.0 Patched (2023-05-31 r84481)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 23.10

Matrix products: default
BLAS:   /home/hpages/R/R-4.3.r84481/lib/libRblas.so 
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.11.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/Los_Angeles
tzcode source: system (glibc)

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
[1] rtracklayer_1.60.1   GenomicRanges_1.52.1 GenomeInfoDb_1.36.4 
[4] IRanges_2.34.1       S4Vectors_0.38.2     BiocGenerics_0.46.0 

loaded via a namespace (and not attached):
 [1] crayon_1.5.2                DelayedArray_0.26.7        
 [3] SummarizedExperiment_1.30.2 GenomicAlignments_1.36.0   
 [5] rjson_0.2.21                RCurl_1.98-1.12            
 [7] Biostrings_2.68.1           XML_3.99-0.14              
 [9] MatrixGenerics_1.12.3       Biobase_2.60.0             
[11] grid_4.3.0                  restfulr_0.0.15            
[13] abind_1.4-5                 bitops_1.0-7               
[15] yaml_2.3.7                  compiler_4.3.0             
[17] codetools_0.2-19            XVector_0.40.0             
[19] BiocParallel_1.34.2         lattice_0.21-9             
[21] BiocIO_1.10.0               parallel_4.3.0             
[23] GenomeInfoDbData_1.2.10     Matrix_1.6-1.1             
[25] tools_4.3.0                 matrixStats_1.0.0          
[27] Rsamtools_2.16.0            zlibbioc_1.46.0            
[29] S4Arrays_1.0.6             

[muqiaowhale]

library(rtracklayer) session <- browserSession(url="https://genome.ucsc.edu/cgi-bin/") Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie sessionInfo() R version 4.2.0 (2022-04-22) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 20.04.4 LTS

Matrix products: default BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3

locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] stats4 stats graphics grDevices utils datasets methods
[8] base

other attached packages: [1] rtracklayer_1.56.1 GenomicRanges_1.48.0 GenomeInfoDb_1.32.2 [4] IRanges_2.30.0 S4Vectors_0.34.0 BiocGenerics_0.42.0

loaded via a namespace (and not attached): [1] XVector_0.36.0 zlibbioc_1.42.0
[3] GenomicAlignments_1.32.1 BiocParallel_1.30.4
[5] lattice_0.20-45 rjson_0.2.21
[7] tools_4.2.0 grid_4.2.0
[9] SummarizedExperiment_1.26.1 parallel_4.2.0
[11] Biobase_2.56.0 matrixStats_0.62.0
[13] yaml_2.3.5 crayon_1.5.1
[15] BiocIO_1.6.0 Matrix_1.5-4
[17] GenomeInfoDbData_1.2.8 restfulr_0.0.15
[19] bitops_1.0-7 codetools_0.2-18
[21] RCurl_1.98-1.7 DelayedArray_0.22.0
[23] compiler_4.2.0 MatrixGenerics_1.8.1
[25] Biostrings_2.64.0 Rsamtools_2.12.0
[27] XML_3.99-0.10

[hpages]

Thanks @muqiaowhale, that's what I suspected i.e. you're using Bioconductor 3.15 which is old, no longer supported, and archived. So you're on your own. Note that current version is 3.17, and 3.18 will be released next week.

In the meantime I went ahead with a Pull Request (see PR #97). @lawremi @sanchit-saini Would be greatly appreciated if you could take a look at it. Thanks!

[slsevilla]

I am able to clear the error with the following code

> library(rtracklayer)
> session <- browserSession()
# Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

> session <- browserSession(url="https://genome.ucsc.edu/cgi-bin/")
> session
# A genome browser session of class 'UCSCSession' with 0 views and 144 tracks

However, when I actually go to pull and plot anything, the error persists:

> library(Gviz)
> library(GenomicRanges)
> data(cpgIslands)
> chr <- as.character(unique(seqnames(cpgIslands)))
> gen <- genome(cpgIslands)
> atrack <- AnnotationTrack(cpgIslands, name = "CpG")
> itrack <- IdeogramTrack(genome = gen, chromosome = chr)
# Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

Info

> sessionInfo()
R version 4.3.0 (2023-04-21)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Ventura 13.6

Matrix products: default
BLAS:   /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/lib/libRlapack.dylib;  LAPACK version 3.11.0

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

time zone: America/New_York
tzcode source: internal

attached base packages:
[1] grid      stats4    stats     graphics  grDevices utils     datasets  methods  
[9] base     

other attached packages:
[1] Gviz_1.44.2          rtracklayer_1.60.1   GenomicRanges_1.52.0 GenomeInfoDb_1.36.4 
[5] IRanges_2.34.1       S4Vectors_0.38.2     BiocGenerics_0.46.0 

[hpages]

@slsevilla That's because there are many other places in rtracklayer internal code where HTTP needs to be replaced with HTTPS. PR #97 (rtracklayer 1.61.2) should take care of all of them.

FWIW your example above works for me with rtracklayer 1.61.2. If you want to give it a try, easiest way is to install rtracklayer 1.61.2 with remotes::install_github("hpages/rtracklayer@https_ucsc"). However you should preferrably upgrade your installation to BioC 3.18 (current devel, will be released next week) before doing that. To upgrade, just run BiocManager::install(version="devel"). This might take a while...

[sanchit-saini]

Hi @hpages Thanks for looking into it.

As you have pointed out, the HTTP needs to be replaced with HTTPS, and you have already fixed it in almost all the places in PR.

I was able to replicate it and the following also needs to be replaced, and I think you were not able to replicate it because it only triggers at the time of mirror redirection, which happens based on IP and server region proximity. I think your IP is close to "https://genome.ucsc.edu" server.

https://github.com/lawremi/rtracklayer/blob/5a681a298f2e4054fb7e3488831b95a1e94ca4ca/R/ucsc.R#L20

url <- sub(".*?a href=\"http([^[:space:]]+cgi-bin/).*", "https\\1", gw)

Can you please also update it in the PR ?

[hpages]

Done: https://github.com/lawremi/rtracklayer/pull/97/commits/b5d2e0f938ecff4f06aaadf27d6b82f720d175b9. Thanks for the feeback @sanchit-saini

[maximilianh]

Hi rtracklayer developers,

yes! We did add this HTTP -> HTTPS redirect, after a ton of testing and external feedback. We didn't think of reaching out to you guys, sorry. We can take back and delay the redirect for a few weeks, but maybe it's easiest if you can update rtracklayer now. Maybe we could also skip the redirect if the request comes from rtracklayer - is there a special user agent?

On Fri, Oct 20, 2023 at 8:17 AM Hervé Pagès @.***> wrote:

Done: b5d2e0f https://github.com/lawremi/rtracklayer/commit/b5d2e0f938ecff4f06aaadf27d6b82f720d175b9. Thanks for the feeback @sanchit-saini https://github.com/sanchit-saini

— Reply to this email directly, view it on GitHub https://github.com/lawremi/rtracklayer/issues/95#issuecomment-1772146578, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TK7M3HJEJ3NSGHEN6LYAIJOBAVCNFSM6AAAAAA6GVNIGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZSGE2DMNJXHA . You are receiving this because you were mentioned.Message ID: @.***>

[hpages]

Thanks for confirming @maximilianh. Let's hope PR #97 can be quickly merged (only 3 days left for that @lawremi).

[ukesh-sys]

Hi there, I see the issue was closed but I still get error when I tried to use: txdb <- makeTxDbFromUCSC("hg38") Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

[hpages]

@ukesh-sys Issue is closed because it is fixed in the devel branch of rtracklayer and GenomicFeatures. You'll need to install rtracklayer 1.61.2 and GenomicFeatures 1.53.3 to effectively get the fix. Before you do so, make sure that you are running BioC 3.18, because these packages are NOT guaranteed to be compatible with previous versions of Bioconductor. Then wait until tomorrow and run BiocManager::install(), or, if you can't wait, do remotes::install_github(c("Bioconductor/GenomicFeatures", "lawremi/rtracklayer")) now. However, note that installing Bioconductor packages directly from GitHub is almost never a good idea, so please consider this as an exception to the rule.

[ukesh-sys]

Thank you so much @hpages for the clarification. I will wait till tomorrow and continue my analysis. Thank you everyone in the team for quick responses and the fix.

[maximilianh]

Hey @hpages, does rtracklayer send a special user agent string or is there some other way that we could exclude rtracklayer requests from the redirect rule ?

On Sat, Oct 21, 2023 at 04:35 Ukesh Karki @.***> wrote:

Thank you so much @hpages https://github.com/hpages for the clarification. I will wait till tomorrow and continue my analysis. Thank you everyone in the team for quick responses and the fix.

— Reply to this email directly, view it on GitHub https://github.com/lawremi/rtracklayer/issues/95#issuecomment-1773615632, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACL4TIXYOVFD5PTKP3EAKTYAMYFXAVCNFSM6AAAAAA6GVNIGGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZTGYYTKNRTGI . You are receiving this because you were mentioned.Message ID: @.***>

[sanchit-saini]

hi @maximilianh user agent string is : rtracklayer https://github.com/lawremi/rtracklayer/blob/07ed9800dc956ca6250ef7c516f36048ed1a86e2/R/web.R#L27

[slsevilla]

I've updated my rtracklayer, and am now getting a different error:

library(rtracklayer)
session <- browserSession()
# Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

session <- browserSession(url="https://genome.ucsc.edu/cgi-bin/")
session
# A genome browser session of class 'UCSCSession' with 0 views and 144 tracks

library(Gviz)
library(GenomicRanges)
data(cpgIslands)
chr <- as.character(unique(seqnames(cpgIslands)))
gen <- genome(cpgIslands)
atrack <- AnnotationTrack(cpgIslands, name = "CpG")
itrack <- IdeogramTrack(genome = gen, chromosome = chr)

# Error in names(df) <- names : attempt to set an attribute on NULL
# In addition: Warning message:
# In curlSetOpt(..., .opts = .opts, curl = h, .encoding = .encoding) :
# Error setting the option for # 3 (status = 43) (enum = 81) (value = 0x555578cccbc0): A libcurl function was given a bad # argument CURLOPT_SSL_VERIFYHOST no longer supports 1 as value!

Session info:

R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Rocky Linux 8.7 (Green Obsidian)

Matrix products: default
BLAS/LAPACK: /usr/local/intel/2022.1.2.146/mkl/2022.0.2/lib/intel64/libmkl_rt.so.2;  LAPACK version 3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/New_York
tzcode source: system (glibc)

attached base packages:
[1] grid      stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Gviz_1.44.2          rtracklayer_1.61.2   GenomicRanges_1.52.1 GenomeInfoDb_1.36.4 
[5] IRanges_2.34.1       S4Vectors_0.38.2     BiocGenerics_0.46.0 

[hpages]

@slsevilla Please make sure to report the full output of sessionInfo() that you get right after running the code you're showing above. Otherwise it's meaningless. Thanks

[hpages]

@slsevilla Also it seems that you are still running BioC 3.17. As explained earlier, you first need to upgrade your installation to BioC 3.18 to benefit from the fixes. Furthermore, there are other important fixes that were applied to the latest GenomicFeatures that you will need but won't get if you don't upgrade your installation to BioC 3.18 first. That was also explained earlier.

[slsevilla]

@hpages - Same outcome even after updating Biocmanager=3.18 and doing a github install which changedrtracklayer=1.62.0 and GenomicFeatures_1.55.0.

Initial browserSession call does not produce an error, but loading IdeogramTrack still does.

> library(rtracklayer)
> session <- browserSession()
> session
A genome browser session of class 'UCSCSession' with 0 views and 144 tracks
> library(Gviz)
> library(GenomicRanges)
> data(cpgIslands)
> chr <- as.character(unique(seqnames(cpgIslands)))
> gen <- genome(cpgIslands)
> atrack <- AnnotationTrack(cpgIslands, name = "CpG")
> itrack <- IdeogramTrack(genome = gen, chromosome = chr)
Error in names(df) <- names : attempt to set an attribute on NULL
In addition: Warning message:
In curlSetOpt(..., .opts = .opts, curl = h, .encoding = .encoding) :
  Error setting the option for # 3 (status = 43) (enum = 81) (value = 0x55559911ef50): A libcurl function was given a bad argument CURLOPT_SSL_VERIFYHOST no longer supports 1 as value!

Setting session does not fix the issue

> session <- browserSession(url="https://genome.ucsc.edu/cgi-bin/")
> session
A genome browser session of class 'UCSCSession' with 0 views and 144 tracks
> itrack <- IdeogramTrack(genome = gen, chromosome = chr)
Error in names(df) <- names : attempt to set an attribute on NULL
In addition: Warning message:
In curlSetOpt(..., .opts = .opts, curl = h, .encoding = .encoding) :
  Error setting the option for # 3 (status = 43) (enum = 81) (value = 0x55559c32d060): A libcurl function was given a bad argument CURLOPT_SSL_VERIFYHOST no longer supports 1 as value!

Session info

> BiocManager::version()
[1] '3.18'
> sessionInfo()
R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Rocky Linux 8.7 (Green Obsidian)

Matrix products: default
BLAS/LAPACK: /usr/local/intel/2022.1.2.146/mkl/2022.0.2/lib/intel64/libmkl_rt.so.2;  LAPACK version 3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/New_York
tzcode source: system (glibc)

attached base packages:
[1] grid      stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Gviz_1.46.0          rtracklayer_1.62.0   GenomicRanges_1.54.0 GenomeInfoDb_1.38.0 
[5] IRanges_2.36.0       S4Vectors_0.40.0     BiocGenerics_0.48.0 

loaded via a namespace (and not attached):
  [1] DBI_1.1.3                   bitops_1.0-7                deldir_1.0-9               
  [4] gridExtra_2.3               biomaRt_2.58.0              rlang_1.1.1                
  [7] magrittr_2.0.3              biovizBase_1.50.0           matrixStats_1.0.0          
 [10] compiler_4.3.0              RSQLite_2.3.1               GenomicFeatures_1.55.0     
 [13] png_0.1-8                   vctrs_0.6.4                 ProtGenerics_1.34.0        
 [16] stringr_1.5.0               pkgconfig_2.0.3             crayon_1.5.2               
 [19] fastmap_1.1.1               backports_1.4.1             dbplyr_2.3.4               
 [22] XVector_0.42.0              utf8_1.2.4                  Rsamtools_2.18.0           
 [25] rmarkdown_2.25              bit_4.0.5                   xfun_0.40                  
 [28] zlibbioc_1.48.0             cachem_1.0.8                progress_1.2.2             
 [31] blob_1.2.4                  DelayedArray_0.28.0         BiocParallel_1.36.0        
 [34] jpeg_0.1-10                 parallel_4.3.0              prettyunits_1.2.0          
 [37] cluster_2.1.4               VariantAnnotation_1.48.0    R6_2.5.1                   
 [40] stringi_1.7.12              RColorBrewer_1.1-3          rpart_4.1.21               
 [43] Rcpp_1.0.11                 SummarizedExperiment_1.32.0 knitr_1.44                 
 [46] base64enc_0.1-3             Matrix_1.6-1.1              nnet_7.3-19                
 [49] tidyselect_1.2.0            rstudioapi_0.15.0           dichromat_2.0-0.1          
 [52] abind_1.4-5                 yaml_2.3.7                  codetools_0.2-19           
 [55] curl_5.1.0                  lattice_0.22-5              tibble_3.2.1               
 [58] Biobase_2.62.0              KEGGREST_1.42.0             evaluate_0.22              
 [61] foreign_0.8-85              BiocFileCache_2.10.0        xml2_1.3.5                 
 [64] Biostrings_2.70.0           pillar_1.9.0                BiocManager_1.30.22        
 [67] filelock_1.0.2              MatrixGenerics_1.14.0       checkmate_2.3.0            
 [70] generics_0.1.3              RCurl_1.98-1.12             ensembldb_2.26.0           
 [73] hms_1.1.3                   ggplot2_3.4.4               munsell_0.5.0              
 [76] scales_1.2.1                glue_1.6.2                  Hmisc_5.1-1                
 [79] lazyeval_0.2.2              tools_4.3.0                 interp_1.1-4               
 [82] BiocIO_1.12.0               data.table_1.14.8           BSgenome_1.70.0            
 [85] GenomicAlignments_1.38.0    XML_3.99-0.14               latticeExtra_0.6-30        
 [88] AnnotationDbi_1.64.0        colorspace_2.1-0            GenomeInfoDbData_1.2.11    
 [91] htmlTable_2.4.1             restfulr_0.0.15             Formula_1.2-5              
 [94] cli_3.6.1                   rappdirs_0.3.3              fansi_1.0.5                
 [97] S4Arrays_1.2.0              dplyr_1.1.3                 AnnotationFilter_1.26.0    
[100] gtable_0.3.4                digest_0.6.33               SparseArray_1.2.0          
[103] rjson_0.2.21                htmlwidgets_1.6.2           memoise_2.0.1              
[106] htmltools_0.5.6.1           lifecycle_1.0.3             httr_1.4.7                 
[109] bit64_4.0.5                

[hpages]

Thank you @slsevilla

At least now session <- browserSession() works for you, so there's some progress.

As for the IdeogramTrack() error, this seems to be a different error so you will need to open a new issue for that. FWIW I've tried your code on 4 different machines (2 Linux and 2 Mac) and I can't reproduce the problem. Before you open a new issue though, let's try to narrow this down. Does the following work for you?

library(rtracklayer)
ucscGenomes()

If this fails, no need to go further, please report the problem in a new issue.

If it works, then keep reading.

Now does the following code work for you?

library(Gviz)
sessionInfo <- Gviz:::.cacheGenomes(c(chr7="hg19"))

Gviz:::.cacheGenomes() is called internally by IdeogramTrack() and it makes calls to rtracklayer to retrieve the track data.

If the Gviz:::.cacheGenomes() code above fails, please open a new issue in this repo (rtracklayer) showing the code + error + sessionInfo(). However, if it works, then it's not clear to me that your IdeogramTrack() error is related to rtracklayer, so in this case I would suggest that you open an issue in the Gviz repository.

Thanks, H.

[lawremi]

See issue #101.

[maximilianh]

OK we've excluded this user agent from the redirect. Can you test this and tell us if this works with rtracklayer for the machine genome-test.gi.ucsc.edu:

$ curl -v http://genome-test.gi.ucsc.edu/ 2>&1 | grep Location -i < Location: https://genome-test.gi.ucsc.edu/

Redirect is not done if the useragent is "rtracklayer":

$ curl --user-agent=rtracklayer -v http://genome-test.gi.ucsc.edu/ 2>&1 | grep Location -i

If you can confirm that this solves the problem for older rtracklayer versions, we can roll out this change to the redirect rule to all servers.

Sorry for the delay, I sent my emails to the wrong person, otherwise this could have been applied right away.

[maximilianh]

Ignore my previous message, this is not working yet. I'll investigate and get back to you.

[maximilianh]

OK, we made this change on a test site. Can you try an old version of rtracklayer against http://genome-test.gi.ucsc.edu and see if that fixes it?

This command does not bring up a HTTPS redirect anymore: curl -A rtracklayer -v http://genome-test.gi.ucsc.edu/ > /dev/null

Once you confirm that it works, we can roll out the change to the public servers.

[slsevilla]

FWIW I attempted to implement this with an older version of rtracklayer (rtracklayer_1.56.1) and still ran into errors.

Running old version of rtracklayer doesn't produce an error at session, but itrack still produces an error (#101):

> library(rtracklayer)
> session <- browserSession()
> session
A genome browser session of class 'UCSCSession' with 0 views and 144 tracks
> library(Gviz)
> library(GenomicRanges)
> data(cpgIslands)
> chr <- as.character(unique(seqnames(cpgIslands)))
> gen <- genome(cpgIslands)
> atrack <- AnnotationTrack(cpgIslands, name = "CpG")
> itrack <- IdeogramTrack(genome = gen, chromosome = chr)
Error in names(df) <- names : attempt to set an attribute on NULL
In addition: Warning message:
In curlSetOpt(..., .opts = .opts, curl = h, .encoding = .encoding) :
  Error setting the option for # 3 (status = 43) (enum = 81) (value = 0x5555a4bdf690): A libcurl function was given a bad argument CURLOPT_SSL_VERIFYHOST no longer supports 1 as value!

I tried setting the session to this test site, but now I'm getting the hguid error again:

> session <- browserSession(url="http://genome-test.gi.ucsc.edu")
Error in .local(.Object, ...) : Failed to obtain 'hguid' cookie

Session info

> sessionInfo()
R version 4.3.0 (2023-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Rocky Linux 8.7 (Green Obsidian)

Matrix products: default
BLAS/LAPACK: /usr/local/intel/2022.1.2.146/mkl/2022.0.2/lib/intel64/libmkl_rt.so.2;  LAPACK version 3.9.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: America/New_York
tzcode source: system (glibc)

attached base packages:
[1] grid      stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Gviz_1.44.2          rtracklayer_1.56.1   GenomicRanges_1.54.1 GenomeInfoDb_1.38.1  IRanges_2.36.0      
[6] S4Vectors_0.40.1     BiocGenerics_0.48.1 

loaded via a namespace (and not attached):
  [1] RColorBrewer_1.1-3          rstudioapi_0.15.0           magrittr_2.0.3             
  [4] GenomicFeatures_1.52.2      rmarkdown_2.25              BiocIO_1.12.0              
  [7] zlibbioc_1.48.0             vctrs_0.6.4                 memoise_2.0.1              
 [10] Rsamtools_2.18.0            RCurl_1.98-1.13             base64enc_0.1-3            
 [13] htmltools_0.5.6.1           S4Arrays_1.2.0              progress_1.2.2             
 [16] curl_5.1.0                  SparseArray_1.2.2           Formula_1.2-5              
 [19] desc_1.4.2                  htmlwidgets_1.6.2           cachem_1.0.8               
 [22] GenomicAlignments_1.38.0    lifecycle_1.0.3             pkgconfig_2.0.3            
 [25] Matrix_1.6-2                R6_2.5.1                    fastmap_1.1.1              
 [28] GenomeInfoDbData_1.2.11     MatrixGenerics_1.14.0       digest_0.6.33              
 [31] colorspace_2.1-0            AnnotationDbi_1.62.2        ps_1.7.5                   
 [34] rprojroot_2.0.3             Hmisc_5.1-1                 RSQLite_2.3.1              
 [37] filelock_1.0.2              fansi_1.0.5                 httr_1.4.7                 
 [40] abind_1.4-5                 compiler_4.3.0              remotes_2.4.2.1            
 [43] bit64_4.0.5                 htmlTable_2.4.1             backports_1.4.1            
 [46] BiocParallel_1.36.0         DBI_1.1.3                   pkgbuild_1.4.2             
 [49] biomaRt_2.56.1              rappdirs_0.3.3              DelayedArray_0.28.0        
 [52] rjson_0.2.21                tools_4.3.0                 foreign_0.8-84             
 [55] nnet_7.3-19                 glue_1.6.2                  restfulr_0.0.15            
 [58] callr_3.7.3                 checkmate_2.2.0             cluster_2.1.4              
 [61] generics_0.1.3              gtable_0.3.4                BSgenome_1.68.0            
 [64] ensembldb_2.24.1            data.table_1.14.8           hms_1.1.3                  
 [67] xml2_1.3.5                  utf8_1.2.3                  XVector_0.42.0             
 [70] pillar_1.9.0                stringr_1.5.0               dplyr_1.1.3                
 [73] BiocFileCache_2.8.0         lattice_0.22-5              bit_4.0.5                  
 [76] deldir_1.0-9                biovizBase_1.48.0           tidyselect_1.2.0           
 [79] Biostrings_2.70.1           knitr_1.44                  gridExtra_2.3              
 [82] ProtGenerics_1.32.0         SummarizedExperiment_1.32.0 xfun_0.40                  
 [85] Biobase_2.62.0              matrixStats_1.1.0           DT_0.30                    
 [88] stringi_1.7.12              lazyeval_0.2.2              yaml_2.3.7                 
 [91] evaluate_0.22               codetools_0.2-19            interp_1.1-4               
 [94] tibble_3.2.1                BiocManager_1.30.22         cli_3.6.1                  
 [97] rpart_4.1.19                munsell_0.5.0               processx_3.8.2             
[100] dichromat_2.0-0.1           Rcpp_1.0.11                 dbplyr_2.3.4               
[103] png_0.1-8                   XML_3.99-0.15               parallel_4.3.0             
[106] ggplot2_3.4.4               blob_1.2.4                  prettyunits_1.2.0          
[109] latticeExtra_0.6-30         jpeg_0.1-10                 AnnotationFilter_1.24.0    
[112] bitops_1.0-7                VariantAnnotation_1.46.0    scales_1.2.1               
[115] crayon_1.5.2                rlang_1.1.1                 KEGGREST_1.40.1   

[maximilianh]

Hi @slsevilla , I can't help with this. I confirmed that with the user agent "rtracklayer" genome-test.gi.ucsc.edu doesn't redirect anymore, but beyond that, I can't do anything. I can't debug rtracklayer...

Here is the command that shows that this server does not redirect to https anymore:

$ curl -A rtracklayer -v http://genome-test.gi.ucsc.edu/ > /dev/null

[maximilianh]

Any news from the rtracklayer developers? Or should we just close this?

[slsevilla]

I haven't had any update. They are sort of saying the same thing - they dont see the issue, although there are several other people who are in the same situation as me.

[hpages]

But the redirect is still in place for http://genome.ucsc.edu/cgi-bin/hgGateway @maximilianh:

hpages@XPS15:~$ curl -A rtracklayer -v http://genome.ucsc.edu/cgi-bin/hgGateway >/dev/null
* processing: http://genome.ucsc.edu/cgi-bin/hgGateway
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 128.114.119.132:80...
* Connected to genome.ucsc.edu (128.114.119.132) port 80
> GET /cgi-bin/hgGateway HTTP/1.1
> Host: genome.ucsc.edu
> User-Agent: rtracklayer
> Accept: */*
> 
< HTTP/1.1 302 Found
< Date: Mon, 20 Nov 2023 19:30:50 GMT
< Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9
< Location: https://genome.ucsc.edu/cgi-bin/hgGateway
< Content-Length: 225
< Content-Type: text/html; charset=iso-8859-1
< 
{ [225 bytes data]
100   225  100   225    0     0   2876      0 --:--:-- --:--:-- --:--:--  2848
* Connection #0 to host genome.ucsc.edu left intact

As long as the redirect is active, old versions of rtracklayer will remain broken.

[hpages]

@maximilianh Any chance the redirect can be removed for http://genome.ucsc.edu/cgi-bin/hgGateway when user agent is "rtracklayer"?

[maximilianh]

This has been rolled out to genome.ucsc.edu, genome-asia.ucsc.edu and genome-euro.ucsc.edu. Can you try again? Sorry for the long delay, we had some issues with our apache configs (and sorted them out now)

[hpages]

ok great, thanks @maximilianh

@slsevilla Can you test with rtracklayer 1.56.1 again?

[slsevilla]

Hi! I don't have access to R 4.2 anymore... will it work with any later version?