Fix failing R3 tests - Githubissues

johnbradley commented 1 year ago

The CircleCI tests for R 3.6.3 are now fail with the following error:

> devtools::install_deps(dep = TRUE)

Skipping 11 packages not available: jsonlite, httr, dplyr, rlang, stringi, RNeXML, roxygen2, knitr, testthat, plyr, rmarkdown
...
ERROR: dependency ‘RNeXML’ is not available for package ‘rphenoscape’
* removing ‘/tmp/Rtmp7Bu8Gl/Rinst87516ff7c6/rphenoscape’

johnbradley commented 1 year ago

I was able to reproduce this problem with docker just installing "RNeXML".

docker run -it rocker/verse:3.6.3 R
> install.packages('RNeXML')
Installing package into ‘/usr/local/lib/R/site-library’
(as ‘lib’ is unspecified)
Warning: unable to access index for repository https://mran.microsoft.com/snapshot/2020-04-24/src/contrib:
  cannot open URL 'https://mran.microsoft.com/snapshot/2020-04-24/src/contrib/PACKAGES'
Warning message:
package ‘RNeXML’ is not available (for R version 3.6.3)

johnbradley commented 1 year ago

I think rocker used MRAN for reproducibility: https://mran.microsoft.com/documents/rro/reproducibility Unfortunately MRAN is begin shutdown https://techcommunity.microsoft.com/t5/azure-sql-blog/microsoft-r-application-network-retirement/ba-p/3707161.

johnbradley commented 1 year ago

I created a branch that uses cloud.r-project.org instead of MRAN: https://github.com/phenoscape/rphenoscape/blob/36a73a8d2ce073fd9a31ee881b8959a85b8c629c/.circleci/config.yml#L11

Unfortunately the R3 test fails building the vignette when checking the cert for kb.phenoscape.org:

Quitting from lines 37-38 [unnamed-chunk-2] (rphenoscape.Rmd)
Error: processing vignette 'rphenoscape.Rmd' failed with diagnostics:
SSL peer certificate or SSH remote key was not OK: [kb.phenoscape.org] server certificate verification failed. CAfile: none CRLfile: none
--- failed re-building ‘rphenoscape.Rmd’

SUMMARY: processing the following file failed:
  ‘rphenoscape.Rmd’

Error: Vignette re-building failed.
Execution halted

https://app.circleci.com/pipelines/github/phenoscape/rphenoscape/163/workflows/2a81bb37-c235-43e3-9fa8-e248208c7eee/jobs/386

hlapp commented 1 year ago

Does this mean we need to install the Lets Encrypt CA?

johnbradley commented 1 year ago

I was able to curl kb.phenoscape.org within the base docker container without error:

$ docker run -it rocker/verse:3.6.3 bash
root@01450cd2b7bb:/# curl https://kb.phenoscape.org
<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Phenoscape Knowledgebase</title>

johnbradley commented 1 year ago

It looks like installing our dependencies is breaking SSL. The following Dockerfile shows the problem:

FROM rocker/verse:3.6.3
ADD . /src
WORKDIR /src
RUN R -e "httr::GET('https://kb.phenoscape.org')"
RUN R -e "devtools::install_deps(dep = TRUE, repos = 'https://cloud.r-project.org')"
RUN R -e "httr::GET('https://kb.phenoscape.org')"

Output from docker build ...:

 => CACHED [4/8] RUN R -e "httr::GET('https://kb.phenoscape.org')"                                                             0.0s
 => [5/8] RUN R -e "devtools::install_deps(dep = TRUE, repos = 'https://cloud.r-project.org')"                               665.2s
 => ERROR [6/8] RUN R -e "httr::GET('https://kb.phenoscape.org')"                                                              0.7s
------                                                                                                                              
 > [6/8] RUN R -e "httr::GET('https://kb.phenoscape.org')":                                                                         
#10 0.328                                                                                                                           
#10 0.328 R version 3.6.3 (2020-02-29) -- "Holding the Windsock"                                                                    
#10 0.328 Copyright (C) 2020 The R Foundation for Statistical Computing                                                             
#10 0.328 Platform: x86_64-pc-linux-gnu (64-bit)                                                                                    
#10 0.328 
#10 0.328 R is free software and comes with ABSOLUTELY NO WARRANTY.
#10 0.328 You are welcome to redistribute it under certain conditions.
#10 0.328 Type 'license()' or 'licence()' for distribution details.
#10 0.328 
#10 0.328 R is a collaborative project with many contributors.
#10 0.328 Type 'contributors()' for more information and
#10 0.328 'citation()' on how to cite R or R packages in publications.
#10 0.328 
#10 0.328 Type 'demo()' for some demos, 'help()' for on-line help, or
#10 0.328 'help.start()' for an HTML browser interface to help.
#10 0.328 Type 'q()' to quit R.
#10 0.328 
#10 0.454 > httr::GET('https://kb.phenoscape.org')
#10 0.710 Error in curl::curl_fetch_memory(url, handle = handle) : 
#10 0.710   SSL peer certificate or SSH remote key was not OK: [kb.phenoscape.org] server certificate verification failed. CAfile: none CRLfile: none
#10 0.710 Calls: <Anonymous> ... request_fetch -> request_fetch.write_memory -> <Anonymous>
#10 0.710 Execution halted

I'm going to see if I can isolate a particular package that when installed breaks SSL.

hlapp commented 1 year ago

I would suspect the Rcurl package; perhaps it's updated as part of the installation of dependencies to a version linked against libcurl that exhibits the problem?

johnbradley commented 1 year ago

It does look to be an issue with installing curl. Simpler Dockerfile:

FROM rocker/verse:3.6.3
RUN R -e "httr::GET('https://kb.phenoscape.org')"
RUN R -e  "install.packages('curl' , repos = 'https://cloud.r-project.org')"
RUN R -e "httr::GET('https://kb.phenoscape.org')"

Output

 => CACHED [2/4] RUN R -e "httr::GET('https://kb.phenoscape.org')"                                                                0.0s
 => CACHED [3/4] RUN R -e  "install.packages('curl' , repos = 'https://cloud.r-project.org')"                                     0.0s
 => ERROR [4/4] RUN R -e "httr::GET('https://kb.phenoscape.org')"                                                                 6.6s
------                                                                                                                                 
 > [4/4] RUN R -e "httr::GET('https://kb.phenoscape.org')":                                                                            
...
#8 2.499 Error in curl::curl_fetch_memory(url, handle = handle) : 
#8 2.499   SSL peer certificate or SSH remote key was not OK: [kb.phenoscape.org] server certificate verification failed. CAfile: none CRLfile: none

hlapp commented 1 year ago

Note BTW that the line [2/4] doesn't actually get re-executed, presumably it itself and no preceding layer has changed from its cached version. So just to double-check, perhaps worth invalidating the cache then rebuild?

johnbradley commented 1 year ago

The line [2/4] was cached because I ran it multiple times. So it worked once and was cached. Here is run with cache turned off:

$ docker build --no-cache -t j .
[+] Building 74.0s (7/7) FINISHED                                                                                                      
 => [internal] load build definition from Dockerfile                                                                              0.6s
 => => transferring dockerfile: 245B                                                                                              0.2s
 => [internal] load .dockerignore                                                                                                 0.3s
 => => transferring context: 34B                                                                                                  0.1s
 => [internal] load metadata for docker.io/rocker/verse:3.6.3                                                                     4.8s
 => CACHED [1/4] FROM docker.io/rocker/verse:3.6.3@sha256:7f1d3fa2b07bb2d5435e1239ac68d2a8d48ef2bd8074225f8d611d29b6660fcf        0.0s
 => [2/4] RUN R -e "httr::GET('https://kb.phenoscape.org')"                                                                      12.0s
 => [3/4] RUN R -e  "install.packages('curl' , repos = 'https://cloud.r-project.org')"                                           49.0s 
 => ERROR [4/4] RUN R -e "httr::GET('https://kb.phenoscape.org
------                                                                                                                                 
 > [4/4] RUN R -e "httr::GET('https://kb.phenoscape.org')":                                                                            
#7 2.726                                                                                                                               
#7 2.726 R version 3.6.3 (2020-02-29) -- "Holding the Windsock"                                                                        
#7 2.726 Copyright (C) 2020 The R Foundation for Statistical Computing                                                                 
#7 2.726 Platform: x86_64-pc-linux-gnu (64-bit)                                                                                        
#7 2.726 
#7 2.726 R is free software and comes with ABSOLUTELY NO WARRANTY.
#7 2.726 You are welcome to redistribute it under certain conditions.
#7 2.726 Type 'license()' or 'licence()' for distribution details.
#7 2.726 
#7 2.726 R is a collaborative project with many contributors.
#7 2.726 Type 'contributors()' for more information and
#7 2.726 'citation()' on how to cite R or R packages in publications.
#7 2.726 
#7 2.726 Type 'demo()' for some demos, 'help()' for on-line help, or
#7 2.726 'help.start()' for an HTML browser interface to help.
#7 2.726 Type 'q()' to quit R.
#7 2.726 
#7 3.038 > httr::GET('https://kb.phenoscape.org')
#7 6.236 Error in curl::curl_fetch_memory(url, handle = handle) : 
#7 6.236   SSL peer certificate or SSH remote key was not OK: [kb.phenoscape.org] server certificate verification failed. CAfile: none CRLfile: none
#7 6.236 Calls: <Anonymous> ... request_fetch -> request_fetch.write_memory -> <Anonymous>
#7 6.240 Execution halted

hlapp commented 1 year ago

Yeah, I just found the same.

hlapp commented 1 year ago

It seems like if we add the following before the dependencies that end up upgrading the R curl package, then there's no error:

apt-get update && apt-get install -y libcurl4-openssl-dev

It least I can do that step by step with success in the rocker/verse:3.6.3 container.

phenoscape / rphenoscape

Fix failing R3 tests #278