GDC api problem - Githubissues

sisterdot commented 5 years ago

hey!

thanks for developing and maintaining TCGAbiolinks it made live so much easier... till yesterday... than i ran into a GDC accessibility issue, which i couldn't resolve based on other suggestions

i am aware that the issue is likely not on the TCGAbiolinks side per se, but would be curious to get your opinion on my problem

i am running the following code as listed in the manual

query <- GDCquery(project = "TCGA-COAD", 
                  data.category = "Clinical", 
                  file.type = "xml", 
                  barcode = c("TCGA-RU-A8FL","TCGA-AA-3972"))

it was running all fine till a few days ago

but since yesterday i am getting the following response

--------------------------------------
o GDCquery: Searching in GDC database
--------------------------------------
Genome of reference: hg38
--------------------------------------------
oo Accessing GDC. This might take a while...
--------------------------------------------
[1] "https://api.gdc.cancer.gov/files/?pretty=true&expand=cases,cases.project,center,analysis&size=531&filters=%7B%22op%22:%22and%22,%22content%22:[%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22cases.project.project_id%22,%22value%22:[%22TCGA-COAD%22]%7D%7D,%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22files.data_category%22,%22value%22:[%22Clinical%22]%7D%7D]%7D&format=JSON"
ooo Project: TCGA-COAD
Error:  Error in getURL(url, fromJSON, timeout(600), simplifyDataFrame = TRUE): 'getURL()' failed:
  URL: https://api.gdc.cancer.gov/files/?pretty=true&expand=cases,cases.project,center,analysis&size=531&filters=%7B%22op%22:%22and%22,%22content%22:[%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22cases.project.project_id%22,%22value%22:[%22TCGA-COAD%22]%7D%7D,%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22files.data_category%22,%22value%22:[%22Clinical%22]%7D%7D]%7D&format=JSON
  error: HTTP error 500.

We will retry to access GDC!
Error in if (json$data$pagination$count == 0) { : 
  argument is of length zero

no response for the following query either

clin.gbm <- GDCquery_clinic("TCGA-GBM", "clinical")

parts of my sessionInfo (worked fine till now)

R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)

locale:
[1] C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] TCGAbiolinks_2.11.2

but also tried another installation with similar success

R version 3.4.0 (2017-04-21)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 7 (wheezy)

other attached packages:
[1] TCGAbiolinks_2.6.12

response:

> query <- GDCquery(project = "TCGA-COAD", 
+                   data.category = "Clinical", 
+                   file.type = "xml", 
+                   barcode = c("TCGA-RU-A8FL","TCGA-AA-3972"))
Error in value[[3L]](cond) : 
  GDC server down, try to use this package later

i dont see a problem with accessibility, as the following works

 curl "https://api.gdc.cancer.gov/analysis/top_cases_counts_by_genes?gene_ids=ENSG00000155657&pretty=true"

so is it the query? is there a general recommendation on how to narrow down the source of the problem?

THANKS for any suggestions!!!!!

lucianamontivero commented 5 years ago

Hi! I'm also having the same problem when trying to use given TCGABiolinks example

# You can define a list of samples to query and download providing relative TCGA barcodes.
listSamples <- c("TCGA-E9-A1NG-11A-52R-A14M-07","TCGA-BH-A1FC-11A-32R-A13Q-07",
                 "TCGA-A7-A13G-11A-51R-A13Q-07","TCGA-BH-A0DK-11A-13R-A089-07",
                 "TCGA-E9-A1RH-11A-34R-A169-07","TCGA-BH-A0AU-01A-11R-A12P-07",
                 "TCGA-C8-A1HJ-01A-11R-A13Q-07","TCGA-A7-A13D-01A-13R-A12P-07",
                 "TCGA-A2-A0CV-01A-31R-A115-07","TCGA-AQ-A0Y5-01A-11R-A14M-07")

# Query platform Illumina HiSeq with a list of barcode 
query <- GDCquery(project = "TCGA-BRCA", 
                  data.category = "Gene expression",
                  data.type = "Gene expression quantification",
                  experimental.strategy = "RNA-Seq",
                  platform = "Illumina HiSeq",
                  file.type = "results",
                  barcode = listSamples, 
                  legacy = TRUE)

I'm also getting the same error error as you.

Sometimes this problem was solved installing TCGABiolinks directly from GitHub, but that's not solving it for me either.

devtools::install_github("BioinformaticsFMRP/TCGAbiolinks")

I don't know if it's because something is installed wrong or if GDC server is actually down. My session info is

R version 3.5.2 (2018-12-20)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Mojave 10.14.2

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.5/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] parallel  stats4    stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] BiocInstaller_1.32.1        SummarizedExperiment_1.12.0 DelayedArray_0.8.0         
 [4] BiocParallel_1.16.5         matrixStats_0.54.0          Biobase_2.42.0             
 [7] GenomicRanges_1.34.0        GenomeInfoDb_1.18.1         IRanges_2.16.0             
[10] S4Vectors_0.20.1            BiocGenerics_0.28.0         DT_0.5                     
[13] dplyr_0.7.8                 TCGAbiolinks_2.11.2        

loaded via a namespace (and not attached):
  [1] backports_1.1.3             circlize_0.4.5              aroma.light_3.12.0         
  [4] plyr_1.8.4                  selectr_0.4-1               ConsensusClusterPlus_1.46.0
  [7] lazyeval_0.2.1              splines_3.5.2               usethis_1.4.0              
 [10] ggplot2_3.1.0               sva_3.30.1                  digest_0.6.18              
 [13] foreach_1.4.4               htmltools_0.3.6             magrittr_1.5               
 [16] memoise_1.1.0               cluster_2.0.7-1             doParallel_1.0.14          
 [19] remotes_2.0.2               limma_3.38.3                ComplexHeatmap_1.20.0      
 [22] Biostrings_2.50.2           readr_1.3.1                 annotate_1.60.0            
 [25] R.utils_2.7.0               prettyunits_1.0.2           colorspace_1.4-0           
 [28] blob_1.1.1                  rvest_0.3.2                 ggrepel_0.8.0              
 [31] xfun_0.4                    callr_3.1.1                 crayon_1.3.4               
 [34] RCurl_1.95-4.11             jsonlite_1.6                genefilter_1.64.0          
 [37] bindr_0.1.1                 survival_2.43-3             zoo_1.8-4                  
 [40] iterators_1.0.10            glue_1.3.0                  survminer_0.4.3            
 [43] gtable_0.2.0                zlibbioc_1.28.0             XVector_0.22.0             
 [46] GetoptLong_0.1.7            pkgbuild_1.0.2              shape_1.4.4                
 [49] scales_1.0.0                DESeq_1.34.1                DBI_1.0.0                  
 [52] edgeR_3.24.3                ggthemes_4.0.1              Rcpp_1.0.0                 
 [55] xtable_1.8-3                progress_1.2.0              cmprsk_2.2-7               
 [58] bit_1.1-14                  matlab_1.0.2                km.ci_0.5-2                
 [61] htmlwidgets_1.3             httr_1.4.0                  RColorBrewer_1.1-2         
 [64] pkgconfig_2.0.2             XML_3.98-1.16               R.methodsS3_1.7.1          
 [67] locfit_1.5-9.1              tidyselect_0.2.5            rlang_0.3.1                
 [70] AnnotationDbi_1.44.0        munsell_0.5.0               tools_3.5.2                
 [73] downloader_0.4              cli_1.0.1                   generics_0.0.2             
 [76] RSQLite_2.1.1               devtools_2.0.1              broom_0.5.1                
 [79] stringr_1.3.1               yaml_2.2.0                  fs_1.2.6                   
 [82] processx_3.2.1              knitr_1.21                  bit64_0.9-7                
 [85] survMisc_0.5.5              purrr_0.2.5                 bindrcpp_0.2.2             
 [88] EDASeq_2.16.3               nlme_3.1-137                R.oo_1.22.0                
 [91] xml2_1.2.0                  biomaRt_2.39.2              compiler_3.5.2             
 [94] rstudioapi_0.9.0            curl_3.3                    tibble_2.0.1               
 [97] geneplotter_1.60.0          stringi_1.2.4               ps_1.3.0                   
[100] desc_1.2.0                  GenomicFeatures_1.34.1      lattice_0.20-38            
[103] Matrix_1.2-15               KMsurv_0.1-5                pillar_1.3.1               
[106] BiocManager_1.30.4          GlobalOptions_0.1.0         data.table_1.12.0          
[109] bitops_1.0-6                rtracklayer_1.42.1          R6_2.3.0                   
[112] latticeExtra_0.6-28         hwriter_1.3.2               ShortRead_1.40.0           
[115] gridExtra_2.3               sessioninfo_1.1.1           codetools_0.2-16           
[118] pkgload_1.0.2               assertthat_0.2.0            rprojroot_1.3-2            
[121] rjson_0.2.20                withr_2.1.2                 GenomicAlignments_1.18.1   
[124] Rsamtools_1.34.0            GenomeInfoDbData_1.2.0      mgcv_1.8-26                
[127] hms_0.4.2                   grid_3.5.2                  tidyr_0.8.2                
[130] ggpubr_0.2

tiagochst commented 5 years ago

GDC might have a problem with the legacy server, any query with more than 150 samples is giving "internal error". I'll send an email to GDC.

atakanekiz commented 5 years ago

I'm having a similar issue upon executing the following code. Any help is appreciated!

query_harmonized <- GDCquery(project = "TCGA-SKCM",
                             data.category = "Transcriptome Profiling",
                             data.type = "miRNA Expression Quantification",
                             legacy = FALSE)

Output:

--------------------------------------
o GDCquery: Searching in GDC database
--------------------------------------
Genome of reference: hg38
--------------------------------------------
oo Accessing GDC. This might take a while...
--------------------------------------------
ooo Project: TCGA-SKCM
Error:  Error in getURL(url, fromJSON, timeout(600), simplifyDataFrame = TRUE): 'getURL()' failed:
  URL: https://api.gdc.cancer.gov/files/?pretty=true&expand=cases.samples.portions.analytes.aliquots,cases.project,center,analysis,cases.samples&size=2320&filters=%7B%22op%22:%22and%22,%22content%22:[%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22cases.project.project_id%22,%22value%22:[%22TCGA-SKCM%22]%7D%7D,%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22files.data_category%22,%22value%22:[%22Transcriptome%20Profiling%22]%7D%7D,%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22files.data_type%22,%22value%22:[%22miRNA%20Expression%20Quantification%22]%7D%7D]%7D&format=JSON
  error: HTTP error 500.

We will retry to access GDC!
Error in if (json$data$pagination$count == 0) { : 
  argument is of length zero

My session info

R version 3.5.2 (2018-12-20)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] scales_1.0.0        openxlsx_4.1.0      survminer_0.4.3     ggpubr_0.2          magrittr_1.5        ggplot2_3.1.0       survival_2.43-3    
 [8] dplyr_0.7.8         tibble_2.0.1        TCGAbiolinks_2.11.2

loaded via a namespace (and not attached):
  [1] backports_1.1.3             circlize_0.4.5              aroma.light_3.10.0          plyr_1.8.4                  selectr_0.4-1              
  [6] ConsensusClusterPlus_1.45.0 lazyeval_0.2.1              splines_3.5.2               BiocParallel_1.15.7         usethis_1.4.0              
 [11] GenomeInfoDb_1.17.2         sva_3.29.1                  digest_0.6.18               foreach_1.4.4               memoise_1.1.0              
 [16] cluster_2.0.7-1             doParallel_1.0.14           limma_3.37.7                remotes_2.0.2               ComplexHeatmap_1.19.2      
 [21] Biostrings_2.48.0           readr_1.3.1                 annotate_1.58.0             matrixStats_0.54.0          R.utils_2.7.0              
 [26] prettyunits_1.0.2           colorspace_1.4-0            blob_1.1.1                  rvest_0.3.2                 ggrepel_0.8.0              
 [31] xfun_0.4                    callr_3.1.1                 crayon_1.3.4                RCurl_1.95-4.11             jsonlite_1.6               
 [36] genefilter_1.63.2           bindr_0.1.1                 zoo_1.8-4                   iterators_1.0.10            glue_1.3.0                 
 [41] gtable_0.2.0                zlibbioc_1.26.0             XVector_0.20.0              GetoptLong_0.1.7            DelayedArray_0.6.6         
 [46] pkgbuild_1.0.2              shape_1.4.4                 BiocGenerics_0.27.1         DESeq_1.32.0                DBI_1.0.0                  
 [51] edgeR_3.23.5                ggthemes_4.0.1              Rcpp_1.0.0                  xtable_1.8-3                progress_1.2.0             
 [56] cmprsk_2.2-7                bit_1.1-14                  matlab_1.0.2                km.ci_0.5-2                 stats4_3.5.2               
 [61] httr_1.4.0                  RColorBrewer_1.1-2          pkgconfig_2.0.2             XML_3.98-1.16               R.methodsS3_1.7.1          
 [66] locfit_1.5-9.1              tidyselect_0.2.5            rlang_0.3.1                 AnnotationDbi_1.42.1        munsell_0.5.0              
 [71] tools_3.5.2                 downloader_0.4              cli_1.0.1                   generics_0.0.2              RSQLite_2.1.1              
 [76] devtools_2.0.1              broom_0.5.1                 stringr_1.3.1               yaml_2.2.0                  processx_3.2.1             
 [81] knitr_1.21                  bit64_0.9-7                 fs_1.2.6                    zip_1.0.0                   survMisc_0.5.5             
 [86] purrr_0.2.5                 bindrcpp_0.2.2              EDASeq_2.15.4               nlme_3.1-137                R.oo_1.22.0                
 [91] xml2_1.2.0                  biomaRt_2.37.6              compiler_3.5.2              rstudioapi_0.9.0            curl_3.2                   
 [96] testthat_2.0.1              geneplotter_1.58.0          stringi_1.2.4               highr_0.7                   ps_1.3.0                   
[101] GenomicFeatures_1.33.2      desc_1.2.0                  lattice_0.20-38             Matrix_1.2-15               KMsurv_0.1-5               
[106] pillar_1.3.1                GlobalOptions_0.1.0         data.table_1.12.0           bitops_1.0-6                rtracklayer_1.40.6         
[111] GenomicRanges_1.33.14       R6_2.3.0                    latticeExtra_0.6-28         hwriter_1.3.2               ShortRead_1.38.0           
[116] gridExtra_2.3               IRanges_2.15.18             sessioninfo_1.1.1           codetools_0.2-16            assertthat_0.2.0           
[121] pkgload_1.0.2               SummarizedExperiment_1.11.6 rprojroot_1.3-2             rjson_0.2.20                withr_2.1.2                
[126] GenomicAlignments_1.16.0    Rsamtools_1.32.3            S4Vectors_0.19.20           GenomeInfoDbData_1.1.0      mgcv_1.8-26                
[131] parallel_3.5.2              hms_0.4.2                   grid_3.5.2                  tidyr_0.8.2                 Biobase_2.41.2

When I try to access the URL in a browser this is what I get:

{
  "message": "internal server error"
}

yinlisssss commented 5 years ago

same issue

queryClin <- GDCquery(project = "TCGA-LUAD",
                    data.category = "Clinical",
                       file.type = "xml")

GDCquery: Searching in GDC database
Genome of reference: hg38

oo Accessing GDC. This might take a while...
--------------------------------------------
ooo Project: TCGA-LUAD
Error:  Error in getURL(url, fromJSON, timeout(600), simplifyDataFrame = TRUE): 'getURL()' failed:
  URL: https://api.gdc.cancer.gov/files/?pretty=true&expand=cases,cases.project,center,analysis&size=623&filters=%7B%22op%22:%22and%22,%22content%22:[%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22cases.project.project_id%22,%22value%22:[%22TCGA-LUAD%22]%7D%7D,%7B%22op%22:%22in%22,%22content%22:%7B%22field%22:%22files.data_category%22,%22value%22:[%22Clinical%22]%7D%7D]%7D&format=JSON
  error: HTTP error 500.

We will retry to access GDC!
Error in if (json$data$pagination$count == 0) { :

rjg2186 commented 5 years ago

@tiagochst

The issue with still persists. Can you please let me know if this is still from GDC. Thanks

cnv.query <- GDCquery(project = "TCGA-COAD", data.category = "Copy number variation")
Error in value[[3L]](cond) :
  GDC server down, try to use this package later

tiagochst commented 5 years ago

@rjg2186 It seems the package cannot access GDC server in your machine. There could be several reasons not related to our package. Please, could you post your sessionInfo() so we can have more information ?

Also, for Copy Number data you need to specify the data.type.

library(TCGAbiolinks)

# 1) Get GISTIC 
query <- GDCquery(project = "TCGA-COAD",
                  data.category = "Copy Number Variation",
                  data.type = "Gene Level Copy Number Scores",              
                  access="open")
GDCdownload(query)
gistic <- GDCprepare(query)

# 2) Get Copy Number Segment
query <- GDCquery(project = "TCGA-COAD", 
                  data.category = "Copy Number Variation",
                  data.type = "Copy Number Segment")
GDCdownload(query)
cns <- GDCprepare(query)

query <- GDCquery(project = "TCGA-COAD", 
                  data.category = "Copy Number Variation",
                  data.type = "Masked Copy Number Segment")
GDCdownload(query)
cns.masked <- GDCprepare(query)

Dokmen commented 5 years ago

I have same issue and I have Windows OS. I removed bioconducter package and I tried to install devtools github. but I have an error " Error in i.p(...) : (converted from warning) installation of package ‘C:/~/AppData/Local/Temp/RtmpgDZJjm/file4bd05fa2264a/TCGAbiolinks_2.11.6.tar.gz’ had non-zero exit status.

I could not solve this problem.

`> sessionInfo() R version 3.5.3 (2019-03-11) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 17134)

Matrix products: default

locale: [1] LC_COLLATE=Turkish_Turkey.1254 LC_CTYPE=Turkish_Turkey.1254 LC_MONETARY=Turkish_Turkey.1254 [4] LC_NUMERIC=C LC_TIME=Turkish_Turkey.1254

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] TCGAbiolinks_2.10.5

loaded via a [1] backports_1.1.4 [4] aroma.light_3.12.0 [7] ConsensusClusterPlus_1.46.0 [10] BiocParallel_1.16.6 [13] ggplot2_3.1.1 [16] foreach_1.4.4 [19] memoise_1.1.0 [22] remotes_2.0.4 [25] Biostrings_2.50.2 [28] matrixStats_0.54.0 [31] prettyunits_1.0.2 [34] rvest_0.3.3 [37] dplyr_0.8.0.1 [40] RCurl_1.95-4.12 [43] zoo_1.8-5 [46] glue_1.3.1 [49] sesame_1.0.0 [52] GetoptLong_0.1.7 [55] wheatmap_0.1.0 [58] scales_1.0.0 [61] edgeR_3.24.3 [64] xtable_1.8-4 [67] bit_1.1-14 [70] km.ci_0.5-2 [73] RColorBrewer_1.1-2 [76] R.methodsS3_1.7.1 [79] tidyselect_0.2.5 [82] AnnotationDbi_1.44.0 [85] cli_1.1.0 [88] RSQLite_2.1.1 [91] broom_0.5.2 [94] fs_1.2.7 [97] bit64_0.9-8 [100] randomForest_4.6-14 [103] mime_0.6 [106] biomaRt_2.38.0 [109] curl_3.3 [112] geneplotter_1.60.0 [115] desc_1.2.0 [118] Matrix_1.2-15 [121] BiocManager_1.30.4 [124] bitops_1.0-6 [127] GenomicRanges_1.34.0 [130] hwriter_1.3.2 [133] gridExtra_2.3 [136] codetools_0.2-16 [139] SummarizedExperiment_1.12.0 [142] withr_2.1.2 [145] S4Vectors_0.20.1 [148] parallel_3.5.3 [151] tidyr_0.8.3 [154] shiny_1.3.2 ` namespace (and not attached): circlize_0.4.6 AnnotationHub_2.14.5
plyr_1.8.4 selectr_0.4-1
lazyeval_0.2.2 splines_3.5.3
usethis_1.5.0 GenomeInfoDb_1.18.2
sva_3.30.1 digest_0.6.18
htmltools_0.3.6 magrittr_1.5
cluster_2.0.7-1 doParallel_1.0.14
limma_3.38.3 ComplexHeatmap_1.20.0
readr_1.3.1 annotate_1.60.1
sesameData_1.0.0 R.utils_2.8.0
colorspace_1.4-1 blob_1.1.1
ggrepel_0.8.0 xfun_0.6
callr_3.2.0 crayon_1.3.4
jsonlite_1.6 genefilter_1.64.0
survival_2.43-3 iterators_1.0.10
survminer_0.4.3 gtable_0.3.0
zlibbioc_1.28.0 XVector_0.22.0
DelayedArray_0.8.0 pkgbuild_1.0.3
shape_1.4.4 BiocGenerics_0.28.0
DESeq_1.34.1 DBI_1.0.0
ggthemes_4.1.1 Rcpp_1.0.1
progress_1.2.0 cmprsk_2.2-7
matlab_1.0.2 preprocessCore_1.44.0
stats4_3.5.3 httr_1.4.0
pkgconfig_2.0.2 XML_3.98-1.19
locfit_1.5-9.1 DNAcopy_1.56.0
rlang_0.3.4 later_0.8.0
munsell_0.5.0 tools_3.5.3
downloader_0.4 generics_0.0.2
ExperimentHub_1.8.0 devtools_2.0.2
stringr_1.4.0 yaml_2.2.0
processx_3.3.0 knitr_1.22
survMisc_0.5.5 purrr_0.3.2
EDASeq_2.16.3 nlme_3.1-137
R.oo_1.22.0 xml2_1.2.0
compiler_3.5.3 rstudioapi_0.10
interactiveDisplayBase_1.20.0 tibble_2.1.1
stringi_1.4.3 ps_1.3.0
GenomicFeatures_1.34.8 lattice_0.20-38
KMsurv_0.1-5 pillar_1.3.1
GlobalOptions_0.1.0 data.table_1.12.2
httpuv_1.5.1 rtracklayer_1.42.2
R6_2.4.0 latticeExtra_0.6-28
promises_1.0.1 ShortRead_1.40.0
IRanges_2.16.0 sessioninfo_1.1.1
pkgload_1.0.2 assertthat_0.2.1
rprojroot_1.3-2 rjson_0.2.20
GenomicAlignments_1.18.1 Rsamtools_1.34.1
GenomeInfoDbData_1.2.0 mgcv_1.8-27
hms_0.4.2 grid_3.5.3
ggpubr_0.2 Biobase_2.42.0

simonjuleseric2 commented 4 months ago

Any news regarding this issue?

BioinformaticsFMRP / TCGAbiolinks

GDC api problem #289