search

ropensci / biomartr

Genomic Data Retrieval with R
https://docs.ropensci.org/biomartr
217 stars 29 forks source link

ftp issues when using meta.retrieval #27

Closed seannamctaggart closed 6 years ago

seannamctaggart commented 6 years ago

Hi,

I am having a similar issue to some of your other users, in that I am getting the following error message:

Something went wrong with the connection to: ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/001/477/545/GCF_001477545.1_Pneu_cari_B80_V3/

when trying to download some genomes. However, unlike the other users, I am getting this message on the first genome that I am trying to download. I am attempting to download all the fungal genomes with the following command:

biomartr::meta.retrieval( db = "refseq", kingdom = "fungi",type = "genome", path="/home/seannamctaggart/data/ncbi_genomes/biomartr/fungi")

Do you have any suggestions?

Many thanks!

Seanna

HajkD commented 6 years ago

Hi @seannamctaggart ,

Thank you for contacting me.

Which biomartr version are you using?

Have you installed the current developer version?

source("http://bioconductor.org/biocLite.R")
biocLite("ropensci/biomartr")

On which system do you run the command? Linux? Mac? Windows?

Do you have a firewall that might prevent downloads conducted with the curl or wget command?

I just ran your command and it worked seamlessly.

If you send me a private email then I can send you the files I downloaded with biomartr via Google Drive.

Many thanks and best wishes, Hajk

HajkD commented 6 years ago

P.S. This is the output list that was generated by my run today:

  [1] "_ncbi_genomes/biomartr/fungi/Pneumocystis_carinii_B80_genomic_refseq.fna.gz"                               
  [2] "_ncbi_genomes/biomartr/fungi/Phycomyces_blakesleeanus_NRRL_1555-_genomic_refseq.fna.gz"                    
  [3] "_ncbi_genomes/biomartr/fungi/Schizosaccharomyces_pombe_genomic_refseq.fna.gz"                              
  [4] "_ncbi_genomes/biomartr/fungi/Schizosaccharomyces_japonicus_yFS275_genomic_refseq.fna.gz"                   
  [5] "_ncbi_genomes/biomartr/fungi/Schizosaccharomyces_octosporus_yFS286_genomic_refseq.fna.gz"                  
  [6] "_ncbi_genomes/biomartr/fungi/Cyberlindnera_jadinii_NRRL_Y-1542_genomic_refseq.fna.gz"                      
  [7] "_ncbi_genomes/biomartr/fungi/Pichia_kudriavzevii_genomic_refseq.fna.gz"                                    
  [8] "_ncbi_genomes/biomartr/fungi/Kluyveromyces_marxianus_DMKU3-1042_genomic_refseq.fna.gz"                     
  [9] "_ncbi_genomes/biomartr/fungi/Scheffersomyces_stipitis_CBS_6054_genomic_refseq.fna.gz"                      
 [10] "_ncbi_genomes/biomartr/fungi/Pichia_membranifaciens_NRRL_Y-2026_genomic_refseq.fna.gz"                     
 [11] "_ncbi_genomes/biomartr/fungi/Wickerhamomyces_anomalus_NRRL_Y-366-8_genomic_refseq.fna.gz"                  
 [12] "_ncbi_genomes/biomartr/fungi/Meyerozyma_guilliermondii_ATCC_6260_genomic_refseq.fna.gz"                    
 [13] "_ncbi_genomes/biomartr/fungi/Saccharomyces_cerevisiae_S288C_genomic_refseq.fna.gz"                         
 [14] "_ncbi_genomes/biomartr/fungi/Torulaspora_delbrueckii_genomic_refseq.fna.gz"                                
 [15] "_ncbi_genomes/biomartr/fungi/Yarrowia_lipolytica_CLIB122_genomic_refseq.fna.gz"                            
 [16] "_ncbi_genomes/biomartr/fungi/Zygosaccharomyces_rouxii_genomic_refseq.fna.gz"                               
 [17] "_ncbi_genomes/biomartr/fungi/Debaryomyces_hansenii_CBS767_genomic_refseq.fna.gz"                           
 [18] "_ncbi_genomes/biomartr/fungi/Kockovaella_imperatae_genomic_refseq.fna.gz"                                  
 [19] "_ncbi_genomes/biomartr/fungi/Bipolaris_maydis_ATCC_48331_genomic_refseq.fna.gz"                            
 [20] "_ncbi_genomes/biomartr/fungi/Bipolaris_zeicola_26-R-13_genomic_refseq.fna.gz"                              
 [21] "_ncbi_genomes/biomartr/fungi/Leptosphaeria_maculans_JN3_genomic_refseq.fna.gz"                             
 [22] "_ncbi_genomes/biomartr/fungi/Histoplasma_capsulatum_NAm1_genomic_refseq.fna.gz"                            
 [23] "_ncbi_genomes/biomartr/fungi/Aspergillus_aculeatus_ATCC_16872_genomic_refseq.fna.gz"                       
 [24] "_ncbi_genomes/biomartr/fungi/Aspergillus_clavatus_NRRL_1_genomic_refseq.fna.gz"                            
 [25] "_ncbi_genomes/biomartr/fungi/Aspergillus_flavus_NRRL3357_genomic_refseq.fna.gz"                            
 [26] "_ncbi_genomes/biomartr/fungi/Aspergillus_niger_CBS_513.88_genomic_refseq.fna.gz"                           
 [27] "_ncbi_genomes/biomartr/fungi/Aspergillus_oryzae_RIB40_genomic_refseq.fna.gz"                               
 [28] "_ncbi_genomes/biomartr/fungi/Amorphotheca_resinae_ATCC_22711_genomic_refseq.fna.gz"                        
 [29] "_ncbi_genomes/biomartr/fungi/Fusarium_fujikuroi_IMI_58289_genomic_refseq.fna.gz"                           
 [30] "_ncbi_genomes/biomartr/fungi/Neurospora_crassa_OR74A_genomic_refseq.fna.gz"                                
 [31] "_ncbi_genomes/biomartr/fungi/Podospora_anserina_S_mat+_genomic_refseq.fna.gz"                              
 [32] "_ncbi_genomes/biomartr/fungi/Sordaria_macrospora_k-hell_genomic_refseq.fna.gz"                             
 [33] "_ncbi_genomes/biomartr/fungi/Sclerotinia_sclerotiorum_1980_UF-70_genomic_refseq.fna.gz"                    
 [34] "_ncbi_genomes/biomartr/fungi/Cryptococcus_neoformans_var._neoformans_JEC21_genomic_refseq.fna.gz"          
 [35] "_ncbi_genomes/biomartr/fungi/Tremella_mesenterica_DSM_1558_genomic_refseq.fna.gz"                          
 [36] "_ncbi_genomes/biomartr/fungi/Ustilago_maydis_521_genomic_refseq.fna.gz"                                    
 [37] "_ncbi_genomes/biomartr/fungi/Tilletiaria_anomala_UBC_951_genomic_refseq.fna.gz"                            
 [38] "_ncbi_genomes/biomartr/fungi/Rhodotorula_toruloides_NP11_genomic_refseq.fna.gz"                            
 [39] "_ncbi_genomes/biomartr/fungi/Puccinia_graminis_f._sp._tritici_CRL_75-36-700-3_genomic_refseq.fna.gz"       
 [40] "_ncbi_genomes/biomartr/fungi/Trametes_versicolor_FP-101664_SS1_genomic_refseq.fna.gz"                      
 [41] "_ncbi_genomes/biomartr/fungi/Schizophyllum_commune_H4-8_genomic_refseq.fna.gz"                             
 [42] "_ncbi_genomes/biomartr/fungi/Agaricus_bisporus_var._bisporus_H97_genomic_refseq.fna.gz"                    
 [43] "_ncbi_genomes/biomartr/fungi/Candida_albicans_SC5314_genomic_refseq.fna.gz"                                
 [44] "_ncbi_genomes/biomartr/fungi/[Candida]_glabrata_genomic_refseq.fna.gz"                                     
 [45] "_ncbi_genomes/biomartr/fungi/Candida_tropicalis_MYA-3404_genomic_refseq.fna.gz"                            
 [46] "_ncbi_genomes/biomartr/fungi/Coccidioides_immitis_RS_genomic_refseq.fna.gz"                                
 [47] "_ncbi_genomes/biomartr/fungi/Fusarium_oxysporum_f._sp._lycopersici_4287_genomic_refseq.fna.gz"             
 [48] "_ncbi_genomes/biomartr/fungi/Fusarium_graminearum_PH-1_genomic_refseq.fna.gz"                              
 [49] "_ncbi_genomes/biomartr/fungi/Trichoderma_harzianum_CBS_226.95_genomic_refseq.fna.gz"                       
 [50] "_ncbi_genomes/biomartr/fungi/Trichophyton_rubrum_CBS_118892_genomic_refseq.fna.gz"                         
 [51] "_ncbi_genomes/biomartr/fungi/Alternaria_alternata_genomic_refseq.fna.gz"                                   
 [52] "_ncbi_genomes/biomartr/fungi/Saitoella_complicata_NRRL_Y-17804_genomic_refseq.fna.gz"                      
 [53] "_ncbi_genomes/biomartr/fungi/Tsuchiyaea_wingfieldii_CBS_7118_genomic_refseq.fna.gz"                        
 [54] "_ncbi_genomes/biomartr/fungi/Exophiala_dermatitidis_NIH_UT8656_genomic_refseq.fna.gz"                      
 [55] "_ncbi_genomes/biomartr/fungi/Encephalitozoon_cuniculi_GB-M1_genomic_refseq.fna.gz"                         
 [56] "_ncbi_genomes/biomartr/fungi/Arthrobotrys_oligospora_ATCC_24927_genomic_refseq.fna.gz"                     
 [57] "_ncbi_genomes/biomartr/fungi/Parastagonospora_nodorum_SN15_genomic_refseq.fna.gz"                          
 [58] "_ncbi_genomes/biomartr/fungi/Naumovozyma_castellii_CBS_4309_genomic_refseq.fna.gz"                         
 [59] "_ncbi_genomes/biomartr/fungi/Naumovozyma_dairenensis_CBS_421_genomic_refseq.fna.gz"                        
 [60] "_ncbi_genomes/biomartr/fungi/Metschnikowia_bicuspidata_var._bicuspidata_NRRL_YB-4993_genomic_refseq.fna.gz"
 [61] "_ncbi_genomes/biomartr/fungi/Penicillium_expansum_genomic_refseq.fna.gz"                                   
 [62] "_ncbi_genomes/biomartr/fungi/Verticillium_dahliae_VdLs.17_genomic_refseq.fna.gz"                           
 [63] "_ncbi_genomes/biomartr/fungi/Encephalitozoon_hellem_ATCC_50504_genomic_refseq.fna.gz"                      
 [64] "_ncbi_genomes/biomartr/fungi/Talaromyces_stipitatus_ATCC_10500_genomic_refseq.fna.gz"                      
 [65] "_ncbi_genomes/biomartr/fungi/Kluyveromyces_lactis_genomic_refseq.fna.gz"                                   
 [66] "_ncbi_genomes/biomartr/fungi/Trichoderma_virens_Gv29-8_genomic_refseq.fna.gz"                              
 [67] "_ncbi_genomes/biomartr/fungi/Laccaria_bicolor_S238N-H82_genomic_refseq.fna.gz"                             
 [68] "_ncbi_genomes/biomartr/fungi/Rhodotorula_graminis_WP1_genomic_refseq.fna.gz"                               
 [69] "_ncbi_genomes/biomartr/fungi/Sporothrix_schenckii_1099-18_genomic_refseq.fna.gz"                           
 [70] "_ncbi_genomes/biomartr/fungi/Colletotrichum_graminicola_M1.001_genomic_refseq.fna.gz"                      
 [71] "_ncbi_genomes/biomartr/fungi/Eremothecium_gossypii_ATCC_10895_genomic_refseq.fna.gz"                       
 [72] "_ncbi_genomes/biomartr/fungi/Aspergillus_terreus_NIH2624_genomic_refseq.fna.gz"                            
 [73] "_ncbi_genomes/biomartr/fungi/Uncinocarpus_reesii_1704_genomic_refseq.fna.gz"                               
 [74] "_ncbi_genomes/biomartr/fungi/Purpureocillium_lilacinum_genomic_refseq.fna.gz"                              
 [75] "_ncbi_genomes/biomartr/fungi/Mixia_osmundae_IAM_14324_genomic_refseq.fna.gz"                               
 [76] "_ncbi_genomes/biomartr/fungi/Thielavia_terrestris_NRRL_8126_genomic_refseq.fna.gz"                         
 [77] "_ncbi_genomes/biomartr/fungi/Vanderwaltozyma_polyspora_DSM_70294_genomic_refseq.fna.gz"                    
 [78] "_ncbi_genomes/biomartr/fungi/Aspergillus_fischeri_NRRL_181_genomic_refseq.fna.gz"                          
 [79] "_ncbi_genomes/biomartr/fungi/Penicillium_digitatum_Pd1_genomic_refseq.fna.gz"                              
 [80] "_ncbi_genomes/biomartr/fungi/Gaeumannomyces_tritici_R3-111a-1_genomic_refseq.fna.gz"                       
 [81] "_ncbi_genomes/biomartr/fungi/Clavispora_lusitaniae_ATCC_42720_genomic_refseq.fna.gz"                       
 [82] "_ncbi_genomes/biomartr/fungi/Lodderomyces_elongisporus_NRRL_YB-4239_genomic_refseq.fna.gz"                 
 [83] "_ncbi_genomes/biomartr/fungi/Talaromyces_marneffei_ATCC_18224_genomic_refseq.fna.gz"                       
 [84] "_ncbi_genomes/biomartr/fungi/Cryptococcus_gattii_WM276_genomic_refseq.fna.gz"                              
 [85] "_ncbi_genomes/biomartr/fungi/Chaetomium_globosum_CBS_148.51_genomic_refseq.fna.gz"                         
 [86] "_ncbi_genomes/biomartr/fungi/Tuber_melanosporum_genomic_refseq.fna.gz"                                     
 [87] "_ncbi_genomes/biomartr/fungi/Bipolaris_victoriae_FI3_genomic_refseq.fna.gz"                                
 [88] "_ncbi_genomes/biomartr/fungi/Neurospora_tetrasperma_FGSC_2508_genomic_refseq.fna.gz"                       
 [89] "_ncbi_genomes/biomartr/fungi/Nosema_ceranae_genomic_refseq.fna.gz"                                         
 [90] "_ncbi_genomes/biomartr/fungi/Fonsecaea_pedrosoi_CBS_271.37_genomic_refseq.fna.gz"                          
 [91] "_ncbi_genomes/biomartr/fungi/Stereum_hirsutum_FP-91666_SS1_genomic_refseq.fna.gz"                          
 [92] "_ncbi_genomes/biomartr/fungi/Botrytis_cinerea_B05.10_genomic_refseq.fna.gz"                                
 [93] "_ncbi_genomes/biomartr/fungi/Aspergillus_nomius_NRRL_13137_genomic_refseq.fna.gz"                          
 [94] "_ncbi_genomes/biomartr/fungi/Penicilliopsis_zonata_CBS_506.65_genomic_refseq.fna.gz"                       
 [95] "_ncbi_genomes/biomartr/fungi/Aspergillus_candidus_genomic_refseq.fna.gz"                                   
 [96] "_ncbi_genomes/biomartr/fungi/Aspergillus_campestris_IBT_28561_genomic_refseq.fna.gz"                       
 [97] "_ncbi_genomes/biomartr/fungi/Aspergillus_glaucus_CBS_516.65_genomic_refseq.fna.gz"                         
 [98] "_ncbi_genomes/biomartr/fungi/Pneumocystis_jirovecii_RU7_genomic_refseq.fna.gz"                             
 [99] "_ncbi_genomes/biomartr/fungi/Candida_dubliniensis_CD36_genomic_refseq.fna.gz"                              
[100] "_ncbi_genomes/biomartr/fungi/Vittaforma_corneae_ATCC_50505_genomic_refseq.fna.gz"                          
[101] "_ncbi_genomes/biomartr/fungi/Capronia_epimyces_CBS_606.96_genomic_refseq.fna.gz"                           
[102] "_ncbi_genomes/biomartr/fungi/Capronia_coronata_CBS_617.96_genomic_refseq.fna.gz"                           
[103] "_ncbi_genomes/biomartr/fungi/Bipolaris_sorokiniana_ND90Pr_genomic_refseq.fna.gz"                           
[104] "_ncbi_genomes/biomartr/fungi/Pyrenophora_tritici-repentis_Pt-1C-BFP_genomic_refseq.fna.gz"                 
[105] "_ncbi_genomes/biomartr/fungi/Wickerhamiella_sorbophila_genomic_refseq.fna.gz"                              
[106] "_ncbi_genomes/biomartr/fungi/Babjeviella_inositovora_NRRL_Y-12698_genomic_refseq.fna.gz"                   
[107] "_ncbi_genomes/biomartr/fungi/Suhomyces_tanzawaensis_NRRL_Y-17324_genomic_refseq.fna.gz"                    
[108] "_ncbi_genomes/biomartr/fungi/Trichoderma_reesei_QM6a_genomic_refseq.fna.gz"                                
[109] "_ncbi_genomes/biomartr/fungi/Ascoidea_rubescens_DSM_1968_genomic_refseq.fna.gz"                            
[110] "_ncbi_genomes/biomartr/fungi/Rhizopus_microsporus_ATCC_52813_genomic_refseq.fna.gz"                        
[111] "_ncbi_genomes/biomartr/fungi/Debaryomyces_fabryi_genomic_refseq.fna.gz"                                    
[112] "_ncbi_genomes/biomartr/fungi/Encephalitozoon_intestinalis_ATCC_50506_genomic_refseq.fna.gz"                
[113] "_ncbi_genomes/biomartr/fungi/Trichoderma_citrinoviride_genomic_refseq.fna.gz"                              
[114] "_ncbi_genomes/biomartr/fungi/Coniosporium_apollinis_CBS_100218_genomic_refseq.fna.gz"                      
[115] "_ncbi_genomes/biomartr/fungi/Trichophyton_benhamiae_CBS_112371_genomic_refseq.fna.gz"                      
[116] "_ncbi_genomes/biomartr/fungi/Nannizzia_gypsea_CBS_118893_genomic_refseq.fna.gz"                            
[117] "_ncbi_genomes/biomartr/fungi/Microsporum_canis_CBS_113480_genomic_refseq.fna.gz"                           
[118] "_ncbi_genomes/biomartr/fungi/Trichophyton_verrucosum_HKI_0517_genomic_refseq.fna.gz"                       
[119] "_ncbi_genomes/biomartr/fungi/Trichoderma_atroviride_IMI_206040_genomic_refseq.fna.gz"                      
[120] "_ncbi_genomes/biomartr/fungi/Lobosporangium_transversale_genomic_refseq.fna.gz"                            
[121] "_ncbi_genomes/biomartr/fungi/Rasamsonia_emersonii_CBS_393.64_genomic_refseq.fna.gz"                        
[122] "_ncbi_genomes/biomartr/fungi/Cordyceps_militaris_CM01_genomic_refseq.fna.gz"                               
[123] "_ncbi_genomes/biomartr/fungi/Malassezia_globosa_CBS_7966_genomic_refseq.fna.gz"                            
[124] "_ncbi_genomes/biomartr/fungi/Malassezia_sympodialis_ATCC_42132_genomic_refseq.fna.gz"                      
[125] "_ncbi_genomes/biomartr/fungi/Malassezia_pachydermatis_genomic_refseq.fna.gz"                               
[126] "_ncbi_genomes/biomartr/fungi/Thermothelomyces_thermophila_ATCC_42464_genomic_refseq.fna.gz"                
[127] "_ncbi_genomes/biomartr/fungi/Coniophora_puteana_RWD-64-598_SS2_genomic_refseq.fna.gz"                      
[128] "_ncbi_genomes/biomartr/fungi/Colletotrichum_higginsianum_IMI_349063_genomic_refseq.fna.gz"                 
[129] "_ncbi_genomes/biomartr/fungi/Trichosporon_asahii_var._asahii_CBS_2479_genomic_refseq.fna.gz"               
[130] "_ncbi_genomes/biomartr/fungi/Anthracocystis_flocculosa_PF-1_genomic_refseq.fna.gz"                         
[131] "_ncbi_genomes/biomartr/fungi/Moesziomyces_antarcticus_genomic_refseq.fna.gz"                               
[132] "_ncbi_genomes/biomartr/fungi/Sphaerulina_musiva_SO2202_genomic_refseq.fna.gz"                              
[133] "_ncbi_genomes/biomartr/fungi/Serpula_lacrymans_var._lacrymans_S7.9_genomic_refseq.fna.gz"                  
[134] "_ncbi_genomes/biomartr/fungi/Cladophialophora_carrionii_CBS_160.54_genomic_refseq.fna.gz"                  
[135] "_ncbi_genomes/biomartr/fungi/Rhinocladiella_mackenziei_CBS_650.93_genomic_refseq.fna.gz"                   
[136] "_ncbi_genomes/biomartr/fungi/Cladophialophora_bantiana_CBS_173.52_genomic_refseq.fna.gz"                   
[137] "_ncbi_genomes/biomartr/fungi/Exophiala_spinifera_genomic_refseq.fna.gz"                                    
[138] "_ncbi_genomes/biomartr/fungi/Metarhizium_acridum_CQMa_102_genomic_refseq.fna.gz"                           
[139] "_ncbi_genomes/biomartr/fungi/Exserohilum_turcica_Et28A_genomic_refseq.fna.gz"                              
[140] "_ncbi_genomes/biomartr/fungi/Fusarium_pseudograminearum_CS3096_genomic_refseq.fna.gz"                      
[141] "_ncbi_genomes/biomartr/fungi/Bipolaris_oryzae_ATCC_44560_genomic_refseq.fna.gz"                            
[142] "_ncbi_genomes/biomartr/fungi/Trichoderma_asperellum_CBS_433.97_genomic_refseq.fna.gz"                      
[143] "_ncbi_genomes/biomartr/fungi/Glarea_lozoyensis_ATCC_20868_genomic_refseq.fna.gz"                           
[144] "_ncbi_genomes/biomartr/fungi/Vavraia_culicis_subsp._floridensis_genomic_refseq.fna.gz"                     
[145] "_ncbi_genomes/biomartr/fungi/Postia_placenta_MAD-698-R-SB12_genomic_refseq.fna.gz"                         
[146] "_ncbi_genomes/biomartr/fungi/Gloeophyllum_trabeum_ATCC_11539_genomic_refseq.fna.gz"                        
[147] "_ncbi_genomes/biomartr/fungi/Cryptococcus_amylolentus_CBS_6039_genomic_refseq.fna.gz"                      
[148] "_ncbi_genomes/biomartr/fungi/Aspergillus_bombycis_genomic_refseq.fna.gz"                                   
[149] "_ncbi_genomes/biomartr/fungi/Spizellomyces_punctatus_DAOM_BR117_genomic_refseq.fna.gz"                     
[150] "_ncbi_genomes/biomartr/fungi/Batrachochytrium_dendrobatidis_JAM81_genomic_refseq.fna.gz"                   
[151] "_ncbi_genomes/biomartr/fungi/Ramularia_collo-cygni_genomic_refseq.fna.gz"                                  
[152] "_ncbi_genomes/biomartr/fungi/Tetrapisispora_phaffii_CBS_4417_genomic_refseq.fna.gz"                        
[153] "_ncbi_genomes/biomartr/fungi/Dichomitus_squalens_LYAD-421_SS1_genomic_refseq.fna.gz"                       
[154] "_ncbi_genomes/biomartr/fungi/Cordyceps_fumosorosea_ARSEF_2679_genomic_refseq.fna.gz"                       
[155] "_ncbi_genomes/biomartr/fungi/Fusarium_verticillioides_7600_genomic_refseq.fna.gz"                          
[156] "_ncbi_genomes/biomartr/fungi/Paracoccidioides_brasiliensis_Pb18_genomic_refseq.fna.gz"                     
[157] "_ncbi_genomes/biomartr/fungi/Cercospora_beticola_genomic_refseq.fna.gz"                                    
[158] "_ncbi_genomes/biomartr/fungi/Phialocephala_scopiformis_genomic_refseq.fna.gz"                              
[159] "_ncbi_genomes/biomartr/fungi/Aspergillus_nidulans_FGSC_A4_genomic_refseq.fna.gz"                           
[160] "_ncbi_genomes/biomartr/fungi/Ordospora_colligata_OC4_genomic_refseq.fna.gz"                                
[161] "_ncbi_genomes/biomartr/fungi/Beauveria_bassiana_ARSEF_2860_genomic_refseq.fna.gz"                          
[162] "_ncbi_genomes/biomartr/fungi/Coccidioides_posadasii_C735_delta_SOWgp_genomic_refseq.fna.gz"                
[163] "_ncbi_genomes/biomartr/fungi/Punctularia_strigosozonata_HHB-11173_SS5_genomic_refseq.fna.gz"               
[164] "_ncbi_genomes/biomartr/fungi/Melampsora_larici-populina_98AG31_genomic_refseq.fna.gz"                      
[165] "_ncbi_genomes/biomartr/fungi/Fomitiporia_mediterranea_MF3_22_genomic_refseq.fna.gz"                        
[166] "_ncbi_genomes/biomartr/fungi/Chaetomium_thermophilum_var._thermophilum_DSM_1495_genomic_refseq.fna.gz"     
[167] "_ncbi_genomes/biomartr/fungi/Exophiala_mesophila_genomic_refseq.fna.gz"                                    
[168] "_ncbi_genomes/biomartr/fungi/Exophiala_oligosperma_genomic_refseq.fna.gz"                                  
[169] "_ncbi_genomes/biomartr/fungi/Phaeoacremonium_minimum_UCRPA7_genomic_refseq.fna.gz"                         
[170] "_ncbi_genomes/biomartr/fungi/Grosmannia_clavigera_kw1407_genomic_refseq.fna.gz"                            
[171] "_ncbi_genomes/biomartr/fungi/Phanerochaete_carnosa_HHB-10118-sp_genomic_refseq.fna.gz"                     
[172] "_ncbi_genomes/biomartr/fungi/Diplodia_corticola_genomic_refseq.fna.gz"                                     
[173] "_ncbi_genomes/biomartr/fungi/Wallemia_ichthyophaga_EXF-994_genomic_refseq.fna.gz"                          
[174] "_ncbi_genomes/biomartr/fungi/Verruconis_gallopava_genomic_refseq.fna.gz"                                   
[175] "_ncbi_genomes/biomartr/fungi/Fonsecaea_monophora_genomic_refseq.fna.gz"                                    
[176] "_ncbi_genomes/biomartr/fungi/Pneumocystis_murina_B123_genomic_refseq.fna.gz"                               
[177] "_ncbi_genomes/biomartr/fungi/Candida_orthopsilosis_Co_90-125_genomic_refseq.fna.gz"                        
[178] "_ncbi_genomes/biomartr/fungi/Pochonia_chlamydosporia_170_genomic_refseq.fna.gz"                            
[179] "_ncbi_genomes/biomartr/fungi/Cyphellophora_europaea_CBS_101466_genomic_refseq.fna.gz"                      
[180] "_ncbi_genomes/biomartr/fungi/Aspergillus_steynii_IBT_23096_genomic_refseq.fna.gz"                          
[181] "_ncbi_genomes/biomartr/fungi/Kuraishia_capsulata_CBS_1993_genomic_refseq.fna.gz"                           
[182] "_ncbi_genomes/biomartr/fungi/Magnaporthe_oryzae_70-15_genomic_refseq.fna.gz"                               
[183] "_ncbi_genomes/biomartr/fungi/Kwoniella_bestiolae_CBS_10118_genomic_refseq.fna.gz"                          
[184] "_ncbi_genomes/biomartr/fungi/Kwoniella_dejecticola_CBS_10117_genomic_refseq.fna.gz"                        
[185] "_ncbi_genomes/biomartr/fungi/Pseudozyma_hubeiensis_SY62_genomic_refseq.fna.gz"                             
[186] "_ncbi_genomes/biomartr/fungi/Spathaspora_passalidarum_NRRL_Y-27907_genomic_refseq.fna.gz"                  
[187] "_ncbi_genomes/biomartr/fungi/Aspergillus_novofumigatus_IBT_16806_genomic_refseq.fna.gz"                    
[188] "_ncbi_genomes/biomartr/fungi/Pseudogymnoascus_verrucosus_genomic_refseq.fna.gz"                            
[189] "_ncbi_genomes/biomartr/fungi/Exophiala_xenobiotica_genomic_refseq.fna.gz"                                  
[190] "_ncbi_genomes/biomartr/fungi/Endocarpon_pusillum_Z07020_genomic_refseq.fna.gz"                             
[191] "_ncbi_genomes/biomartr/fungi/Meliniomyces_bicolor_E_genomic_refseq.fna.gz"                                 
[192] "_ncbi_genomes/biomartr/fungi/Lachancea_thermotolerans_CBS_6340_genomic_refseq.fna.gz"                      
[193] "_ncbi_genomes/biomartr/fungi/Pestalotiopsis_fici_W106-1_genomic_refseq.fna.gz"                             
[194] "_ncbi_genomes/biomartr/fungi/Trichoderma_gamsii_genomic_refseq.fna.gz"                                     
[195] "_ncbi_genomes/biomartr/fungi/Kazachstania_africana_CBS_2517_genomic_refseq.fna.gz"                         
[196] "_ncbi_genomes/biomartr/fungi/Kwoniella_pini_CBS_10737_genomic_refseq.fna.gz"                               
[197] "_ncbi_genomes/biomartr/fungi/Komagataella_phaffii_GS115_genomic_refseq.fna.gz"                             
[198] "_ncbi_genomes/biomartr/fungi/Ogataea_polymorpha_genomic_refseq.fna.gz"                                     
[199] "_ncbi_genomes/biomartr/fungi/Kwoniella_mangroviensis_CBS_8507_genomic_refseq.fna.gz"                       
[200] "_ncbi_genomes/biomartr/fungi/Cladophialophora_yegresii_CBS_114405_genomic_refseq.fna.gz"                   
[201] "_ncbi_genomes/biomartr/fungi/[Candida]_auris_genomic_refseq.fna.gz"                                        
[202] "_ncbi_genomes/biomartr/fungi/Metarhizium_brunneum_ARSEF_3297_genomic_refseq.fna.gz"                        
[203] "_ncbi_genomes/biomartr/fungi/Aureobasidium_namibiae_CBS_147.97_genomic_refseq.fna.gz"                      
[204] "_ncbi_genomes/biomartr/fungi/Scedosporium_apiospermum_genomic_refseq.fna.gz"                               
[205] "_ncbi_genomes/biomartr/fungi/Metarhizium_robertsii_ARSEF_23_genomic_refseq.fna.gz"                         
[206] "_ncbi_genomes/biomartr/fungi/Cladophialophora_immunda_genomic_refseq.fna.gz"                               
[207] "_ncbi_genomes/biomartr/fungi/Encephalitozoon_romaleae_SJ-2008_genomic_refseq.fna.gz"                       
[208] "_ncbi_genomes/biomartr/fungi/Nematocida_parisii_ERTm1_genomic_refseq.fna.gz"                               
[209] "_ncbi_genomes/biomartr/fungi/Kazachstania_naganishii_CBS_8797_genomic_refseq.fna.gz"                       
[210] "_ncbi_genomes/biomartr/fungi/Fibroporia_radiculosa_genomic_refseq.fna.gz"                                  
[211] "_ncbi_genomes/biomartr/fungi/Pseudogymnoascus_destructans_genomic_refseq.fna.gz"                           
[212] "_ncbi_genomes/biomartr/fungi/Marssonina_brunnea_f._sp._'multigermtubi'_MB_m1_genomic_refseq.fna.gz"        
[213] "_ncbi_genomes/biomartr/fungi/Hyphopichia_burtonii_NRRL_Y-1933_genomic_refseq.fna.gz"                       
[214] "_ncbi_genomes/biomartr/fungi/Aspergillus_fumigatus_Af293_genomic_refseq.fna.gz"                            
[215] "_ncbi_genomes/biomartr/fungi/Sugiyamaella_lignohabitans_genomic_refseq.fna.gz"                             
[216] "_ncbi_genomes/biomartr/fungi/Fonsecaea_nubica_genomic_refseq.fna.gz"                                       
[217] "_ncbi_genomes/biomartr/fungi/Schizosaccharomyces_cryophilus_OY26_genomic_refseq.fna.gz"                    
[218] "_ncbi_genomes/biomartr/fungi/Cutaneotrichosporon_oleaginosum_genomic_refseq.fna.gz"                        
[219] "_ncbi_genomes/biomartr/fungi/Fonsecaea_multimorphosa_CBS_102226_genomic_refseq.fna.gz"                     
[220] "_ncbi_genomes/biomartr/fungi/Heterobasidion_irregulare_TC_32-1_genomic_refseq.fna.gz"                      
[221] "_ncbi_genomes/biomartr/fungi/Ogataea_parapolymorpha_DL-1_genomic_refseq.fna.gz"                            
[222] "_ncbi_genomes/biomartr/fungi/Exophiala_aquamarina_CBS_119918_genomic_refseq.fna.gz"                        
[223] "_ncbi_genomes/biomartr/fungi/Wickerhamomyces_ciferrii_genomic_refseq.fna.gz"                               
[224] "_ncbi_genomes/biomartr/fungi/Aureobasidium_subglaciale_EXF-2481_genomic_refseq.fna.gz"                     
[225] "_ncbi_genomes/biomartr/fungi/Zymoseptoria_tritici_IPO323_genomic_refseq.fna.gz"                            
[226] "_ncbi_genomes/biomartr/fungi/Paracoccidioides_lutzii_Pb01_genomic_refseq.fna.gz"                           
[227] "_ncbi_genomes/biomartr/fungi/Verticillium_alfalfae_VaMs.102_genomic_refseq.fna.gz"                         
[228] "_ncbi_genomes/biomartr/fungi/Tetrapisispora_blattae_CBS_6284_genomic_refseq.fna.gz"                        
[229] "_ncbi_genomes/biomartr/fungi/Saccharomyces_eubayanus_genomic_refseq.fna.gz"                                
[230] "_ncbi_genomes/biomartr/fungi/Penicillium_rubens_Wisconsin_54-1255_genomic_refseq.fna.gz"                   
[231] "_ncbi_genomes/biomartr/fungi/Cladophialophora_psammophila_CBS_110553_genomic_refseq.fna.gz"                
[232] "_ncbi_genomes/biomartr/fungi/Colletotrichum_orchidophilum_genomic_refseq.fna.gz"                           
[233] "_ncbi_genomes/biomartr/fungi/Xylona_heveae_TC161_genomic_refseq.fna.gz"                                    
[234] "_ncbi_genomes/biomartr/fungi/Lachancea_lanzarotensis_genomic_refseq.fna.gz"                                
[235] "_ncbi_genomes/biomartr/fungi/Fonsecaea_erecta_genomic_refseq.fna.gz"                                       
[236] "_ncbi_genomes/biomartr/fungi/Kalmanozyma_brasiliensis_GHG001_genomic_refseq.fna.gz"                        
[237] "_ncbi_genomes/biomartr/fungi/Talaromyces_atroroseus_genomic_refseq.fna.gz"                                 
[238] "_ncbi_genomes/biomartr/fungi/Paraphaeosphaeria_sporulosa_genomic_refseq.fna.gz"                            
[239] "_ncbi_genomes/biomartr/fungi/Mitosporidium_daphniae_genomic_refseq.fna.gz"                                 
[240] "_ncbi_genomes/biomartr/fungi/Phialophora_attae_genomic_refseq.fna.gz"                                      
[241] "_ncbi_genomes/biomartr/fungi/Blastomyces_gilchristii_SLH14081_genomic_refseq.fna.gz"                       
[242] "_ncbi_genomes/biomartr/fungi/Wallemia_mellicola_CBS_633.66_genomic_refseq.fna.gz"                          
[243] "_ncbi_genomes/biomartr/fungi/Baudoinia_panamericana_UAMH_10762_genomic_refseq.fna.gz"                      
[244] "_ncbi_genomes/biomartr/fungi/Penicillium_arizonense_genomic_refseq.fna.gz"                                 
[245] "_ncbi_genomes/biomartr/fungi/Pseudocercospora_fijiensis_CIRAD86_genomic_refseq.fna.gz"
seannamctaggart commented 6 years ago

Hello Hajk;

MANY apologies for the delayed reply. Thank you so much for your response. I was not using the most recent version of Biomartr, but have now installed the newest version as you suggested. I am using ubuntu. The error message has changed slightly to:

biomartr::meta.retrieval( db = "refseq", kingdom = "fungi",type = "genome") #, path="/home/seannamctaggart/data/ncbi_genomes/biomartr/fungi") trying URL 'ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/fungi/assembly_summary.txt' Error: The FTP site 'ftp://ftp.ncbi.nlm.nih.gov/' cannot be reached. Are you connected to the internet? Is the the FTP site 'ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/fungi/assembly_summary.txt' currently available? In addition: Warning message: In download.file(url, ...) : URL 'ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/fungi/assembly_summary.txt': status was 'Failure when receiving data from the peer'

I am connected to the internet and the FTP site is available, so I have contacted my ICT department to see if there is a firewall issue.

Again, many thanks for your reply and offer to help. I will be needing more than just the fungal sequences, and will be updating the files on a regular basis. I was drawn to your package as it seemed like the best way to ensure a reproducible and simple way to download the files and I just need to keep track of the download dates.

Best wishes,

Seanna

seannamctaggart commented 6 years ago

Hello Hans;

I am still unsure of reason behind the problem, but it seems that using RStudio Server (accessed through a web-browser) is causing the problem. When I use Rstudio directly on my laptop, I have no issues with the package.

Best wishes,

Seanna

HajkD commented 6 years ago

Hi Seanna,

Thank you so much for clarifying and for troubleshooting.

It might be that RStudio Server has some firewall related issues with massive data retrieval.

I can dig deeper into and see if there is a way to set RStudio Server options in a way so that they are compatible with biomartr.

Many thanks and kind regards, Hajk

seannamctaggart commented 6 years ago

Hello Hajk;

I had a question about the meta-retrieval command. I am downloading far few sequences than I would have expected. For example, meta.retrievalhttps://ropensci.github.io/biomartr/reference/meta.retrieval.html(kingdom = "bacteria", db = "refseq", type = "genome") results in only 91 sequences downloaded, despite the fact that the /genomes/refseq/bacteria folder contains thousands of folders.

Could you please clarify if I am doing something wrong?

Best,

Seanna

From: Hajk-Georg Drost [mailto:notifications@github.com] Sent: 09 June 2018 13:22 To: ropensci/biomartr biomartr@noreply.github.com Cc: Seanna McTaggart Seanna.McTaggart@lifearc.org; Mention mention@noreply.github.com Subject: Re: [ropensci/biomartr] ftp issues when using meta.retrieval (#27)

Hi Seanna,

Thank you so much for clarifying and for troubleshooting.

It might be that RStudio Server has some firewall related issues with massive data retrieval.

I can dig deeper into and see if there is a way to set RStudio Server options in a way so that they are compatible with biomartr.

Many thanks and kind regards, Hajk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/biomartr/issues/27#issuecomment-395965211, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AlhL0XaU5C9zwAVxE8K6MXxuoM5u9FZWks5t673XgaJpZM4UBfXe.

LifeArc is a company limited by guarantee no. 2698321, incorporated in England and a charity registered in England and Wales no. 1015243, and a charity registered in Scotland no. SC037861. Registered office Lynton House, 7-12 Tavistock Square, LONDON, WC1H 9LT, UK. T: +44 (0)20 7391 2700; F: +44(0)20 7391 2800.

This communication contains confidential information and may be legally privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any disclosure, distribution, copying or use of this communication or the information in it is prohibited and may be unlawful. If you have received this communication by mistake, please notify the sender or contact LifeArc by telephone on +44 (0)20 7391 2700 and then delete the e-mail and any copies of it.

No representation is made that this email or any attachments are free of viruses. Scanning for the presence of viruses in this e-mail, and any attachments, is highly recommended and is the responsibility of the recipient.

Any statements contained in this communication are personal to the author and are not necessarily the statements of LifeArc unless specifically stated.

HajkD commented 6 years ago

Hi Seanna,

Many thanks for your question.

Have you tried specifying the argument reference = FALSE?

biomartr::meta.retrieval(kingdom = "bacteria", db = "refseq", type = "genome", reference = FALSE)

The reason why you only retrieved 91 genomes is that the argument reference is set to TRUE as a default in meta.retrieval() (see documentation at ?meta.retrieval) and therefore only reference genomes are downloaded from NCBI RefSeq. Most of the hundreds or thousands of bacterial genomes on NCBI are not reference or representative genomes and thus if you still want to retrieve them you will need to specify reference = FALSE in meta.retrieval().

I will extend the examples in the Vignette to make this point more clear.

Thank you very much for making me aware of the possible shortcoming of the documentation and please let me know if it now works for you.

Kind regards, Hajk

seannamctaggart commented 6 years ago

Hi Hajk;

Thank you so much for this quick and comprehensive reply. Just to make sure that I understand the terminology, reference or representative genome refers to an ‘average’ genome or a genome compiled from several individuals of the same species (strain in bacteria?).

Thanks again. I have also tried the ncbi-genome-download function and it appears to be quite a bit faster to download.

Best wishes,

Seanna

From: Hajk-Georg Drost [mailto:notifications@github.com] Sent: 02 July 2018 15:30 To: ropensci/biomartr biomartr@noreply.github.com Cc: Seanna McTaggart Seanna.McTaggart@lifearc.org; Mention mention@noreply.github.com Subject: Re: [ropensci/biomartr] ftp issues when using meta.retrieval (#27)

Hi Seanna,

Many thanks for your question.

Have you tried specifying the argument reference = FALSE?

biomartr::meta.retrieval(kingdom = "bacteria", db = "refseq", type = "genome", reference = FALSE)

The reason why you only retrieved 91 genomes is that the argument reference is set to TRUE as a default in meta.retrieval() (see documentation at ?meta.retrieval) and therefore only reference genomes are downloaded from NCBI RefSeq. Most of the hundreds or thousands of bacterial genomes on NCBI are not reference or representative genomes and thus if you still want to retrieve them you will need to specify reference = FALSE in meta.retrieval().

I will extend the examples in the Vignette to make this point more clear.

Thank you very much for making me aware of the possible shortcoming of the documentation and please let me know if it now works for you.

Kind regards, Hajk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ropensci/biomartr/issues/27#issuecomment-401824764, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AlhL0exfMh0tu4QgETHzpxVtqUdBCKcyks5uCi51gaJpZM4UBfXe.

LifeArc is a company limited by guarantee no. 2698321, incorporated in England and a charity registered in England and Wales no. 1015243, and a charity registered in Scotland no. SC037861. Registered office Lynton House, 7-12 Tavistock Square, LONDON, WC1H 9LT, UK. T: +44 (0)20 7391 2700; F: +44(0)20 7391 2800.

This communication contains confidential information and may be legally privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any disclosure, distribution, copying or use of this communication or the information in it is prohibited and may be unlawful. If you have received this communication by mistake, please notify the sender or contact LifeArc by telephone on +44 (0)20 7391 2700 and then delete the e-mail and any copies of it.

No representation is made that this email or any attachments are free of viruses. Scanning for the presence of viruses in this e-mail, and any attachments, is highly recommended and is the responsibility of the recipient.

Any statements contained in this communication are personal to the author and are not necessarily the statements of LifeArc unless specifically stated.

HajkD commented 6 years ago

Hi Seanna,

The exact definition of reference or representative genomes is given by NCBI (see e.g. ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/Acetoanaerobium_noterae/latest_assembly_versions/GCF_900168025.1_IMG-taxon_2588253727_annotated_assembly/README.txt). But usually it means that this genome assembly was generated from a model/reference species using high-quality and deep sequencing techniques. Usually, these reference genomes are also used for generating annotations and for mapping reads etc (also of reads from closely related genomes). Non-reference genomes usually were generated with lower sequencing depth and were assembled using reference genomes as assembly guides.

I hope this helps?

Many thanks and kind regards, Hajk