search

claraqin / neonMicrobe

Processing NEON soil microbe marker gene sequence data into ASV tables.
GNU Lesser General Public License v3.0
9 stars 4 forks source link

Some fastq file URLs return 404 error #9

Closed claraqin closed 4 years ago

claraqin commented 4 years ago

Sometimes, the rawDataFilePath column of the mmg_soilRawDataFiles table (in product ID 10108) will contain bad URLs. This will continue for a while during NEON's file renaming effort. For example:

trying URL 'https://neon-microbial-raw-seq-files.s3.data.neonscience.org/2017/BMI_B69PP_ITS_R1_fastq.tar.gz'
Error in download.file(url = as.character(u.urls[i]), destfile = ifelse(dir.exists(outdir),  : 
  cannot open URL 'https://neon-microbial-raw-seq-files.s3.data.neonscience.org/2017/BMI_B69PP_ITS_R1_fastq.tar.gz'
In addition: Warning message:
In download.file(url = as.character(u.urls[i]), destfile = ifelse(dir.exists(outdir),  :
  cannot open URL 'https://neon-microbial-raw-seq-files.s3.data.neonscience.org/2017/BMI_B69PP_ITS_R1_fastq.tar.gz': HTTP status was '404 Not Found'
claraqin commented 4 years ago

To get around this issue for now, replacing the "_fastq" in the URL with ".fastq" usually works.