Bulk download failing

[bschilder]

Originally posted by @jdblischak in Issue #1 :

• Caveat: I didn't actually perform the bulk download from within the app. It always crashed at 87%. I extracted the code from app.R to combine all the results.

[bschilder]

Doesn't crash per se on Macbook with Chrome, but at 87% it returns an html file instead.

``` --   ```

[bschilder]

When running Shiny App locally, produces csv fine. This suggests it's an issue on shinyapps.io's server. Are there caps on the total file size that can be downloaded? echolocatoR_Finemapping_Portal.all_data.multi_finemap.csv is 686.6Mb, which is large but not crazy large.

[bschilder]

Reading some shiny documentation and found that this may be related to how I'm calling the download. Switched:

 datasetInput_bulk <- reactive({
    withProgress(message = 'Gathering and merging all results', value = 0, {
      dat_paths <- subset(all_paths, file_type=="multi_finemap")
      merged_DT <- lapply(1:nrow(dat_paths), function(i){
        ROW <- dat_paths[i,]
        dat <- data.table::fread(ROW$file_path, nThread = 1)
        dat <- cbind(study=ROW$study, study_type=ROW$study_type, LD_ref=ROW$LD_ref, unique(dat))
        incProgress(1/nrow(dat_paths), detail = paste(round(i/nrow(dat_paths),2)*100,"%"))
        return(dat)
      }) %>% data.table::rbindlist(fill = T)
    })
    return(merged_DT)
  })

datasetInput_bulk <- function(){
    withProgress(message = 'Gathering and merging all results', value = 0, {
      dat_paths <- subset(all_paths, file_type=="multi_finemap") 
      merged_DT <- lapply(1:nrow(dat_paths), function(i){
        ROW <- dat_paths[i,]
        dat <- data.table::fread(ROW$file_path, nThread = 1)
        dat <- cbind(study=ROW$study, study_type=ROW$study_type, LD_ref=ROW$LD_ref, unique(dat))
        incProgress(1/nrow(dat_paths), detail = paste(round(i/nrow(dat_paths),2)*100,"%"))
        return(dat)
      }) %>% data.table::rbindlist(fill = T)
    })
    return(merged_DT)
  }

As before, this works fine locally.

However, after rebuilding the image and pushing the new one to shinyapps.io (takes a while), I found this made no difference. Produces the same html file instead of csv. Also noticed the "Disconnected from server" message in the bottom left. This wasn't due to lack of activity, as I tried interacting with the app the whole time to ensure it didn't go to sleep.

Potentially related issue, will try the suggested solution: https://github.com/rstudio/shiny/issues/2152

[jdblischak]

Doesn't crash per se on Macbook with Chrome, but at 87% it returns an html file instead.

@bschilder To clarify, this is the same behavior I observed. I was also using Chrome on macOS. At 87%, it downloads an HTML file and then shows the message "Disconnected from server".

[bschilder]

@jdblischak thanks for confirming. Still trying to figure out a fix for this. In the meantime, I could provide a Google Drive link to the merged summary stats.

[bschilder]

Just added a link to the Pre-merged bulk download here:

[jdblischak]

Just added a link to the Pre-merged bulk download here:

I just confirmed that I am able to successfully download the pre-merged bulk data from Google Drive. Thanks!

Note however that the bulk data would be more useful if every loci was processed with the same version of echolocatoR. As we discussed in #2, I found it difficult to compare the performance of each method across loci since the potential values in each column are not consistent across loci.