`zenodo$publishRecord`: 504 Gateway Timeout

bschilder commented 2 years ago

Hello,

I just made a record and uploaded all 8,000+ files (29Gb total). However every time I try to publish it I get the following error:

Error

Error: lexical error: invalid char in json text.
                                       <html><body><h1>504 Gateway Tim
                     (right here) ------^

I've noticed similar issues when I try to publish the record directly from the Zenodo website. It simply times out and doesn't actually publish. Indeed, a very similar Issue was posted here: https://github.com/zenodo/zenodo/issues/2131

Is there a way to adjust the timeout period programmatically? options(timeout=) doesn't seem to make a difference.

Code

newrec <- zen4R::ZenodoRecord$new() 
    newrec$setTitle(title = "PAINTOR")
    newrec$setDescription(description = 
                              paste("Functional annotations used for fine-mapping with PAINTOR.", 
                                    "https://github.com/gkichaev/PAINTOR_V3.0/wiki/2b.-Overlapping-annotations",
                                    "These annotations are used by echolocatoR to perform",
                                    "in silico validation analyses.",
                                    "Data has been reprocessed to enable API access and rapid querying",
                                    "https://github.com/RajLabMSSM/echolocatoR"))
    newrec$setUploadType(uploadType = "dataset")
    newrec$setDOI("10.5281/zenodo.7063240")
    newrec$addCreator(firstname = "Brian", lastname = "Schilder", 
                      affiliation = "Imperial College London") 
    newrec <- zenodo$depositRecord(newrec) 
    #### Upload files ####
    uploads <- lapply(stats::setNames(all_files,
                                      all_files),
                      function(x){
        ### Go to sleep for 1h to reset limit of 5000 files/hour.
        if(which(all_files==x) %in% seq(4999,4999*100,5000)) {
            message("Waiting for 1 hour to reset upload limits.")
            Sys.sleep(3600)  
        }    
        ### Go to sleep for 1min to reset limit of 100 files/min.
          if(which(all_files==x) %in% seq(99,99*10000,100)) {
              message("Waiting for 60 seconds to reset upload limits.")
              Sys.sleep(60)  
          }   
        message("Uploading: ",x)
        zenodo$uploadFile(x, record = newrec) 
    })
    file_res <- zenodo$uploadFile(zipped_file, record = newrec) 
    #### Publish record ####
    options(timeout=3600*2)

    rec_res <- zenodo$publishRecord(recordId = newrec$id)

Session info

``` R Under development (unstable) (2022-02-25 r81808) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 20.04.5 LTS Matrix products: default BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.8.so locale: [1] C attached base packages: [1] stats graphics grDevices utils datasets methods base loaded via a namespace (and not attached): [1] httr_1.4.4 compiler_4.2.0 keyring_1.3.0 assertthat_0.2.1 R6_2.5.1 [6] zen4R_0.7 cli_3.3.0 tools_4.2.0 curl_4.3.2 xml2_1.3.3 [11] data.table_1.14.2 jsonlite_1.8.0 ```

Many thanks in advance, Brian

bschilder commented 2 years ago

Just saw this as well. Hoping this is simply a temporary outage and will resolve itself, but will let you know what happens: https://github.com/zenodo/zenodo/issues/934

eblondel commented 1 year ago

@bschilder any news on that?

bschilder commented 1 year ago

It appears to be an inherent flaw with Zenodo's uploading system, which is unable to handle many files.

Still waiting for a follow up response from the Zenodo team. You can follow the discussion here: https://github.com/zenodo/zenodo/issues/934

ThierryO commented 1 year ago

I'm having a similar issue.

> myrec <- zenodo$depositRecordVersion(
+   record = myrec, delete_latest_files = TRUE, publish = FALSE
+ )
[zen4R][INFO] ZenodoManager - Creating new version for record '5834713' (concept DOI: '10.5281/zenodo.5812597') 
Error: lexical error: invalid char in json text.
                                       <html><body><h1>504 Gateway Tim
                     (right here) ------^

eblondel commented 12 months ago

This was due to Zenodo infra issues. I believe it's solved since that time.

eblondel / zen4R