Error in rAzureBatch::uploadBlob(id, nodeScriptsZip) : the given file does not exist

[djm158]

Before submitting a bug please check the following:

• [x] Start a new R session
• [x] Check your credentials file
• [x] Install the latest doAzureParallel package
• [x] Submit a minimal, reproducible example
• [x] run sessionInfo()

Description

script being run:

setCredentials("credentials.json")
cluster <- makeCluster("temp_cluster.json")
# set the parameters for the monte carlo simulation
mean_change = 1.001
volatility = 0.01
opening_price = 100

# define a new function to simulate closing prices
getClosingPrice <- function() {
  days <- 1825 # ~ 5 years
  movement <- rnorm(days, mean=mean_change, sd=volatility)
  path <- cumprod(c(opening_price, movement))
  closingPrice <- path[days]
  return(closingPrice)
}

start_s <- Sys.time()
# Run 10,000 simulations in series
closingPrices_s <- foreach(i = 1:10, .combine='c') %do% {
  replicate(1000, getClosingPrice())
}
end_s <- Sys.time()

# plot the 50 closing prices in a histogram to show the distribution of outcomes
hist(closingPrices_s)

# How long did it take?
difftime(end_s, start_s)

# Estimate runtime for 10 million (linear approximation)
1000 * difftime(end_s, start_s, unit = "min")

# Run 10 million simulations with doAzureParallel

# We will run 100 iterations where each iteration executes 100,000 simulations
opt <- list(chunkSize = 13) # optimizie runtime. Chunking allows us to run multiple iterations on a single instance of R.

start_p <- Sys.time()
closingPrices_p <- foreach(i = 1:100, .combine='c', .options.azure = opt) %dopar% {
  replicate(100000, getClosingPrice())
}
end_p <- Sys.time()
# How long did it take?
difftime(end_p, start_p, unit = "min")

# plot the 10 million closing prices in a histogram to show the distribution of outcomes
hist(closingPrices_p)

The pool is created and looks okay in my Azure portal. The problem is here:

closingPrices_p <- foreach(i = 1:100, .combine='c', .options.azure = opt) %dopar% {
  replicate(100000, getClosingPrice())
}

I get this error (verbose logging included):

The specified cluster 'temp' already exists. Cluster 'temp' will be used. Your cluster has been registered. Dedicated Node Count: 3 Low Priority Node Count: 3 -> PUT /job20180220192359?restype=container HTTP/2 -> Host: dmcgrath.blob.core.windows.net -> Accept-Encoding: gzip, deflate -> Accept: application/json, text/xml, application/xml, / -> x-ms-date: Tue, 20 Feb 2018 19:23:59 GMT -> x-ms-version: 2016-05-31 -> Authorization: SharedKey -> User-Agent: rAzureBatch/0.5.5 -> Content-Length: 0 -> <- HTTP/2 201 <- last-modified: Tue, 20 Feb 2018 19:24:00 GMT <- etag: "0x8D578977E9A416F" <- server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0 <- x-ms-request-id: 0b191c14-001e-0105-3380-aa88f8000000 <- x-ms-version: 2016-05-31 <- date: Tue, 20 Feb 2018 19:23:59 GMT <- Error in rAzureBatch::uploadBlob(id, nodeScriptsZip) : The given file does not exist. In addition: Warning message: running command '"zip" -r9X "node_scripts.zip" "C:/Users/dmcgrath/Documents/R/win-library/3.4/doAzureParallel/startup/cluster_setup.sh" "C:/Users/dmcgrath/Documents/R/win-library/3.4/doAzureParallel/startup/install_bioconductor.R" "C:/Users/dmcgrath/Documents/R/win-library/3.4/doAzureParallel/startup/install_cran.R" "C:/Users/dmcgrath/Documents/R/win-library/3.4/doAzureParallel/startup/install_github.R" "C:/Users/dmcgrath/Documents/R/win-library/3.4/doAzureParallel/startup/merger.R" "C:/Users/dmcgrath/Documents/R/win-library/3.4/doAzureParallel/startup/worker.R" -j -q' had status 127

my cluster config - temp_cluster.json:

{
  "name": "temp",
  "vmSize": "Standard_D2_v2",
  "maxTasksPerNode": 1,
  "poolSize": {
    "dedicatedNodes": {
      "min": 3,
      "max": 3
    },
    "lowPriorityNodes": {
      "min": 3,
      "max": 3
    },
    "autoscaleFormula": "QUEUE"
  },
  "containerImage": "rocker/tidyverse:latest",
  "rPackages": {
    "cran": [],
    "github": [],
    "bioconductor": []
  },
  "commandLine": []
}

Instruction to repro the problem if applicable

Run my script shown above.

sessionInfo():

R version 3.4.1 (2017-06-30) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale: [1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252 [4] LC_NUMERIC=C LC_TIME=English_United States.1252

attached base packages: [1] stats graphics grDevices utils datasets methods base

other attached packages: [1] doAzureParallel_0.6.2 iterators_1.0.8 foreach_1.4.4 RevoUtilsMath_10.0.0 RevoUtils_10.0.5
[6] RevoMods_11.0.0 MicrosoftML_1.5.0 mrsdeploy_1.1.2 RevoScaleR_9.2.1 lattice_0.20-35
[11] rpart_4.1-11

loaded via a namespace (and not attached): [1] codetools_0.2-15 CompatibilityAPI_1.1.0 digest_0.6.12 rAzureBatch_0.5.5 bitops_1.0-6
[6] grid_3.4.1 R6_2.2.2 jsonlite_1.5 httr_1.3.1 curl_2.6
[11] rjson_0.2.15 tools_3.4.1 RCurl_1.95-4.8 compiler_3.4.1 mrupdate_1.0.1

[brnleehng]

Hi @djm158

We were able to repro the issue. The issue was we zip our job preparation files to be uploaded once instead of multiple calls. However the utils package points to an external zip program from RTools. If RTools is not on the environment path, then it will throw this error. Do you have RTools installed? https://cran.r-project.org/bin/windows/Rtools/

This will fix the issue.

I'll take a look at other zip R packages that doesn't require this installation.

Thanks, Brian

[djm158]

thanks @brnleehng this resolved my issue!