Getting all results of the OpenML Bot via the R API does NOT work!

[PhilippPro]

@berndbischl says it is possible to download all results of the OpenML Bot via the R API and to put it in a nice data.frame. I will show that it is not possible and still errors occur as @DanielKuehn87 and me expected.

I use following script (from HowToWriteABot.Rmd):

library(OpenML)
# Put the results into a table
# get performances
system.time(my_runs <- listOMLRunEvaluations(tag = "botV1", evaluation.measure = "area_under_roc_curve"))
# subset results on some information(s) and measure(s)
my_runs = my_runs[, c("run.id", "task.id", "area.under.roc.curve" )]

# get hyperparameters
system.time(runs <- listOMLRuns(tag = "botV1"))
system.time(paras <- listOMLSetup(runs$setup.id))
paras_names = names(lrn.par.set$pars)
paras = paras[paras$parameter.name %in% paras_names, c("setup.id", "parameter.name", "value")]
library(tidyr)
library(dplyr)
paras = spread(paras, key = parameter.name, value = value)
paras = merge(paras, runs[, c("run.id", "setup.id")], key = "setup.id")
paras = select(paras, -setup.id)

# Put things together
results = merge(my_runs, paras, by = "run.id")
# Put it in a nice order
results = results[, c(setdiff(names(results), "area.under.roc.curve"), "area.under.roc.curve")]
# Now you can compare the performances of your different hyperparameters
print(head(results))

# Get some informations about the dataset
data.id = task$input$data.set$desc$id
qualities = getOMLDataSetQualities(data.id = data.id)
qualities =  spread(qualities, key = name, value = value)
qualities = cbind(task.id = 3, qualities[, c("NumberOfClasses", "NumberOfInstances")])
# Add them to existing table
results = merge(results, qualities, by = "task.id")
print(head(results))

It fails in the first line:

> system.time(my_runs <- listOMLRunEvaluations(tag = "botV1", evaluation.measure = "area_under_roc_curve"))
Downloading from 'https://www.openml.org/api/v1/json/evaluation/list/tag/botV1/function/area_under_roc_curve' to '<mem>'.
Fehler in doHTTRCall(method, url = url, query = list(api_key = conf$apikey),  : 
  ERROR (code = <NA>) in server response: <NA>
  Server returned a HTML document!
Timing stopped at: 0.044 0.016 83.935 

Also this does not work:

> system.time(runs <- listOMLRuns(tag = "botV1"))
Downloading from 'https://www.openml.org/api/v1/json/run/list/tag/botV1' to '<mem>'.
Fehler in doHTTRCall(method, url = url, query = list(api_key = conf$apikey),  : 
  ERROR (code = <NA>) in server response: <NA>
  Server returned a HTML document!
Timing stopped at: 0.044 0.016 65.558 

[joaquinvanschoren]

Thanks. Can you post the exact error message? The stuff that is apparently in the HTML?

[berndbischl]

@giuseppec can you also take a look?

[PhilippPro]

Where can I find the HTML?

[giuseppec]

This seems to be a server issue, moved it here https://github.com/openml/OpenML/issues/634