search

ISI-MIP / isimip-files-api

Webservice to asynchronously perform operations on NetCDF files before download
MIT License
1 stars 0 forks source link

Failed attempt to convert the country python code to R #5

Open GeraldCNelson opened 6 months ago

GeraldCNelson commented 6 months ago

I can run the country code in python successfully but my attempt to convert to R below fails. R code first, then error messages.

*R code

library(httr)
library(jsonlite)
library(logger)
library(fs)
library(xml2)

# Define the URL and data
url <- 'https://files.isimip.org/api/v2'

paths <- list(
  'ISIMIP3b/InputData/climate/atmosphere/bias-adjusted/global/daily/ssp585/GFDL-ESM4/gfdl-esm4_r1i1p1f1_w5e5_ssp585_tas_global_daily_2015_2020.nc'
)

data <- list(
  paths = paths,
  operations = list(
    list(
      operation = 'mask_country',
      country = 'aus',       # three letter code for, e.g. australia
      compute_mean = FALSE,  # optional: set to TRUE to get a time series of the field mean
      output_csv = FALSE     # optional: set to TRUE to get a CSV file instead of NetCDF
    )
  )
)

download_path <- 'download'

# Log the request payload
log_info("Request payload", payload = toJSON(data, auto_unbox = TRUE))

# Perform the initial request to the server
response <- POST(url, body = toJSON(data, auto_unbox = TRUE), encode = "json")

# Log the HTTP status and response content
log_info("HTTP status", status_code = status_code(response))
response_content <- content(response, "text", encoding = "UTF-8")
log_info("HTTP response", response_content)

if (http_error(response)) {
  log_error("Job submission failed", error = response_content)
} else {
  # Check if the response content is JSON or XML
  if (startsWith(response_content, "{")) {
    job <- fromJSON(response_content, simplifyVector = TRUE)
  } else if (startsWith(response_content, "<")) {
    job <- as_list(read_xml(response_content))
  } else {
    stop("Unknown response format")
  }

  log_info("Job submitted", id = job$id, status = job$status)

  while (job$status %in% c("queued", "started")) {
    # wait for 4 sec
    Sys.sleep(4)

    # check the status of the job
    job_response <- GET(job$job_url)
    job_content <- content(job_response, "text", encoding = "UTF-8")
    if (startsWith(job_content, "{")) {
      job <- fromJSON(job_content, simplifyVector = TRUE)
    } else if (startsWith(job_content, "<")) {
      job <- as_list(read_xml(job_content))
    } else {
      stop("Unknown response format")
    }

    log_info("Job updated", id = job$id, status = job$status, meta = job$meta)
  }

  if (job$status == "finished") {
    # download file
    zip_path <- file.path(download_path, job$file_name)
    dir_create(path_dir(zip_path))
    log_info("Downloading", file_url = job$file_url)

    download.file(job$file_url, zip_path, mode = "wb")

    # extract zip file
    out_path <- file.path(download_path, tools::file_path_sans_ext(basename(zip_path)))
    dir_create(out_path)
    log_info("Extracting", zip_path = out_path)

    unzip(zip_path, exdir = out_path)
  }
}
jochenklar commented 6 months ago

Hi @GeraldCNelson , thanks for testing! I think you need to add content_type_json() to the POST call to set the Content-Type header to application/json. An I need to fix the api to return 415 Unsupported Media Type if the header is not set.

jochenklar commented 6 months ago

I would like to use parts of your code to create R examples. Is that ok with you?

GeraldCNelson commented 6 months ago

Sure! Let me know how it goes.

-- Gerald C. Nelson Professor Emeritus, University of Illinois, Urbana-Champaign Telephone: +1-217-390-7888 Email: @.**@.> http://bit.ly/1arho7d

From: Jochen Klar @.> Date: Monday, June 3, 2024 at 3:59 AM To: ISI-MIP/isimip-files-api @.> Cc: Gerald Charles Nelson @.>, Mention @.> Subject: Re: [ISI-MIP/isimip-files-api] Failed attempt to convert the country python code to R (Issue #5)

I would like to use parts of your code to create R examples. Is that ok with you?

— Reply to this email directly, view it on GitHubhttps://github.com/ISI-MIP/isimip-files-api/issues/5#issuecomment-2144783254, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB6UJMVIYZY4VULD6TXWQI3ZFQ47BAVCNFSM6AAAAABIIIKAAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBUG44DGMRVGQ. You are receiving this because you were mentioned.Message ID: @.***>

GeraldCNelson commented 6 months ago

Let me know when your fix to the api is ready and I’ll give it a try.

Did I mention that I used ChatGPT to convert the python to R?

-- Gerald C. Nelson Professor Emeritus, University of Illinois, Urbana-Champaign Telephone: +1-217-390-7888 Email: @.**@.> http://bit.ly/1arho7d

From: Jochen Klar @.> Date: Monday, June 3, 2024 at 3:48 AM To: ISI-MIP/isimip-files-api @.> Cc: Gerald Charles Nelson @.>, Mention @.> Subject: Re: [ISI-MIP/isimip-files-api] Failed attempt to convert the country python code to R (Issue #5)

Hi @GeraldCNelsonhttps://github.com/GeraldCNelson , thanks for testing! I think you need to add content_type_json() to the POST call to set the Content-Type header to application/json. An I need to fix the api to return 415 Unsupported Media Type if the header is not set.

— Reply to this email directly, view it on GitHubhttps://github.com/ISI-MIP/isimip-files-api/issues/5#issuecomment-2144761977, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB6UJMT4NOBGIANM3DK4YADZFQ3X5AVCNFSM6AAAAABIIIKAAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBUG43DCOJXG4. You are receiving this because you were mentioned.Message ID: @.***>

jochenklar commented 6 months ago

Great! It should work now, or at least give the proper error. But you need to adjust the script using like this:

response <- POST(url, body = toJSON(data, auto_unbox = TRUE), encode = "json", content_type_json())
GeraldCNelson commented 6 months ago

Still getting a status code of 400. Here’s the structure of the response list

List of 10 $ url : chr https://files.isimip.org/api/v2 $ status_code: int 400 $ headers :List of 7 ..$ server : chr "nginx/1.16.1" ..$ date : chr "Mon, 03 Jun 2024 15:05:05 GMT" ..$ content-type : chr "application/json" ..$ content-length : chr "76" ..$ connection : chr "keep-alive" ..$ access-control-allow-origin: chr "" ..$ access-control-allow-origin: chr "" ..- attr(, "class")= chr [1:2] "insensitive" "list" $ all_headers:List of 1 ..$ :List of 3 .. ..$ status : int 400 .. ..$ version: chr "HTTP/1.1" .. ..$ headers:List of 7 .. .. ..$ server : chr "nginx/1.16.1" .. .. ..$ date : chr "Mon, 03 Jun 2024 15:05:05 GMT" .. .. ..$ content-type : chr "application/json" .. .. ..$ content-length : chr "76" .. .. ..$ connection : chr "keep-alive" .. .. ..$ access-control-allow-origin: chr "" .. .. ..$ access-control-allow-origin: chr "" .. .. ..- attr(, "class")= chr [1:2] "insensitive" "list" $ cookies :'data.frame': 0 obs. of 7 variables: ..$ domain : logi(0) ..$ flag : logi(0) ..$ path : logi(0) ..$ secure : logi(0) ..$ expiration: 'POSIXct' num(0) ..$ name : logi(0) ..$ value : logi(0) $ content : raw [1:76] 7b 22 65 72 ... $ date : POSIXct[1:1], format: "2024-06-03 15:05:05" $ times : Named num [1:6] 0 0.000018 0 0.000091 0.161818 ... ..- attr(, "names")= chr [1:6] "redirect" "namelookup" "connect" "pretransfer" ... $ request :List of 7 ..$ method : chr "POST" ..$ url : chr https://files.isimip.org/api/v2 ..$ headers : Named chr [1:2] "application/json, text/xml, application/xml, /" "application/json" .. ..- attr(, "names")= chr [1:2] "Accept" "Content-Type" ..$ fields : NULL ..$ options :List of 4 .. ..$ useragent : chr "libcurl/8.6.0 r-curl/5.2.1 httr/1.4.7" .. ..$ post : logi TRUE .. ..$ postfieldsize: int 6903 .. ..$ postfields : raw [1:6903] 5b 22 66 75 ... ..$ auth_token: NULL ..$ output : list() .. ..- attr(, "class")= chr [1:2] "write_memory" "write_function" ..- attr(, "class")= chr "request" $ handle :Class 'curl_handle'

-- Gerald C. Nelson Professor Emeritus, University of Illinois, Urbana-Champaign Telephone: +1-217-390-7888 Email: @.**@.> http://bit.ly/1arho7d

From: Jochen Klar @.> Date: Monday, June 3, 2024 at 9:03 AM To: ISI-MIP/isimip-files-api @.> Cc: Gerald Charles Nelson @.>, Mention @.> Subject: Re: [ISI-MIP/isimip-files-api] Failed attempt to convert the country python code to R (Issue #5)

Great! It should work now, or at least give the proper error. But you need to adjust the script using like this:

response <- POST(url, body = toJSON(data, auto_unbox = TRUE), encode = "json", content_type_json())

— Reply to this email directly, view it on GitHubhttps://github.com/ISI-MIP/isimip-files-api/issues/5#issuecomment-2145433933, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB6UJMWTKUEK54KJ6262JHTZFSAVRAVCNFSM6AAAAABIIIKAAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBVGQZTGOJTGM. You are receiving this because you were mentioned.Message ID: @.***>

jochenklar commented 6 months ago

Can you try this:

library(httr)
library(jsonlite)
library(logger)
library(fs)
library(xml2)

url <- 'https://files.isimip.org/api/v2'

paths <- list(
  'ISIMIP3b/InputData/climate/atmosphere/bias-adjusted/global/daily/ssp585/GFDL-ESM4/gfdl-esm4_r1i1p1f1_w5e5_ssp585_tas_global_daily_2015_2020.nc',
  'ISIMIP3b/InputData/climate/atmosphere/bias-adjusted/global/daily/ssp585/GFDL-ESM4/gfdl-esm4_r1i1p1f1_w5e5_ssp585_tas_global_daily_2021_2030.nc'
)

data <- list(
  paths = paths,
  operations = list(
    list(
      operation = 'mask_country',
      country = 'aus',       # three letter code for, e.g. australia
      compute_mean = FALSE,  # optional: set to TRUE to get a time series of the field mean
      output_csv = FALSE     # optional: set to TRUE to get a CSV file instead of NetCDF
    )
  )
)

download_path <- 'download'

response <- POST(url, body = toJSON(data, auto_unbox = TRUE), encode = "json", content_type_json())
response_content <- content(response, "text", encoding = "UTF-8")
print(response_content)

job <- fromJSON(response_content, simplifyVector = TRUE)

while (job$status %in% c("queued", "started")) {
  Sys.sleep(4)

  job_response <- GET(job$job_url)
  job_content <- content(job_response, "text", encoding = "UTF-8")
  print(job_content)

  job <- fromJSON(job_content, simplifyVector = TRUE)
}

if (job$status == "finished") {
  zip_path <- file.path(download_path, job$file_name)
  dir_create(path_dir(zip_path))
  download.file(job$file_url, zip_path, mode = "wb")

  out_path <- file.path(download_path, tools::file_path_sans_ext(basename(zip_path)))
  dir_create(out_path)

  unzip(zip_path, exdir = out_path)
}

The "logging" is a bit crude, but there is less boilerplate code. This script worked on my machine.

GeraldCNelson commented 6 months ago

Worked on mine too! I see you are downloading aus which is nice to have demonstrated. Is the polygon download api working?

-- Gerald C. Nelson Professor Emeritus, University of Illinois, Urbana-Champaign Telephone: +1-217-390-7888 Email: @.**@.> http://bit.ly/1arho7d

From: Jochen Klar @.> Date: Monday, June 3, 2024 at 11:11 AM To: ISI-MIP/isimip-files-api @.> Cc: Gerald Charles Nelson @.>, Mention @.> Subject: Re: [ISI-MIP/isimip-files-api] Failed attempt to convert the country python code to R (Issue #5)

Can you try this:

library(httr)

library(jsonlite)

library(logger)

library(fs)

library(xml2)

url <- 'https://files.isimip.org/api/v2'

paths <- list(

'ISIMIP3b/InputData/climate/atmosphere/bias-adjusted/global/daily/ssp585/GFDL-ESM4/gfdl-esm4_r1i1p1f1_w5e5_ssp585_tas_global_daily_2015_2020.nc',

'ISIMIP3b/InputData/climate/atmosphere/bias-adjusted/global/daily/ssp585/GFDL-ESM4/gfdl-esm4_r1i1p1f1_w5e5_ssp585_tas_global_daily_2021_2030.nc'

)

data <- list(

paths = paths,

operations = list(

list(

  operation = 'mask_country',

  country = 'aus',       # three letter code for, e.g. australia

  compute_mean = FALSE,  # optional: set to TRUE to get a time series of the field mean

  output_csv = FALSE     # optional: set to TRUE to get a CSV file instead of NetCDF

)

)

)

download_path <- 'download'

response <- POST(url, body = toJSON(data, auto_unbox = TRUE), encode = "json", content_type_json())

response_content <- content(response, "text", encoding = "UTF-8")

print(response_content)

job <- fromJSON(response_content, simplifyVector = TRUE)

while (job$status %in% c("queued", "started")) {

Sys.sleep(4)

job_response <- GET(job$job_url)

job_content <- content(job_response, "text", encoding = "UTF-8")

print(job_content)

job <- fromJSON(job_content, simplifyVector = TRUE)

}

if (job$status == "finished") {

zip_path <- file.path(download_path, job$file_name)

dir_create(path_dir(zip_path))

download.file(job$file_url, zip_path, mode = "wb")

out_path <- file.path(download_path, tools::file_path_sans_ext(basename(zip_path)))

dir_create(out_path)

unzip(zip_path, exdir = out_path)

}

The "logging" is a bit crude, but there is less boilerplate code. This script worked on my machine.

— Reply to this email directly, view it on GitHubhttps://github.com/ISI-MIP/isimip-files-api/issues/5#issuecomment-2145728378, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB6UJMUYQSBO7T5DVWJGEWTZFSPUXAVCNFSM6AAAAABIIIKAAOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBVG4ZDQMZXHA. You are receiving this because you were mentioned.Message ID: @.***>

jochenklar commented 6 months ago

Yes, I converted my python example notebooks to R.

You can use this one: https://git.rwth-aachen.de/nfdi4earth/pilotsincubatorlab/incubator/climate-impact-data-extraction/-/blob/main/R/2-mask-region-high-resolution.R, but if you work on 0.5 deg data, you don't need the first operation (cutout_bbox). You can provide a shape as GeoJSON or shapefile.