USEPA / ccdR

Utilities for Interacting with the CTX APIs in R without prior API knowledge. All data is also available on the CompTox Chemical Dashboard (CCD) https://comptox.epa.gov/dashboard/
https://api-ccte.epa.gov/docs/
GNU General Public License v3.0
2 stars 0 forks source link

get_bioactivity_details unnest_wider on line 96 fails when mc6Param is NA #52

Closed cthunes closed 1 month ago

cthunes commented 2 months ago

Can replicate this with m4id = 739820

Error in `tidyr::unnest_wider()` at ccdR/R/bioactivity-APIs.R:96:6:
ℹ In column: `mc6Param`.
ℹ In row: 1.
Caused by error:
! Can't unnest elements with missing names.
ℹ Supply `names_sep` to generate automatic names.

mc6Param is NA when the curve had no associated flags. Not sure if the better solution is to:

  1. only unnest_wider when the mc#Param fields are not NA, or
  2. replace mc6Param with default columns manually, with flag = NA and mc6MthdId = NA

Thoughts on if ccdR should just prevent the error, and then tcpl can instead handle this? @brown-jason

I don't think this could affect the other mc#Param fields, since those should all always contain some parameters, so this is probably just an mc6Param issue.

cthunes commented 2 months ago

The fix is not compatible for responses with length>1 (ie, using SPID, AEID, or DTXSID) in almost any case. We need to find a way to standardize the format of the table with different lengths.

For context, any response from the API with only one row is automatically put into a simple list, and we are converting it to a table. This is most common with a m4id request.

Any response from the API with multiple rows (usually SPID, AEID, and DTXSID) are automatically put into a table format by jsonlite, but the format for nested list columns are slightly different than our conversion we do for m4id. jsonlite puts these into a nested data.table form, while our conversion puts those columns in a nested list form.

The fix for this ticket works for nested list form but it will break for nested data.table form.

Error in !sapply(res[which(names(res) %in% param_cols)], is.na) : 
  invalid argument type

Reopening this ticket so we can do one of the following as a resolution:

  1. fix the way we convert to a table to match jsonlite and then change this fix accordingly,
  2. change the way jsonlite puts nested columns in table form to be in list form, or
  3. change this fix to be compatible with either nested form