search

hubverse-org / hubValidations

Testing framework for hubverse hub validations
https://hubverse-org.github.io/hubValidations/
Other
1 stars 4 forks source link

Column type validation issue on date columns in parquet submissions #118

Closed ruarai closed 3 weeks ago

ruarai commented 3 weeks ago

Our hub has run into an issue with the column type checking. It seems submitting a parquet file with date-times in a date column will raise an error in check_tbl_col_types.R:

[col_types]: EXEC ERROR: Error in purrr::map_chr(tbl, ~if (inherits(.x,
"numeric")) [ : ℹ In index: 1. ℹ With name: forecast_date. Caused by error: !
Result must be length 1, not 2.

This can be resolved of course by making sure the relevant columns are of type date rather than date-time (but this is not super obvious from this error to our submitters, at least).

The error seems to arise as class(x) for a date-time vector will return a vector of two values, i.e.:

> class(Sys.time())
[1] "POSIXct" "POSIXt"

I was going to submit a PR for a simple fix to just return the first element from class(.x) - but worried this could cause issues later?

Sorry I don't have a reprex for this one.

annakrystalli commented 3 weeks ago

Thanks for the bug report @ruarai !

Now sorted and we'll also be adding a section in our docs regarding this. See https://github.com/hubverse-org/hubDocs/issues/177 and https://github.com/orgs/hubverse-org/discussions/31