AUC was still using csv data and needed to switch to json. The json records contain nil values which causes, strip, gsub, prepend, etc. to fail. This adds new versions of those macros that guard against that. We are also seeing some errors with the values that get into cho_edm_type and cho_has_type so this treats those values as a controlled vocabulary and raises an error when a value not in the list is found. I'm planning a follow up PR to add nil-safe versions of all of the rest of the macros in https://github.com/traject/traject/blob/main/lib/traject/macros/transformation.rb. But this will require moving some things around and updating all of the configs that use them. Also planning another PR to remove all of the xml and csv macros that we no longer need.
How was this change tested?
Local transform, and new tests.
Which documentation and/or configurations were updated?
Why was this change made?
AUC was still using csv data and needed to switch to json. The json records contain nil values which causes,
strip,gsub,prepend, etc. to fail. This adds new versions of those macros that guard against that. We are also seeing some errors with the values that get intocho_edm_typeandcho_has_typeso this treats those values as a controlled vocabulary and raises an error when a value not in the list is found. I'm planning a follow up PR to add nil-safe versions of all of the rest of the macros in https://github.com/traject/traject/blob/main/lib/traject/macros/transformation.rb. But this will require moving some things around and updating all of the configs that use them. Also planning another PR to remove all of the xml and csv macros that we no longer need.How was this change tested?
Local transform, and new tests.
Which documentation and/or configurations were updated?
n/a