harmo_process() error(s) using Rmonize_examples() objects

[twey2]

Running harmo_process() with Rmonize_examples() objects produces an ERROR in an algorithm that is written correctly in the DPE and produces some other issues. (Package versions: Rmonize_1.1.0.9106, madshapR_1.1.0.9111, fabR_2.1.0.9103)

Running the following script:

harmonized_dossier <- harmo_process(
  dossier,
  dataschema,
  data_proc_elem,
  harmonized_col_dataset = 'adm_study_id')

Produces the following processing error in dataset_study5:

The Mlstr_harmo::algorithm appears to be written correctly. The input variable appears to be formatted as a mixed variable (categorical and non-categorical values present).

The error is not reproduced if the algorithm script is run on the input variable directly in R.

In addition, the harmonization process incorrectly prints a message about undetermined statuses (there are none).

There are no warning messages about the processing error in dataset_study5. In the harmonized dossier, dataset_study5 is (correctly) empty.

[GuiFabre]

Hello @twey2 the bug has been corrected. The problem came from madshapR data_dict_zap_dataset() function which aims at separating the data dictionary from its associated dataset. An interesting case come with a factor variable (or haven_labelled variable in haven). If the the variable remains factor, then (technically) the data dictionary for its particular variable remains (the categories are levels/labels). If not, the column is a regular column, coerced with its associated valueType (no more category/level). Both of the scenarios has its pros and cons.

Originally the default behavior was to drop the levels, then it adressed some issues in madshapR (grouping variables were cast to factor, then unfactored somewhere else in the function summarize()). So, to avoid that, the default behavior was to keep them (a factor is intrinsic to a variable, whether there is a dictionary or not).

Then it consequently adressed other issues in Rmonize (variable to be harmonize kept their factor status, so it is harder to work with it, such as with floor(var) function). But the user shouldn't be bothered with such problems (a factor is intrinsic to R objects and not the values, and the harmonization should work whether the column is a factor or not. Only the value matters)

So to make peace with both scenarios, a parameter zap_factor has been added (working similarly with stringAsFactor = TRUE/FALSE) which preserves (by defaut) the factor, make it work everywhere in both packages ;

and allows to drop them (internally) in the function harmo process.

[twey2]

This error has been fixed in my most recent check. (Package versions Rmonize_1.1.0.9107, madshapR_1.1.0.9115, fabR_2.1.0.9103) But, lubridate must be loaded otherwise there is an error in study5 sdc_age_m algorithm. Can be fixed by changing algorithm to use "lubridate::years".

[GuiFabre]

Yes, lubridate must be loaded to make the dpe work. This is why it appears in the Suggests, and in 'library(lubridate)' is explicitly called in the example.

thanks !

[twey2]

I think it would be better to change "years" to "lubridate::years" in the algorithm anyway? The algorithm does already use "lubridate::interval".

[GuiFabre]

yes, sure !