Our expand grid functions are not aware that target_end_date value is dependent on the reference_date value. They treat the values of each task ID as independent and produce the following erroneous grid of valid value combinations where the values of target_end_date are clearly inconsistent with the horizon.
1) It can validate erroneous combinations of values (eg a row with reference_date: 2023-10-14, horizon: 2 and reference_date: 2023-10-21). This can be mitigated by the additional optional test that checks the values of target_end_date with respect to horizon and reference_date.
2) MORE IMPORTANTLY: It can cause erroneous failures in required values checks. For example, here's a subset of the missing values the check for required values erroneously returns.
What's going on is, some quantile values are being submitted for optional horizon 2 as well as required horizon 1. The horizon 2 values have different target_end_date values (2023-10-28 rather than 2023-10-21 for horizon 1). Note as well that all values in target_end_date are configured as optional. The check is therefore detecting that data for the optional target_end_date value 2023-10-28 and optional horizon value 2 has been supplied but not for optional target_end_date value 2023-10-28 and required horizon value 1. It is therefore throwing an error even though a target_end_date value 2023-10-28 and required horizon value 1 is invalid for a reference date of 2023-10-14.
This is a tricky issue and I'm not 100% sure how to proceed. The easiest way I can think of is to be able to ignore task IDs in certain situations like these. Will likely need some time to fix though as it likely needs work on complex function across hubUtils and hubValidations. Keen on hearing thoughts!
The problem has been uncovered by trying to validate https://github.com/annakrystalli/FluSight-forecast-hub/blob/test-pr/model-output/UMass-trends_ensemble/2023-10-14-UMass-trends_ensemble.csv
Our expand grid functions are not aware that
target_end_datevalue is dependent on thereference_datevalue. They treat the values of each task ID as independent and produce the following erroneous grid of valid value combinations where the values oftarget_end_dateare clearly inconsistent with the horizon.Created on 2023-09-29 with reprex v2.0.2
This has 2 implications:
1) It can validate erroneous combinations of values (eg a row with reference_date: 2023-10-14, horizon: 2 and reference_date: 2023-10-21). This can be mitigated by the additional optional test that checks the values of
target_end_datewith respect tohorizonandreference_date. 2) MORE IMPORTANTLY: It can cause erroneous failures in required values checks. For example, here's a subset of the missing values the check for required values erroneously returns.Created on 2023-09-29 with reprex v2.0.2
What's going on is, some quantile values are being submitted for optional
horizon2 as well as requiredhorizon1. The horizon 2 values have differenttarget_end_datevalues (2023-10-28rather than2023-10-21for horizon 1). Note as well that all values intarget_end_dateare configured as optional. The check is therefore detecting that data for the optionaltarget_end_datevalue2023-10-28and optionalhorizonvalue2has been supplied but not for optionaltarget_end_datevalue2023-10-28and requiredhorizonvalue1. It is therefore throwing an error even though atarget_end_datevalue2023-10-28and requiredhorizonvalue1is invalid for a reference date of2023-10-14.Overall this is caused by the logic discussed in https://github.com/Infectious-Disease-Modeling-Hubs/hubValidations/issues/17 and encapsulated by the required values only model out submission template and arises from us not being able to define relationships between values in different columns.
Created on 2023-09-29 with reprex v2.0.2
This is a tricky issue and I'm not 100% sure how to proceed. The easiest way I can think of is to be able to ignore task IDs in certain situations like these. Will likely need some time to fix though as it likely needs work on complex function across
hubUtilsandhubValidations. Keen on hearing thoughts!