Create rule DDF00037: Among the ScheduledInstances in a ScheduleTimeline, at least one has a relationship to ScheduleTimelineExit

ASL-rmarshall commented 8 months ago

Create rule defined for #110

ASL-rmarshall commented 7 months ago

I don't think this is blocked - it should be possible to use the similar logic as for DDF00034 (see #146), but will need to figure out the logic to get a distinct list that indicates whether timelineExitId is populated or not.

BSnoeijerCD commented 7 months ago

@ASL-rmarshall : I agreed with you and tried this one this morning in two different ways including the logic of DDF00034. However, recognizing empty values is still an issue. See my comments in slack. I renamed the rule to DDF00037 in CORE.

Positive and Negative check data are uploaded to the corresponding directories.

ASL-rmarshall commented 7 months ago

@BSnoeijerCD I have been unable to find a combination of existing operations/operators that give the correct response. However, I have been able to use the new cross-class joining (in a dev environment) to execute a similar check as "Every ScheduleTimeline must have at least one ScheduleTimelineExit, and each ScheduleTimelineExit must be referenced by at least one ScheduledActivityInstance or ScheduledDecisionInstance":

Check:
  all:
    - name: SCHEDULEDACTIVITYINSTANCE.id
      operator: empty
    - name: SCHEDULEDDECISIONINSTANCE.id
      operator: empty
Match Datasets:
  - Name: SCHEDULETIMELINEEXIT
    Keys:
      - Left: id
        Right: parent_id
    Join Type: left
  - Name: SCHEDULEDACTIVITYINSTANCE
    Keys:
      - Left: id.SCHEDULETIMELINEEXIT
        Right: timelineExitId
    Join Type: left
  - Name: SCHEDULEDDECISIONINSTANCE
    Keys:
      - Left: id.SCHEDULETIMELINEEXIT
        Right: timelineExitId
    Join Type: left

This left-joins ScheduleTimeline to ScheduleTimeExit on ScheduleTimeline.id = ScheduleTimelineExit.parent_id (merges in the valid exits for the ScheduleTimeline) and then left-joins this combined dataset to both ScheduledActivityInstance (on ScheduleTimelineExit.id = ScheduledActivityInstance.timelineExitId) and ScheduledDecisionInstance (on ScheduleTimelineExit.id = ScheduledDecisionInstance.timelineExitId). It then reports where both ScheduledActivityInstance.id and ScheduledDecisionInstance.id in the final merged dataset are empty, which will happen if:

There is no exit defined for the ScheduleTimeline
The is no ScheduledActivityInstance or ScheduledDecisionInstance that uses a ScheduleTimelineExit defined for the ScheduleTimeline

If preferred, this could be split into two separate rules:

"Every ScheduleTimeline must have at least one ScheduleTimelineExit"
"Each ScheduleTimelineExit defined for a ScheduleTimeline must be referenced by at least one ScheduledActivityInstance or ScheduledDecisionInstance"

BSnoeijerCD commented 7 months ago

Hi @ASL-rmarshall , Nice! How do you check that a scheduleTimeline has at least 1 exit in this way? If the scheduleTimeline has no exits it will not be in the scheduleTimelineExit class. I added a new rule description to the RuleRequirements sheet: Chk0103 which matches this quite well. We can also check that the parentId of the scheduletimelineExit matches the parentId of the scheduledActivityInstance and scheduledDecisionInstance. Then it is completely covered. Did you already add this to the CORE engine?

ASL-rmarshall commented 7 months ago

@BSnoeijerCD To check that a ScheduleTimeline has at least one ScheduleTimelineExit, you'd do the following:

Check:
  all:
    - name: SCHEDULETIMELINEEXIT.id
      operator: empty
Match Datasets:
  - Name: SCHEDULETIMELINEEXIT
    Keys:
      - Left: id
        Right: parent_id
    Join Type: left
Scope:
  Entities:
    Include: ScheduleTimeline

The Match Datasets produces a dataset with all records for ScheduleTimeline joined with any records from ScheduleTimelineExit where ScheduleTimelineExit.parent_id = ScheduleTimeline.id. If there is no matching record in ScheduleTimelineExit, the SCHEDULETIMELINEEXIT.id column will be blank and the record (for the ScheduleTimeline) will be reported.

The left join functionality is not yet available in the CORE engine (the update is still waiting for Gerry's review), so I have not yet added to the Rule Editor any rule definitions that rely on this functionality.

BSnoeijerCD commented 7 months ago

@BSnoeijerCD To check that a ScheduleTimeline has at least one ScheduleTimelineExit, you'd do the following:
Check:
  all:
    - name: SCHEDULETIMELINEEXIT.id
      operator: empty
Match Datasets:
  - Name: SCHEDULETIMELINEEXIT
    Keys:
      - Left: id
        Right: parent_id
    Join Type: left
Scope:
  Entities:
    Include: ScheduleTimeline
The Match Datasets produces a dataset with all records for ScheduleTimeline joined with any records from ScheduleTimelineExit where ScheduleTimelineExit.parent_id = ScheduleTimeline.id. If there is no matching record in ScheduleTimelineExit, the SCHEDULETIMELINEEXIT.id column will be blank and the record (for the ScheduleTimeline) will be reported.

The left join functionality is not yet available in the CORE engine (the update is still waiting for Gerry's review), so I have not yet added to the Rule Editor any rule definitions that rely on this functionality.

Thank you. That will work for the rule defined by Diane: #159

ASL-rmarshall commented 7 months ago

Re-marked this as blocked because it does require joining by non-matching keys.

ASL-rmarshall commented 3 months ago

Saving draft POC rule definition before overwriting for DDF4:

Authorities:
  - Organization: 'CDISC'
    Standards:
      - Name: USDM
        References:
          - Citations:
              - Cited Guidance: 'USDM 2.6'
                Document: 'USDM v2.6'
            Origin: USDM Conformance Rules
            Rule Identifier:
              Id: 'DDF00037'
              Version: '1'
            Validator Rule Message: 'Each ScheduledTimeline must have at least one exit, and
              therefore at least one ScheduledActivityInstance linked to a
              ScheduledTimelineExit.'
            Version: '1.0'
        Version: '3.0'
Check:
  all:
    - name: timelineExitId
      operator: equal_to
      value: Null
    - name: timelineExitId
      operator: has_same_values
Core:
  Status: Draft
  Version: '1'
Description: 'Among the ScheduledActivityInstances of ScheduleTimeline, at least
  one must have a relationship to a ScheduledTimelineExit.'
Executability: Fully Executable
Outcome:
  Message: 'The ScheduledTimeline does not include any ScheduledActivityInstance
    related to a ScheduledTimelineExit.'
  Output Variables:
    - parent_entity
    - parent_id
    - parent_rel
    - id
    - timelineExitId
Rule Type: Record Data
Scope:
  Entities:
    Include:
      - ScheduledActivityInstance
Sensitivity: Record

cdisc-org / ddf-core-poc

Create rule DDF00037: Among the ScheduledInstances in a ScheduleTimeline, at least one has a relationship to ScheduleTimelineExit #123