Develop a generic reporting feature on RITE (with Galois's help)

[pauls4GE]

Questions:

• What does a generic reporting tool support? Any random standard or just DO178C and ARP4754?
  • If we want to support any random standard, will first need to figure out a way to represent any given standard to a normalized form
  • How to normalize standards that do not follow the hierarchical (process/table -> objectives -> evidence) structure ?
• How should the tool work? Should it be a plan-based tool like the current tools? Or some other technique?
• Do we need to go to the blackboard and redesign a completely new approach by using experience in building the non-generic tools?

[baoluomeng]

Questions:

• What does a generic reporting tool support? Any random standard or just DO178C and ARP4754?

Let's aim for a generic reporting feature (arbitrary standards).

• If we want to support any random standard, will first need to figure out a way to represent any given standard to a normalized form
• How to normalize standards that do not follow the hierarchical (process/table -> objectives -> evidence) structure ?

You will need to create a GUI to allow user to enter hierarchical informations. The normalization process is done by users. We can ignore standards that do not follow the hierarchical structure. It looks like the major standards all follow a hierarchical structure in some way.

• How should the tool work? Should it be a plan-based tool like the current tools? Or some other technique?

Start with developing a GUI using JavaFX to enter PSAC. A good reference is issue: #139.

• Do we need to go to the blackboard and redesign a completely new approach by using experience in building the non-generic tools?

Yes, we can.

[pauls4GE]

Questions:

• What does a generic reporting tool support? Any random standard or just DO178C and ARP4754?

Let's aim for a generic reporting feature (arbitrary standards).

• If we want to support any random standard, will first need to figure out a way to represent any given standard to a normalized form
• How to normalize standards that do not follow the hierarchical (process/table -> objectives -> evidence) structure ?

You will need to create a GUI to allow user to enter hierarchical informations. The normalization process is done by users. We can ignore standards that do not follow the hierarchical structure. It looks like the major standards all follow a hierarchical structure in some way.

There are two major challenges here:

1. How do we let users encode the logic for "How" an objective must be met? The current tools have this hardcoded in Java, but for a generic tool, we cannot hardcode this.
2. How do we design the normalization approach so that queries can be automatically generated to fetch data related to an objective? Currently, the tool relies on a set of queries being created manually for a given project ontology.

• How should the tool work? Should it be a plan-based tool like the current tools? Or some other technique?

Start with developing a GUI using JavaFX to enter PSAC. A good reference is issue: #139.

Designing the GUI is an easy task and not a hurdle at all. The challenge lies in developing the generic backend. Before we start with a GUI, we need to figure out how the backend will work. I believe the current GUI will be sufficient. Integrating the floating JAVAFx views into Eclipse views can be achieved after we have designed the initial version.

• Do we need to go to the blackboard and redesign a completely new approach by using experience in building the non-generic tools?

Yes, we can.

[baoluomeng]

Questions:

• What does a generic reporting tool support? Any random standard or just DO178C and ARP4754?

Let's aim for a generic reporting feature (arbitrary standards).

• If we want to support any random standard, will first need to figure out a way to represent any given standard to a normalized form
• How to normalize standards that do not follow the hierarchical (process/table -> objectives -> evidence) structure ?

You will need to create a GUI to allow user to enter hierarchical informations. The normalization process is done by users. We can ignore standards that do not follow the hierarchical structure. It looks like the major standards all follow a hierarchical structure in some way.

There are two major challenges here:

1. How do we let users encode the logic for "How" an objective must be met? The current tools have this hardcoded in Java, but for a generic tool, we cannot hardcode this.

Say for "APR4754A, Objective 4.1 requirements are correct and complete", you can customize a new field "evidence" saying that "requirements have been reviewed and have gone through formal analysis" (text can be customized). You can then associate this evidence field with the objective.

2. How do we design the normalization approach so that queries can be automatically generated to fetch data related to an objective? Currently, the tool relies on a set of queries being created manually for a given project ontology.

The queries can be supplied by users. You can associate those queries with the objective on the GUI.

• How should the tool work? Should it be a plan-based tool like the current tools? Or some other technique?

Start with developing a GUI using JavaFX to enter PSAC. A good reference is issue: #139.

Designing the GUI is an easy task and not a hurdle at all. The challenge lies in developing the generic backend. Before we start with a GUI, we need to figure out how the backend will work. I believe the current GUI will be sufficient. Integrating the floating JAVAFx views into Eclipse views can be achieved after we have designed the initial version.

I really think you should start with creating native views using JavaFX. The code is already there. Kicking the can down the road would most likely end up with not implementing it.

• Do we need to go to the blackboard and redesign a completely new approach by using experience in building the non-generic tools?

Yes, we can.

[pauls4GE]

Questions:

• What does a generic reporting tool support? Any random standard or just DO178C and ARP4754?

Let's aim for a generic reporting feature (arbitrary standards).

• If we want to support any random standard, will first need to figure out a way to represent any given standard to a normalized form
• How to normalize standards that do not follow the hierarchical (process/table -> objectives -> evidence) structure ?

You will need to create a GUI to allow user to enter hierarchical informations. The normalization process is done by users. We can ignore standards that do not follow the hierarchical structure. It looks like the major standards all follow a hierarchical structure in some way.

There are two major challenges here:

1. How do we let users encode the logic for "How" an objective must be met? The current tools have this hardcoded in Java, but for a generic tool, we cannot hardcode this.

Say for "APR4754A, Objective 4.1 requirements are correct and complete", you can customize a new field "evidence" saying that "requirements have been reviewed and have gone through formal analysis" (text can be customized). You can then associate this evidence field with the objective.

Not all objectives are this straightforward. For example, to check objective 2.2 (see this), one needs to find traceability between multiple classes. So just connecting an evidence class to an objective is not always sufficient. We need to create a way for users to be able to encode such non-trivial logic formally so that our backend can "interpret" that logic and perform what is necessary. So this also creates an additional question of how do we create code that can work for arbitrary logic when the standards are unknown.

2. How do we design the normalization approach so that queries can be automatically generated to fetch data related to an objective? Currently, the tool relies on a set of queries being created manually for a given project ontology.

The queries can be supplied by users. You can associate those queries with the objective on the GUI.

• How should the tool work? Should it be a plan-based tool like the current tools? Or some other technique?

Start with developing a GUI using JavaFX to enter PSAC. A good reference is issue: #139.

Designing the GUI is an easy task and not a hurdle at all. The challenge lies in developing the generic backend. Before we start with a GUI, we need to figure out how the backend will work. I believe the current GUI will be sufficient. Integrating the floating JAVAFx views into Eclipse views can be achieved after we have designed the initial version.

I really think you should start with creating native views using JavaFX. The code is already there. Kicking the can down the road would most likely end up with not implementing it.

Sure. When we start designing the GUI for this feature, we should start doing this. What I meant was we need to first decide on the backend and how to genericize the approach before working on the GUI. The GUI will not work without the backend.

• Do we need to go to the blackboard and redesign a completely new approach by using experience in building the non-generic tools?

Yes, we can. I think the current tools are not designed to be made generic. So we have to go back to the drawing board.

[baoluomeng]

Not all objectives are this straightforward. For example, to check objective 2.2 (see this), one needs to find traceability between multiple classes. So just connecting an evidence class to an objective is not always sufficient. We need to create a way for users to be able to encode such non-trivial logic formally so that our backend can "interpret" that logic and perform what is necessary. So this also creates an additional question of how do we create code that can work for arbitrary logic when the standards are unknown.

You may want to gradually increase the scope. Starting with something simple, then add more complexity. Based on your experience, you can list possible situations and find ways to tackle them. We can brainstorm solutions.

[jshackleton]

After spending some time with the ARP-4754 compliance tool, here are my thoughts:

• I agree that the logic that captures the results of a compliance process (pass/fail/partial data/no data) can be captured in a query, replacing the java. Specifically, SHACL was designed for this kind of complex traversal of connections and computing custom results.
• An overlay data model then would include a set of SHACL queries/operations that hook into the generic reporting tool. A config file could define the top-level queries for each compliance process, and the expected results (for now) are a set of enumeration values (pass/fail/partial data/no data).
• I also agree that auto-generating node groups and queries would be a huge improvement, but I don't have sense of how that might work yet.

My understanding is that SHACL support is already available in RACK and has been exposed to the CLI in the latest version. For now I don't see a need to update the GUI. Once we encounter results that require more customization that is different that the pass/fail type options of ARP-4754 and DO-178, then we can revisit the GUI.

If you think this is the right path forward, I can begin experimenting with this query approach in my own branch.

[pauls4GE]

After spending some time with the ARP-4754 compliance tool, here are my thoughts:

• I agree that the logic that captures the results of a compliance process (pass/fail/partial data/no data) can be captured in a query, replacing the java. Specifically, SHACL was designed for this kind of complex traversal of connections and computing custom results.
• An overlay data model then would include a set of SHACL queries/operations that hook into the generic reporting tool. A config file could define the top-level queries for each compliance process, and the expected results (for now) are a set of enumeration values (pass/fail/partial data/no data).
• I also agree that auto-generating node groups and queries would be a huge improvement, but I don't have sense of how that might work yet.

My understanding is that SHACL support is already available in RACK and has been exposed to the CLI in the latest version. For now I don't see a need to update the GUI. Once we encounter results that require more customization that is different that the pass/fail type options of ARP-4754 and DO-178, then we can revisit the GUI.

If you think this is the right path forward, I can begin experimenting with this query approach in my own branch.

Sure. It will be interesting to see how SHACL can be integrated with the reporting tool.