search

nasa / fpp

F Prime Prime: A modeling language for F Prime
https://fprime.jpl.nasa.gov
Apache License 2.0
41 stars 31 forks source link

Add fpp-to-json execution test with nasa/fprime devel #465

Closed mosa11aei closed 1 week ago

mosa11aei commented 2 weeks ago

I preface by saying that this test will most likely be fairly controversial, and so I am open to a discussion regarding this test. However, I believe there are a few good reasons as to why an fpp-to-json analysis test of the entire nasa/fprime repo is worthwhile.

  1. The nasa/fprime repo is a complete model. The RPI and Ref deployments in it provide sample projects which utilize the other fprime-dependent fpp definitions. The expectation is that if the entire nasa/fprime repo cannot be analyzed, then something behaviorally is wrong, either on the F Prime side of things, or the fpp-to-json tool.
  2. The repository provides a gold mine of testable semantic patterns and behavioral patterns that would take quite a long time to capture in individual test cases. For example, patterned connection graphs are dependent on Fw (the F Prime framework), which would need to be at least symbolically replicated (i.e., the symbols exist but are gibberish), or (even better) with properly defined symbols. So at that point, why not just go direct to the source? The added benefit here is that the Java JSON bitmap also stays up-to-date.
  3. The natural question which I had was "what if we just test a sample deployment"? In that case, the deployment is still not model complete; it depends on the definitions and elements in nasa/fprime.
  4. The (other) question is on what branch should this happen? I recommend that it is kept on the "devel" branch of nasa/fprime. It can be added as a merge workflow whenever anything gets pushed into "devel", but since that branch includes all of the future F Prime release features, it seems like the minimum viable set of analysis material. More branches can be tested, but it would be redundant.
  5. Test cases, especially functional test cases, which are what fpp-to-json and the other tools utilize, are important to ensure that the output analysis matches whatever change occurs. However, in the case of analyzing something as monstrous as the entire nasa/fprime repository, what seems to be a better goal is to verify whether analysis can be done on it. I see very little real-world use cases to analyze the full nasa/fprime repository, and so if the test happens on the fpp-side of things and passes, then errors enountered with the output of fpp-to-json can be caught farther down the pipeline of usage.

I think the biggest concern with this test is that "nasa/fprime depends on fpp, so it seems cyclical". However, the actual F Prime repository does not depend on the analysis output from fpp-to-json. Nor do we necessarily care about the C++ outputs/templates in our analysis. So I believe this test is still valid and worthwhile to include.

As for future integrations of this test, it is lightweight enough such that it just uses a simple bash script, and only depends on git, which can be assumed is installed on the testing system. It could be, in the future added to CI, since the execution time is ~20sec on an x86 Intel MacBook Pro with 64gb RAM. Even on a much much less powerful system, it will most likely take less time to execute than many of the other CI tasks that fpp has, which can take upwards of a few hours to fully complete. Additionally, it may be possible to implement a similar test for the other fpp-to-x tools. However, I believe those have larger caveats to consider than fpp-to-json.

Ties to issue https://github.com/nasa/fpp/issues/464

mosa11aei commented 2 weeks ago

After a discussion with @bocchino, it was thought that this script may be better suited to be a CI task within F Prime itself. This is because it will ensure that F Prime is consistently analyzable with every pull request conducted.

This broadens the discussion to @LeStarch @thomas-bc.

thomas-bc commented 1 week ago

@mosa11aei is the primary intent to test the validity of the models in nasa/fprime, or to test the fpp-to-json utility itself?

bocchino commented 1 week ago

is the primary intent to test the validity of the models in nasa/fprime, or to test the fpp-to-json utility itself?

The goal is to do functional testing of the FPP tool on F Prime models, similarly to https://github.com/nasa/fprime/tree/devel/FppTest. Perhaps the fpp-to-json tests could go here.

bocchino commented 1 week ago

We could also break out the F Prime-FPP functional testing into a separate repo in fprime-community. In this case we would need to factor FppTest into a separate project that depends on F Prime.