Closed vincentraymond-ua closed 1 year ago
Merging #333 (ed71083) into master (f1f3c15) will increase coverage by
0.52%
. The diff coverage is0.00%
.:exclamation: Current head ed71083 differs from pull request most recent head f70d59c. Consider uploading reports for the commit f70d59c to get more accurate results
@@ Coverage Diff @@
## master #333 +/- ##
==========================================
+ Coverage 33.94% 34.47% +0.52%
==========================================
Files 114 114
Lines 22752 22661 -91
==========================================
+ Hits 7724 7813 +89
+ Misses 15028 14848 -180
Impacted Files | Coverage Δ | |
---|---|---|
...omates/program_analysis/JSON2GroMEt/json2gromet.py | 0.00% <0.00%> (ø) |
|
...mates/program_analysis/CAST2GrFN/model/cast/var.py | 68.65% <0.00%> (-1.50%) |
:arrow_down: |
...ogram_analysis/CAST2GrFN/model/cast/scalar_type.py | 44.73% <0.00%> (-1.42%) |
:arrow_down: |
...alysis/CAST2GrFN/ann_cast/variable_version_pass.py | 80.39% <0.00%> (-0.14%) |
:arrow_down: |
...lysis/CAST2GrFN/visitors/cast_to_agraph_visitor.py | 16.81% <0.00%> (+0.39%) |
:arrow_up: |
...am_analysis/CAST2GrFN/ann_cast/id_collapse_pass.py | 76.47% <0.00%> (+0.74%) |
:arrow_up: |
...lysis/CAST2GrFN/ann_cast/cast_to_annotated_cast.py | 77.14% <0.00%> (+0.75%) |
:arrow_up: |
...ates/program_analysis/PyAST2CAST/py_ast_to_cast.py | 13.81% <0.00%> (+1.37%) |
:arrow_up: |
...gram_analysis/CAST2GrFN/ann_cast/annotated_cast.py | 69.19% <0.00%> (+10.99%) |
:arrow_up: |
...rogram_analysis/CAST2GrFN/model/cast/source_ref.py | 86.07% <0.00%> (+11.39%) |
:arrow_up: |
... and 2 more |
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more
I'm concluding that the specs from files like https://github.com/ml4ai/automates-v2/blob/master/docs/source/gromet_FN_v0.1.5.yaml are really only for human consumption. They are there so that others can manually write code to process the format. That seems unfortunate. If correct, I would remove the discussion of code generation.
Hi @kwalcock : Thanks for your comment -- I want to make sure I understand.
Some background (which you may already know, but in case I didn't review with you...): It is the case that we currently use the swagger-codegen generated data model (actually three: CAST, GrometFN, and Metadata) in the Program Analysis (PA) pipeline. The auto-generated code ends up being directly versioned within the automates repo in these locations (all under <automates_root>/automates/
):
.../program_analysis/CAST2GrFN/model/
(at some point this will be moved to a better location).../gromet/fn/
.../gromet/metadata/
Inserting the swagger-codegen generated code into these respective locations does require some post-processing to fix paths. I created a script that manages this:
Now, I believe you are referring to the "preamble" text that is at the start of each of the swagger model yaml files that describes how to use swagger-codegen to generate the source code class structure. These instructions were originally created as a reminder to ourselves for how these are generated (the original version of these instructions were created by Daniel Dicken when he first created the swagger spec). Until I had created the above script, I would perform the generation and insertion manually. Since creating the script, I have maintained the instructions at the top for each model spec, in case others who might wish to programmatically interact with the data models but not using automates want to do so. But within automates, we use the script described above to maintain the data model.
@vincentraymond-ua maintains a serializer/deserializer for the internal automates data structures based on the GrometFN (in particular, the GrometFNModel and GrometFNModuleCollection) and Metadata, where the export target is a particular JSON format that we call the "Gromet FN relational database schema". This JSON format is governed by ASKEM "gromet schema" conventions that don't exactly match the swagger model, which is why we maintain the serializer/deserializer and don't just directly use the build-in .to_json
and load
. It is a bummer that we can't just directly do things that way.
So, this PR is about Vincent updating the serializer/deserializer to support the latest changes to the Metadata data model (v0.1.5).
Now that we have the explicit internal data structure (using the GrometFN and Metadata data structures derived from the swagger models), along with our support for the serializer/deserializer, I would recommend using those to programmatically interact with the Gromet FN model, as it is generated by the Program Analysis function network extraction pipeline. Justin Lieffers is also currently developing a graph database representation which is suited for querying and manipulating function networks, and this is created directly from the Gromet FN model in the automates PA pipeline. But I think the instructions for how the data models are generated directly using swagger-codegen might also still be of use to some.
Does this help clarify what you are responding to in your comment? I do think that the way we have things documented is leading to confusion. Given the above context, what would your recommend for making this clear? (... And are there other issues we should consider?)
Thanks!
Updates the json_to_gromet importer to support the v0.1.5 format for both Gromet FN and Gromet Metadata.
Additionally, updates the process_file_system function to be more functional by returning the Gromet module collection and disabling writing to file unless write_to_file argument is True.