Refactor capgen branch

[gold2718]

Refactor capgen branch

• Add new runtime info routine, ccpp_physics_suite_schemes
• Convert framework options to DDT with methods (runtime environment).
• Convert shebangs to python3
• Change ccpp_error_flag to ccpp_error_code
• Add polymorphic variable property
• Remove optional variable property
• Introduce FortranVar to allow Fortran to have undocumented optional variables
• Fix hang in datatable write plus catch more bad units
• Split var_props.py from metavar.py to allow independent development
• Added VarCompatObject to hold compatibility and transform information about a pair of Var objects. This object looks for compatible differences in units, kind, or dimensions.
• Created unit tests for variable compatibility.
• Move PrettyElementTree to xml_tools
• Refactor ccpp_suite to separate out SuiteObjects
• Created /test/run_tests.sh to run all capgen tests.

User interface changes?: Yes

• Optional keyword removed from metadata. Optional Fortran variables should be omitted from metadata tables
• --kind-type argument replaces --kind-phys and takes a list of model kinds.

Fixes: [Github issue #s] addresses #329 addresses #378 addresses #403 fixes #408

Testing:

• all doctests
• capgen_test
• advection_test
• unit tests in test/unit_tests

[gold2718]

From our previous conversation, my understanding was that optional Fortran variables may be omitted from metadata tables, but they can be present. It's a hardcoded choice which flavor of a scheme to run.

Oops, bad communication. capgen does work this way, I added a test in capgen_test to verify that an optional Fortran variable which is included in the metadata table is handled properly.

[gold2718]

Do you think the improved error handler will be ready before we transition the UFS to capgen?

I have not prioritized this work because it would delay unification readiness. We need to re-evaluate the schedule if you want to add this.

[gold2718]

I think at least one more Python file needs the Python3 shebang:

Yep, I already found this by testing on a different machine (it turns out that the machine I was using had python ==> python3 so I did not see the problem. Unfortunately, in trying to clean up all this test stuff, I ended up touching a bunch of files so the PR grew.

[gold2718]

In general: This PR is extremely difficult if not impossible to review in a reasonable amount of time.

I agree with this comment and am happy to try and make shorter PRs. However, if we agree that I will do this, I need to go in and re-estimate all the task hours as a PR does take me a significant amount of time. Your call.

[climbfuji]

In general: This PR is extremely difficult if not impossible to review in a reasonable amount of time.

I agree with this comment and am happy to try and make shorter PRs. However, if we agree that I will do this, I need to go in and re-estimate all the task hours as a PR does take me a significant amount of time. Your call.

It's a judgement call, I think we need to be pragmatic. Maybe along the lines "I can split this up easily without spending a lot of time, so let's to it" and "This would be a lot of overhead, let's keep it together".

[climbfuji]

I ran the tests with your latest commit bbb668c on my Mac, and all tests passed.

[climbfuji]

@gold2718 I am ok with this PR, do you want to make more changes or should I approve and merge? Thanks.

[gold2718]

@gold2718 I am ok with this PR, do you want to make more changes or should I approve and merge? Thanks.

From my point of view, it is okay to merge if you are comfortable with the current state. I can try to work in future PRs.