REF/ENH: standalone solution parser and rewrite of input/output format support

[klauer]

If this breaks your usage of blark, let me know!

This is a pretty large refactor/enhancement, comments/ideas/complaints are welcome.

Goals

Primarily: to close https://github.com/klauer/blark/issues/53

More specifically (caveat: some goals may be walked back; some may not be written here yet) -

• Continue to support the current input formats (plain text .st files, and TwinCAT source code files)
• Remove dependency on pytmc (for code loading, the dependency store, and everything else).
• Require lxml for project parsing.
• Make file loading more easily customizable, with the option to load/save arbitrary file formats so long as you can provide an adapter (or for command-line input, code with metadata described by a JSON schema).
• Allow for mapping of blark input line numbers back to source code files
• Allow for in-place code rewriting of source code files (-> blark format --write my_pou.TcPOU)
• Redo blark.parse() and similar API functions to return a consistent dataclass type. This will group together the source code, a map of line numbers, the lark.Tree, and an option to easily transform it into blark.transform.SourceCode dataclasses.
• Redo the blark parse CLI as it's pretty bad in master
• Redo the blark format CLI to support these input/output formats and more closely mirror how blark parse will work

At some point after this PR, I might consider:

• Adding blark scrape - a tool to scrape codebases and dump out the requested information.
• Rewriting pytmc pragmalint (a tool for linting a specific set of pragmas in our TwinCAT projects) around the above. Piping results from blark scrape --rule PRAGMA into pytmc pragmalint could help refer back to actual line numbers in files. Our current linting method (as exhibited here is off by a few lines due to pytmc throwing away line number information)

Status

• The solution parser looks to be working mostly fine for now
  • It needs more comments/docstrings like everything else here
  • GlobalTextList items are causing some failures (example)
• Dependency store might be OK, needs looking into
  • Test suite addition of a 'blark twincat root' is here: https://github.com/klauer/blark-twincat-root-example
  • The idea being you can load up libraries/projects and relate them to one another (maybe eventually paving the way for a TwinCAT LSP)
• Need to circle back and add more tests. Coverage is likely abysmal
• Need to check if summary classes and sphinx domain need adjusting

Example

Take, for example, a TwinCAT POU like FB_DummyHA.TcPOU. It's broken up into multiple XML sections, which we have no interest in adding grammar for.

In blark master:

• We require that this file get flattened into something like this:
  FB_DummyHA blark-parse-able source

FUNCTION_BLOCK FB_DummyHA IMPLEMENTS I_HigherAuthority, I_LowerAuthority
VAR_INPUT
    ReqAcknowledged : BOOL := TRUE; // Use to control when the HA CHeckRequest Returns TRUE
END_VAR
VAR_OUTPUT
END_VAR
VAR
END_VAR

END_FUNCTION_BLOCK

METHOD CheckRequest : BOOL
VAR_INPUT
        nReqID  : DWORD;
END_VAR
CheckRequest := ReqAcknowledged;
END_METHOD

// <Arbiter Internal>
// Elevates the arbitrated BP set to something above.
// Could be another arbiter, or a BP requester/ IO,
// or an FB that locks in a specific portion of the BP set.
METHOD ElevateRequest : BOOL
VAR_INPUT
    HigherAuthority : I_HigherAuthority;
END_VAR

END_METHOD

METHOD RemoveRequest : BOOL
VAR_INPUT
        (*StateID to remove*)
        nReqID  : DWORD;
END_VAR
RemoveRequest := TRUE;
END_METHOD

METHOD RequestBP : BOOL
VAR_INPUT
        (*StateID of state requesting beam parameter set*)
        nReqID  : DWORD;
        (*Requested beam params*)
        stReqBP : ST_BeamParams;
END_VAR
RequestBP := TRUE;
END_METHOD

PROPERTY nLowerAuthorityID : DWORD
VAR
END_VAR
nLowerAuthorityID := PMPS_GVL.EXCLUDED_ASSERTION_ID;
END_PROPERTY

• The library pytmc handles the project parsing and code extraction. Try pytmc code lcls-twincat-pmps/PMPS/HelperFunctions/FB_DummyHA.TcPOU and you should be able to get the same output.
• Now imagine if there's a syntax error in there somewhere, or we want to rewrite a certain section of this code to reformat it (or mutate it in any other way). The line numbers that blark sees no longer match up with what's in the XML.

After this PR is finished

• blark will handle determining file type formats on its own (currently only by filename extension; -> blark.input)
• blark will be able to introspect TwinCAT solutions, projects, and individual source code files (-> blark.solution)
• The parsed code items from the solution will produce implementation-agnostic instances of BlarkSourceItem (or BlarkCompositeSourceItem). BlarkSourceItem s may contain:
  • An the identifier of what's being parsed
  • A filename, if applicable
  • A map of line numbers
  • An optional grammar rule it should be parsed with
  • A user-specified object to go along with it
• Composite ones intend to just group those items together

In our above example:

• blark parse FB_DummyHA.TcPOU would be routed through blark.solution due to its TwinCAT file extension
• The solution class TcPOU would generate BlarkSourceItems for each part, which may be wrapped into a BlarkCompositeSourceItem with identifier FB_DummyHA:

  [1] FB_DummyHA/declaration (function_block)
  [2] FB_DummyHA.CheckRequest/declaration (method)
  [3] FB_DummyHA.CheckRequest/implementation (method)
  [4] FB_DummyHA.ElevateRequest/declaration (method)
  [5] FB_DummyHA.nLowerAuthorityID.get/declaration (property)
  [6] FB_DummyHA.nLowerAuthorityID.get/implementation (property)
  [7] FB_DummyHA.RemoveRequest/declaration (method)
  [8] FB_DummyHA.RemoveRequest/implementation (method)
  [9] FB_DummyHA.RequestBP/declaration (method)
  [10] FB_DummyHA.RequestBP/implementation (method)

• Each of these generates its own line number map

What about other input/output formats?

I haven't decided, maybe one or more of the following paths forward:

• I might be OK with including other input/output formats in the blark codebase if they aren't overly complicated/heavy (PLC Open XML? What do other codesys-using vendor projects look like?)
• Alternatively, input/output plugins could be a thing
• Failing that, suggesting users use blark as a library - or rely on JSON as an interchange format - could be OK

[codecov-commenter]

Codecov Report

Merging #64 (04c98d5) into master (4ea1ea0) will increase coverage by 4.9%. The diff coverage is 77.6%.

Additional details and impacted files

[](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer) ```diff @@ Coverage Diff @@ ## master #64 +/- ## ======================================== + Coverage 72.7% 77.7% +4.9% ======================================== Files 18 23 +5 Lines 3814 4884 +1070 ======================================== + Hits 2774 3796 +1022 - Misses 1040 1088 +48 ``` | [Impacted Files](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer) | Coverage Δ | | |---|---|---| | [blark/\_\_main\_\_.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-YmxhcmsvX19tYWluX18ucHk=) | `0.0% <0.0%> (ø)` | | | [blark/summary.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-Ymxhcmsvc3VtbWFyeS5weQ==) | `66.2% <ø> (+3.7%)` | :arrow_up: | | [blark/dependency\_store.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-YmxhcmsvZGVwZW5kZW5jeV9zdG9yZS5weQ==) | `72.3% <69.2%> (+72.3%)` | :arrow_up: | | [blark/solution.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-Ymxhcmsvc29sdXRpb24ucHk=) | `72.7% <72.7%> (ø)` | | | [blark/format.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-YmxhcmsvZm9ybWF0LnB5) | `81.3% <77.4%> (+23.3%)` | :arrow_up: | | [blark/parse.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-YmxhcmsvcGFyc2UucHk=) | `79.3% <79.4%> (+16.6%)` | :arrow_up: | | [blark/typing.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-YmxhcmsvdHlwaW5nLnB5) | `82.2% <82.2%> (ø)` | | | [blark/util.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-YmxhcmsvdXRpbC5weQ==) | `83.8% <83.4%> (+1.7%)` | :arrow_up: | | [blark/tests/test\_dependency\_store.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-YmxhcmsvdGVzdHMvdGVzdF9kZXBlbmRlbmN5X3N0b3JlLnB5) | `84.2% <84.2%> (ø)` | | | [blark/tests/test\_cli.py](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer#diff-YmxhcmsvdGVzdHMvdGVzdF9jbGkucHk=) | `90.3% <86.3%> (-2.2%)` | :arrow_down: | | ... and [7 more](https://codecov.io/gh/klauer/blark/pull/64?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer) | | ... and [2 files with indirect coverage changes](https://codecov.io/gh/klauer/blark/pull/64/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=K+Lauer)

[klauer]

Functionally, I think this has most of what I want from the solution parser, reformatting tools and such.

I'll claim that the API is subject to change if/when I look to adding more input/output formats. For now, it's probably fine.

I'll look back into some TODO items and aim to merge this in the nearish future.

[klauer]

This is too big of a refactor to ask for a real review of anyone (well, blark's one contributor...).

As noted above, comments/complaints/requests on this change are welcome - just make an issue. All API introduced here subject to change at my own whim otherwise