Ensure apischema round-trips all test suite cases

Goal

All blark dataclasses from blark.transform should be able to be converted to JSON by way of apischema.serialize
All generated JSON should similarly be able to be transformed back to dataclasses by way of apischema.deserialize
apischema should remain an optional dependency of blark

"JSON" above refers to JSON-serializable data types, including Python built-ins such as dicts, lists, ints, floats, and strings. The conversion of that to an actual serialized JSON string is an additional step.

Issue

During the course of testing, I ran into some issues with apischema on Python 3.9+ (at least according to inline code comments). I wasn't able to get to the heart of the issue to determine a path forward, I merely set up a "skip if on Python 3.9+ or apischema is unavailable".

Test suite reference: https://github.com/klauer/blark/blob/263e263004ae2c03f647743b557f9ab4141b92fe/blark/tests/test_transformer.py#L19-L21

Optional dependency in the transformer dataclass section: https://github.com/klauer/blark/blob/263e263004ae2c03f647743b557f9ab4141b92fe/blark/transform.py#L18-L28

Failing tests as of 263e263004ae2c03f647743b557f9ab4141b92fe

With APISCHEMA_SKIP set to False on Python 3.9 and blark 263e263004ae2c03f647743b557f9ab4141b92fe, the following tests fail on the serialization portion (at least sometimes due to a recursion error in apischema). apischema=0.17.5, 84 failed, 144 passed, 5 xfailed in 77.86s (0:01:17)

``` FAILED tests/test_transformer.py::test_expression_roundtrip[array_type_declaration-TypeName : ARRAY [1..2, 3..4] OF INT] - Re... FAILED tests/test_transformer.py::test_expression_roundtrip[array_type_declaration-TypeName : ARRAY [1..2] OF INT := [1, 2]] FAILED tests/test_transformer.py::test_expression_roundtrip[array_type_declaration-TypeName : ARRAY [1..2, 3..4] OF INT := [2(3), 3(4)]] FAILED tests/test_transformer.py::test_expression_roundtrip[array_type_declaration-TypeName : ARRAY [1..2, 3..4] OF Tc.SomeType] FAILED tests/test_transformer.py::test_expression_roundtrip[structure_type_declaration-TypeName :\nSTRUCT\nEND_STRUCT] - Recu... FAILED tests/test_transformer.py::test_expression_roundtrip[structure_type_declaration-TypeName EXTENDS Other.Type :\nSTRUCT\nEND_STRUCT] FAILED tests/test_transformer.py::test_expression_roundtrip[structure_type_declaration-TypeName : POINTER TO\nSTRUCT\nEND_STRUCT] FAILED tests/test_transformer.py::test_expression_roundtrip[structure_type_declaration-TypeName : POINTER TO\nSTRUCT\n iValue : INT;\nEND_STRUCT] FAILED tests/test_transformer.py::test_expression_roundtrip[structure_type_declaration-TypeName : POINTER TO\nSTRUCT\n iValue : INT := 3 + 4;\n stTest : ST_Testing := (1, 2);\n eValue : E_Test := E_Test.ABC;\n arrValue : ARRAY [1..2] OF INT := [1, 2];\n arrValue1 : INT (1..2);\n arrValue1 : (Value1 := 1) INT;\n sValue : STRING := 'abc';\n iValue1 AT %I* : INT := 5;\n iValue2 AT %Q* : INT := 5;\n iValue3 AT %M* : INT := 5;\n sValue1 : STRING[10] := 'test';\nEND_STRUCT] FAILED tests/test_transformer.py::test_input_roundtrip[input_declarations-VAR_INPUT\nEND_VAR] - RecursionError: maximum recur... FAILED tests/test_transformer.py::test_input_roundtrip[input_declarations-VAR_INPUT RETAIN\nEND_VAR] - RecursionError: maximu... FAILED tests/test_transformer.py::test_input_roundtrip[input_declarations-VAR_INPUT RETAIN\n iValue : INT;\n sValue : STRING := 'abc';\n wsValue : WSTRING := "abc";\nEND_VAR] FAILED tests/test_transformer.py::test_input_roundtrip[input_declarations-VAR_INPUT RETAIN\n iValue : INT;\n sValue : STRING := 'abc';\n wsValue : WSTRING := "abc";\n fbTest : FB_Test(1, 2, 3);\n fbTest : FB_Test(A := 1, B := 2, C => 3);\n fbTest : FB_Test(1, 2, A := 1, B := 2, C => 3);\n fbTest : FB_Test := (1, 2, 3);\nEND_VAR] FAILED tests/test_transformer.py::test_output_roundtrip[output_declarations-VAR_OUTPUT\nEND_VAR] - RecursionError: maximum re... FAILED tests/test_transformer.py::test_output_roundtrip[output_declarations-VAR_OUTPUT RETAIN\nEND_VAR] - RecursionError: max... FAILED tests/test_transformer.py::test_output_roundtrip[output_declarations-VAR_OUTPUT RETAIN\n iValue : INT;\n sValue : STRING := 'abc';\n wsValue : WSTRING := "abc";\nEND_VAR] FAILED tests/test_transformer.py::test_output_roundtrip[output_declarations-VAR_OUTPUT RETAIN\n iValue : INT;\n sValue : STRING := 'abc';\n wsValue : WSTRING := "abc";\n fbTest : FB_Test(1, 2, 3);\n fbTest : FB_Test(A := 1, B := 2, C => 3);\n fbTest : FB_Test(1, 2, A := 1, B := 2, C => 3);\n fbTest : FB_Test := (1, 2, 3);\nEND_VAR] FAILED tests/test_transformer.py::test_input_output_roundtrip[input_output_declarations-VAR_IN_OUT\nEND_VAR] - RecursionError... FAILED tests/test_transformer.py::test_input_output_roundtrip[input_output_declarations-VAR_IN_OUT\n iValue : INT;\n sValue : STRING := 'abc';\n wsValue : WSTRING := "abc";\nEND_VAR] FAILED tests/test_transformer.py::test_input_output_roundtrip[input_output_declarations-VAR_IN_OUT\n iValue : INT;\n sValue : STRING := 'abc';\n wsValue : WSTRING := "abc";\n fbTest : FB_Test(1, 2, 3);\n fbTest : FB_Test(A := 1, B := 2, C => 3);\n fbTest : FB_Test(1, 2, A := 1, B := 2, C => 3);\n fbTest : FB_Test := (1, 2, 3);\nEND_VAR] FAILED tests/test_transformer.py::test_global_roundtrip[global_var_declarations-VAR_GLOBAL\nEND_VAR] - RecursionError: maximu... FAILED tests/test_transformer.py::test_global_roundtrip[global_var_declarations-VAR_GLOBAL CONSTANT\nEND_VAR] - RecursionErro... FAILED tests/test_transformer.py::test_global_roundtrip[global_var_declarations-VAR_GLOBAL PERSISTENT\nEND_VAR] - RecursionEr... FAILED tests/test_transformer.py::test_global_roundtrip[global_var_declarations-VAR_GLOBAL CONSTANT PERSISTENT\nEND_VAR] - Re... FAILED tests/test_transformer.py::test_global_roundtrip[global_var_declarations-VAR_GLOBAL CONSTANT PERSISTENT\n iValue : INT := 5;\n fbTest1 : FB_Test(1, 2);\n fbTest2 : FB_Test(A := 1, B := 2);\n fbTest3 : FB_TestC;\n i_iFscEB1Ch0AI AT %I* : INT;\nEND_VAR] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK fbName\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK fbName IMPLEMENTS I_fbName\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK fbName IMPLEMENTS I_fbName, I_fbName2\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK ABSTRACT fbName EXTENDS OtherFbName\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK PRIVATE fbName EXTENDS OtherFbName\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK PUBLIC fbName EXTENDS OtherFbName\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK INTERNAL fbName EXTENDS OtherFbName\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK PROTECTED fbName EXTENDS OtherFbName\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK FINAL fbName EXTENDS OtherFbName\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK fbName\nVAR_INPUT\n bExecute : BOOL;\nEND_VAR\nVAR_OUTPUT\n iResult : INT;\nEND_VAR\nVAR_IN_OUT\n iShared : INT;\nEND_VAR\nVAR CONSTANT\n iConstant : INT := 5;\nEND_VAR\nVAR\n iInternal : INT;\nEND_VAR\nVAR RETAIN\n iRetained : INT;\nEND_VAR\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[located_var_declarations-VAR RETAIN\n iValue AT %IB1 : INT := 5;\nEND_VAR] FAILED tests/test_transformer.py::test_fb_roundtrip[temp_var_decls-VAR_TEMP\n iGlobalVar : INT;\nEND_VAR] - RecursionError... FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK fbName\n iValue := 1;\n iValue;\n iValue S= 1;\n iValue R= 1;\n iValue REF= GVL.iTest;\n fbOther(A := 5, B => iValue, NOT C => iValue1);\n IF 1 THEN\n iValue := 1;\n IF 1 THEN\n iValue := 1;\n END_IF\n END_IF\n Method();\n RETURN;\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_type_declaration-FUNCTION_BLOCK fbName\n Method();\n IF 1 THEN\n EXIT;\n END_IF\nEND_FUNCTION_BLOCK] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_method_declaration-METHOD PRIVATE MethodName : RETURNTYPE\nEND_METHOD] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_method_declaration-METHOD PRIVATE MethodName : ARRAY [1..2] OF INT\nEND_METHOD] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_method_declaration-METHOD PUBLIC MethodName : RETURNTYPE\nEND_METHOD] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_method_declaration-METHOD PUBLIC MethodName\nEND_METHOD] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_method_declaration-METHOD MethodName : RETURNTYPE\n VAR_INPUT\n bExecute : BOOL;\n END_VAR\n VAR_OUTPUT\n iResult : INT;\n END_VAR\n iResult := 5;\nEND_METHOD] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_method_declaration-METHOD PUBLIC MethodName\n VAR_INST\n bExecute : BOOL;\n END_VAR\nEND_METHOD] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_method_declaration-METHOD PUBLIC ABSTRACT MethodName : LREAL\n VAR_INPUT\n I : UINT;\n END_VAR\nEND_METHOD] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_property_declaration-PROPERTY PRIVATE PropertyName : RETURNTYPE\nEND_PROPERTY] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_property_declaration-PROPERTY PRIVATE PropertyName : ARRAY [1..2] OF INT\nEND_PROPERTY] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_property_declaration-PROPERTY PUBLIC PropertyName : RETURNTYPE\nEND_PROPERTY] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_property_declaration-PROPERTY PUBLIC PropertyName\nEND_PROPERTY] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_property_declaration-PROPERTY PropertyName : RETURNTYPE\n VAR_INPUT\n bExecute : BOOL;\n END_VAR\n VAR_OUTPUT\n iResult : INT;\n END_VAR\n iResult := 5;\nEND_PROPERTY] FAILED tests/test_transformer.py::test_fb_roundtrip[function_block_property_declaration-PROPERTY PUBLIC ABSTRACT PropertyName : LREAL\n VAR_INPUT\n I : UINT;\n END_VAR\nEND_PROPERTY] FAILED tests/test_transformer.py::test_statement_roundtrip[if_statement-IF 1 THEN\n iValue := 1;\n IF 1 THEN\n iValue := 1;\n END_IF\nELSIF 2 THEN\n iValue := 2;\nELSIF 3 THEN\n iValue := 2;\nELSE\n iValue := 3;\nEND_IF] FAILED tests/test_transformer.py::test_statement_roundtrip[if_statement-IF 1 THEN\n IF 2 THEN\n IF 3 * x THEN\n y();\n ELSE\n END_IF\n END_IF\nEND_IF] FAILED tests/test_transformer.py::test_statement_roundtrip[if_statement-IF 1 AND_THEN 1 THEN\n y();\nEND_IF] - RecursionEr... FAILED tests/test_transformer.py::test_statement_roundtrip[if_statement-IF 0 OR_ELSE 1 THEN\n y();\nEND_IF] - RecursionErr... FAILED tests/test_transformer.py::test_statement_roundtrip[case_statement-CASE expr OF\n1:\n abc();\n2, 3, GVL.Constant:\n def();\nELSE\n ghi();\nEND_CASE] FAILED tests/test_transformer.py::test_statement_roundtrip[case_statement-CASE a.b.c^.d OF\n1..10:\n OneToTen := OneToTen + 1;\nEnumValue:\nEND_CASE] FAILED tests/test_transformer.py::test_statement_roundtrip[while_statement-WHILE expr\nDO\n iValue := iValue + 1;\nEND_WHILE] FAILED tests/test_transformer.py::test_statement_roundtrip[repeat_statement-REPEAT\n iValue := iValue + 1;\nUNTIL expr\nEND_REPEAT] FAILED tests/test_transformer.py::test_statement_roundtrip[for_statement-FOR iIndex := 0 TO 10\nDO\n iValue := iIndex * 2;\nEND_FOR] FAILED tests/test_transformer.py::test_statement_roundtrip[for_statement-FOR iIndex := 0 TO 10 BY 1\nDO\n iValue := iIndex * 2;\nEND_FOR] FAILED tests/test_transformer.py::test_statement_roundtrip[for_statement-FOR iIndex := (iValue - 5) TO iValue * 10 BY iValue MOD 10\nDO\n arrArray[iIndex] := iIndex * 2;\nEND_FOR] FAILED tests/test_transformer.py::test_statement_roundtrip[for_statement-FOR iIndex[1] := 0 TO 10\nDO\n iValue := iIndex * 2;\nEND_FOR] FAILED tests/test_transformer.py::test_function_roundtrip[int_with_input] - RecursionError: maximum recursion depth exceeded ... FAILED tests/test_transformer.py::test_function_roundtrip[int_with_pointer_retval] - RecursionError: maximum recursion depth ... FAILED tests/test_transformer.py::test_function_roundtrip[int_with_input_output] - RecursionError: maximum recursion depth ex... FAILED tests/test_transformer.py::test_function_roundtrip[no_return_type] - RecursionError: maximum recursion depth exceeded ... FAILED tests/test_transformer.py::test_function_roundtrip[dotted_return_type] - RecursionError: maximum recursion depth excee... FAILED tests/test_transformer.py::test_program_roundtrip[program_declaration-PROGRAM ProgramName\nEND_PROGRAM] - RecursionErr... FAILED tests/test_transformer.py::test_program_roundtrip[program_declaration-PROGRAM ProgramName\n VAR_INPUT\n iValue : INT;\n END_VAR\n VAR_ACCESS\n AccessName : SymbolicVariable : TypeName READ_WRITE;\n END_VAR\n iValue := iValue + 1;\nEND_PROGRAM] FAILED tests/test_transformer.py::test_input_output_comments[input_output_declarations-/ Var in and out\n(* Var in and out *)\nVAR_IN_OUT\n / Variable\n iVar : INT;\nEND_VAR] FAILED tests/test_transformer.py::test_incomplete_located_var_decls[incomplete_located_var_declarations-VAR\n iValue AT %Q* : INT;\n sValue AT %I* : STRING [255];\n wsValue AT %I* : WSTRING [255];\nEND_VAR] FAILED tests/test_transformer.py::test_incomplete_located_var_decls[incomplete_located_var_declarations-VAR RETAIN\n iValue AT %I* : INT;\n iValue1 AT %Q* : INT;\nEND_VAR] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE\nEND_TYPE] - RecursionError: maximum ... FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE TypeName :\n STRUCT\n xPLC_CnBitsValid : BOOL;\n xPLC_CnBits : ARRAY [0..20] OF BYTE;\n END_STRUCT\nEND_TYPE] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE ArrayTypeName : ARRAY [1..10, 2..20] OF INT;\nEND_TYPE] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE StringTypeName : STRING[10] := 'Literal';\nEND_TYPE] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE SimpleTypeName EXTENDS OtherType : POINTER TO INT;\nEND_TYPE] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE SubrangeTypeName : POINTER TO INT (1..5);\nEND_TYPE] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE EnumeratedTypeName : REFERENCE TO Identifier;\nEND_TYPE] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE EnumeratedTypeName : REFERENCE TO (IdentifierA, INT#IdentifierB := 1);\nEND_TYPE] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE TypeName :\n UNION\n intval : INT;\n as_bytes : ARRAY [0..2] OF BYTE;\n END_UNION\nEND_TYPE] FAILED tests/test_transformer.py::test_data_type_declaration[data_type_declaration-TYPE TypeName :\n UNION\n intval : INT;\n enum : (iValue := 1, iValue2 := 2) INT;\n END_UNION\nEND_TYPE] ```

Random thoughts

It's likely that the issue is due to mistaken/incorrect type hints. There is at least a tiny possibility that this is an apischema bug.
Type hints for the problematic ones in the failed tests above need reviewing to see what's wrong.
CI should be updated to "run test suite with optional dependencies" and "run test suite without optional dependencies" at some point
Ideally, the round-tripping of code -> dataclass -> code and code -> dataclass -> JSON -> dataclass would be separate in the test suite. As of now, these are only just tacked on in roundtrip_rule.

klauer / blark