m8pple
commented
3 years ago On my (development-dt10) branch, I get the results below.
All of those files could be correctly accepted/rejected by the two from-scratch parsers (python and C++) included in PIP20 (as well as graph_schema), and the functional tests work in the minimal functional simulator included in PIP20 (as well as in the various simulators in graph_schema).
dt10@joxer:/mnt/e/dt10_all/POETS/Orchestrator$ Tests/ReferenceXML/test_parse.sh
TAP version 13
not ok 0 - Parse Tests/ReferenceXML/v4/PEP20/apps/amg_poisson_8_8_v4.xml
not ok 1 - Parse Tests/ReferenceXML/v4/PEP20/apps/apsp_100_10_v4.xml
not ok 2 - Parse Tests/ReferenceXML/v4/PEP20/apps/apsp_vec_barrier_150_10.xml
not ok 3 - Parse Tests/ReferenceXML/v4/PEP20/apps/betweeness_centrality_16_16_20_20_v4.xml
ok 4 - Parse Tests/ReferenceXML/v4/PEP20/apps/example_device_idle.xml
not ok 5 - Parse Tests/ReferenceXML/v4/PEP20/apps/gals_heat_8x8_v4.xml
not ok 6 - Parse Tests/ReferenceXML/v4/PEP20/apps/ising_spin_fix_16_2_v4.xml
not ok 7 - Parse Tests/ReferenceXML/v4/PEP20/apps/relaxation_heat_16x16_v4.xml
not ok 8 - Parse Tests/ReferenceXML/v4/PEP20/apps/storm_16_4_8_v4.xml
ok 9 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/one-dev-two-ports.xml
ok 10 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/one-dev.xml
ok 11 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-no-spaces.xml
not ok 12 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata1.xml
not ok 13 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata2.xml
not ok 14 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata3.xml
not ok 15 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata4.xml
ok 16 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest.xml
ok 17 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/example-dt10.xml
not ok 18 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-fan-inst-in-proc.xml
ok 19 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-no-inst.xml
not ok 20 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-one-inst-in-proc.xml
not ok 21 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-two-inst-in-proc.xml
ok 22 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-properties.xml
ok 23 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-shared-code.xml
ok 24 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-state.xml
ok 25 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name1.xml
ok 26 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name2.xml
ok 27 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/malformed-path.xml
not ok 28 - Parse Tests/ReferenceXML/v4/PEP20/apps/amg_poisson_8_8_v4.xml
not ok 29 - Parse Tests/ReferenceXML/v4/PEP20/apps/apsp_100_10_v4.xml
not ok 30 - Parse Tests/ReferenceXML/v4/PEP20/apps/apsp_vec_barrier_150_10.xml
not ok 31 - Parse Tests/ReferenceXML/v4/PEP20/apps/betweeness_centrality_16_16_20_20_v4.xml
ok 32 - Parse Tests/ReferenceXML/v4/PEP20/apps/example_device_idle.xml
not ok 33 - Parse Tests/ReferenceXML/v4/PEP20/apps/gals_heat_8x8_v4.xml
not ok 34 - Parse Tests/ReferenceXML/v4/PEP20/apps/ising_spin_fix_16_2_v4.xml
not ok 35 - Parse Tests/ReferenceXML/v4/PEP20/apps/relaxation_heat_16x16_v4.xml
not ok 36 - Parse Tests/ReferenceXML/v4/PEP20/apps/storm_16_4_8_v4.xml
ok 37 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/one-dev-two-ports.xml
ok 38 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/one-dev.xml
ok 39 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-no-spaces.xml
not ok 40 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata1.xml
not ok 41 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata2.xml
not ok 42 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata3.xml
not ok 43 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata4.xml
ok 44 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest.xml
ok 45 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/example-dt10.xml
not ok 46 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-fan-inst-in-proc.xml
ok 47 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-no-inst.xml
not ok 48 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-one-inst-in-proc.xml
not ok 49 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-two-inst-in-proc.xml
ok 50 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-properties.xml
ok 51 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-shared-code.xml
ok 52 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-state.xml
ok 53 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name1.xml
ok 54 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name2.xml
ok 55 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/malformed-path.xml
not ok 56 - Compile Tests/ReferenceXML/v4/PEP20/apps/amg_poisson_8_8_v4.xml
not ok 57 - Compile Tests/ReferenceXML/v4/PEP20/apps/apsp_100_10_v4.xml
not ok 58 - Compile Tests/ReferenceXML/v4/PEP20/apps/apsp_vec_barrier_150_10.xml
not ok 59 - Compile Tests/ReferenceXML/v4/PEP20/apps/betweeness_centrality_16_16_20_20_v4.xml
not ok 60 - Compile Tests/ReferenceXML/v4/PEP20/apps/example_device_idle.xml
not ok 61 - Compile Tests/ReferenceXML/v4/PEP20/apps/gals_heat_8x8_v4.xml
not ok 62 - Compile Tests/ReferenceXML/v4/PEP20/apps/ising_spin_fix_16_2_v4.xml
not ok 63 - Compile Tests/ReferenceXML/v4/PEP20/apps/relaxation_heat_16x16_v4.xml
not ok 64 - Compile Tests/ReferenceXML/v4/PEP20/apps/storm_16_4_8_v4.xml
ok 65 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/one-dev-two-ports.xml
ok 66 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/one-dev.xml
ok 67 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-no-spaces.xml
not ok 68 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata1.xml
not ok 69 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata2.xml
not ok 70 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata3.xml
not ok 71 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata4.xml
ok 72 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest.xml
not ok 73 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/example-dt10.xml
not ok 74 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-fan-inst-in-proc.xml
ok 75 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-no-inst.xml
not ok 76 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-one-inst-in-proc.xml
not ok 77 - Compile Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-two-inst-in-proc.xml
ok 78 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-properties.xml
ok 79 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-shared-code.xml
ok 80 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-state.xml
ok 81 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name1.xml
ok 82 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name2.xml
ok 83 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/malformed-path.xml
not ok 84 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L1-graph-topology/duplicate-device-id.xml
not ok 85 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L1-graph-topology/duplicate-edge-path.xml
not ok 86 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L1-graph-topology/non-existent-message-type.xml
not ok 87 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L1-graph-topology/reversed-edge-path.xml
not ok 88 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L2-graph-attributes/struct-with-extra-features1.xml
not ok 89 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L2-graph-attributes/struct-with-extra-features2.xml
not ok 90 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L2-graph-attributes/struct-with-extra-features3.xml
ok 91 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L3-compilation/invalid-rts-flag.xml
ok 92 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L3-compilation/modifying-device-properties.xml
ok 93 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L3-compilation/modifying-device-properties2.xml
heliosfa
The commit b7df4b7e416fdb8af408a6c0fc1719c5d9f345fa includes the reference xml from PIP20, plus a testing script that tries to parse and compile all the xml. It both tries to test whether good xml succeeds, and whether bad xml fails. The output is intended to be TAPS compliant.
I have not looked at any particular successes or failures, not least because my branch of the Orchestrator deviates from 1.0.0-alpha in that it adheres to the XML v4 specification. If we are going to adjust the grammar or move to v5 then that will be something else.
It should be possible to do hardware-in-the-loop testing using similar techniques. I've got such automated integration tests for my branch of Tinsel, which sort of well enough to stop me breaking it when I am optimising things.