search

POETSII / Orchestrator

The Orchestrator is the configuration and run-time management system for POETS platforms.
1 stars 1 forks source link

Automated testing of xml parsing and compilation using `expect` #208

Closed m8pple closed 3 years ago

m8pple commented 3 years ago

Based on the discussion in #198, it seemed important to get some method for automated unit-testing of xml parsing and compilation. Given the lack of stop-on-error (#197), the only way seemed to be to emulate interaction.

Using GNU expect I was able to put together a script that will parse and compile an xml file, and attempt to tell you whether it succeeded or not through its error code.

This is commited to 0fdcebc45309c0df9485ef0c8879e3c6d9672598 in compile_app.exp.

It probably misses various error conditions, but seems like a decent starting point.

expect is not installed on ayres et. al., but I was able to install it as non-admin using brew.

m8pple commented 3 years ago

After fixing #220 , #221 , #222, we get commit 74e27478dee3e7031cda1c608e8abb1080a83756. This allows the orchestrator to parse (but not compile) all XML from IC given as part of the PIP20 / v4 discussions:

dt10@joxer:/mnt/e/dt10_all/POETS/Orchestrator$ Tests/ReferenceXML/test_parse.sh 
TAP version 13
ok 0 - Parse Tests/ReferenceXML/v4/PEP20/apps/amg_poisson_8_8_v4.xml
ok 1 - Parse Tests/ReferenceXML/v4/PEP20/apps/apsp_100_10_v4.xml
ok 2 - Parse Tests/ReferenceXML/v4/PEP20/apps/apsp_vec_barrier_150_10.xml
ok 3 - Parse Tests/ReferenceXML/v4/PEP20/apps/betweeness_centrality_16_16_20_20_v4.xml
ok 4 - Parse Tests/ReferenceXML/v4/PEP20/apps/example_device_idle.xml
ok 5 - Parse Tests/ReferenceXML/v4/PEP20/apps/gals_heat_8x8_v4.xml
ok 6 - Parse Tests/ReferenceXML/v4/PEP20/apps/ising_spin_fix_16_2_v4.xml
ok 7 - Parse Tests/ReferenceXML/v4/PEP20/apps/relaxation_heat_16x16_v4.xml
ok 8 - Parse Tests/ReferenceXML/v4/PEP20/apps/storm_16_4_8_v4.xml
ok 9 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/one-dev-two-ports.xml
ok 10 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/one-dev.xml
ok 11 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-no-spaces.xml
ok 12 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata1.xml
ok 13 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest-plus-metadata2.xml
ok 14 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L3-compilation/tiniest.xml
ok 15 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/example-dt10.xml
ok 16 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-fan-inst-in-proc.xml
ok 17 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-no-inst.xml
ok 18 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-one-inst-in-proc.xml
ok 19 - Parse Tests/ReferenceXML/v4/PEP20/tests/valid/L4-run-time/external-two-inst-in-proc.xml
ok 20 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-properties.xml
ok 21 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-shared-code.xml
ok 22 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-state.xml
ok 23 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name1.xml
ok 24 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name2.xml
ok 25 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/malformed-path.xml
ok 26 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-properties.xml
ok 27 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-shared-code.xml
ok 28 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/duplicate-state.xml
ok 29 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name1.xml
ok 30 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/invalid-element-name2.xml
ok 31 - Should fail to parse Tests/ReferenceXML/v4/PEP20/tests/invalid/L0-syntax/malformed-path.xml
dt10@joxer:/mnt/e/dt10_all/POETS/Orchestrator$

I haven't tried any Southampton XML, but if you drop it in the appropriate directory it will be added to the tests.

m8pple commented 3 years ago

This is ported into the pull request in #264 . However, as noted there if testing is not something you want in the Orchestrator repo, then feel free to strip it out.

heliosfa commented 3 years ago

resolved in #264