Hypervisor compilation failure with hybrid_rt scenarios

[tttech-industrial-buchsbaum]

At latest master compilation of hypervisor with whl-ipc-i5/hybrid_rt or cfl-k700-i7/hybrid_rt as board/scenario fails. This came up when installing xmlschema for verification.

Platform Build on a debian-:bullseye docker image with the required packages installed: For elementpath and xmlschema the newest available packages have been repackaged and installed:

acrn-devel@host:/acrn/acrn-hypervisor$ dpkg -l python3-xmlschema python3-elementpath
ii  python3-elementpath                  2.5.0-1                        all          Providing XPath selectors for Python's XML data structures
ii  python3-xmlschema                    1.10.0-1                       all          implementation of XML Schema for Python

All other required packages are taken directly from Debian Bullseye

Codebase This has been observed on latest master, i.e. e748842da0fd642b3d5b2d5af0bc766062ecb78c

Scenario Board: whl-ipc-i5 or cfl-k700-i7, Scenario: hybrid_rt

To Reproduce

acrn-devel@host:/acrn/acrn-hypervisor$ make clean
acrn-devel@host:/acrn/acrn-hypervisor$ make BOARD=misc/config_tools/data/whl-ipc-i5/whl-ipc-i5.xml SCENARIO=misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml hypervisor
Board XML is fetched from /acrn/acrn-hypervisor/misc/config_tools/data/whl-ipc-i5/whl-ipc-i5.xml
Scenario XML is being fetched from /acrn/acrn-hypervisor/misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml
/acrn/acrn-hypervisor/build/hypervisor/configs/allocation.xml generated
/acrn/acrn-hypervisor/build/hypervisor/configs/unified.xml generated
/acrn/acrn-hypervisor/build/hypervisor/configs/config.mk generated
Traceback (most recent call last):
  File "/acrn/acrn-hypervisor/hypervisor/../misc/config_tools/scenario_config/validator.py", line 306, in <module>
    main(args)
  File "/acrn/acrn-hypervisor/hypervisor/../misc/config_tools/scenario_config/validator.py", line 285, in main
    nr_all_errors = validate_one(validation_pipeline, obj, args.board, args.scenario)
  File "/acrn/acrn-hypervisor/hypervisor/../misc/config_tools/scenario_config/validator.py", line 232, in validate_one
    validation_pipeline.run(pipeline_obj)
  File "/acrn/acrn-hypervisor/misc/config_tools/scenario_config/pipeline.py", line 68, in run
    stage.run(obj)
  File "/acrn/acrn-hypervisor/hypervisor/../misc/config_tools/scenario_config/validator.py", line 205, in run
    errors = obj.get("validator").check_semantics(obj.get("board_etree"), obj.get("scenario_etree"))
  File "/acrn/acrn-hypervisor/hypervisor/../misc/config_tools/scenario_config/validator.py", line 95, in check_semantics
    for error in it:
  File "/usr/lib/python3/dist-packages/xmlschema/validators/schema.py", line 1511, in iter_errors
    for result in xsd_element.iter_decode(elem, **kwargs):
  File "/usr/lib/python3/dist-packages/xmlschema/validators/elements.py", line 619, in iter_decode
    for error in assertion(elem, **kwargs):
  File "/usr/lib/python3/dist-packages/xmlschema/validators/assertions.py", line 112, in __call__
    if not self.token.evaluate(context):
  File "/acrn/acrn-hypervisor/misc/config_tools/scenario_config/elementpath_overlay.py", line 123, in evaluate
    if self.boolean_value([x for x in self[-1].select(private_context)]):
  File "/acrn/acrn-hypervisor/misc/config_tools/scenario_config/elementpath_overlay.py", line 123, in <listcomp>
    if self.boolean_value([x for x in self[-1].select(private_context)]):
  File "/usr/local/lib/python3.9/dist-packages/elementpath/xpath_token.py", line 114, in select
    item = self.evaluate(context)
  File "/acrn/acrn-hypervisor/misc/config_tools/scenario_config/elementpath_overlay.py", line 123, in evaluate
    if self.boolean_value([x for x in self[-1].select(private_context)]):
  File "/acrn/acrn-hypervisor/misc/config_tools/scenario_config/elementpath_overlay.py", line 123, in <listcomp>
    if self.boolean_value([x for x in self[-1].select(private_context)]):
  File "/usr/local/lib/python3.9/dist-packages/elementpath/xpath_token.py", line 114, in select
    item = self.evaluate(context)
  File "/acrn/acrn-hypervisor/misc/config_tools/scenario_config/elementpath_overlay.py", line 120, in evaluate
    for results in copy(context).iter_product(selectors, varnames):
  File "/usr/local/lib/python3.9/dist-packages/elementpath/xpath_context.py", line 412, in iter_product
    value = next(iterators[k])
  File "/usr/local/lib/python3.9/dist-packages/elementpath/xpath2/_xpath2_functions.py", line 524, in select_distinct_values_function
    yield from distinct_values()
  File "/usr/local/lib/python3.9/dist-packages/elementpath/xpath2/_xpath2_functions.py", line 502, in distinct_values
    for item in self[0].select(context):
  File "/usr/local/lib/python3.9/dist-packages/elementpath/xpath1/_xpath1_operators.py", line 631, in select_union_operator
    yield from context.iter_results(results)
  File "/usr/local/lib/python3.9/dist-packages/elementpath/xpath_context.py", line 362, in iter_results
    results.remove(self.item)
KeyError: UntypedAtomic('00:08.0')
make[1]: Leaving directory '/acrn/acrn-hypervisor/hypervisor'
make -C /acrn/acrn-hypervisor/hypervisor BOARD=/acrn/acrn-hypervisor/misc/config_tools/data/whl-ipc-i5/whl-ipc-i5.xml SCENARIO=/acrn/acrn-hypervisor/misc/config_tools/data/whl-ipc-i5/hybrid_rt.xml HV_OBJDIR=/acrn/acrn-hypervisor/build/hypervisor RELEASE=
make[1]: Entering directory '/acrn/acrn-hypervisor/hypervisor'
/acrn/acrn-hypervisor/build/hypervisor/include/config.h generated
Start pre-build static check ...
make -C ../misc/hv_prebuild BOARD=whl-ipc-i5 SCENARIO=hybrid_rt TARGET_DIR=/acrn/acrn-hypervisor/build/hypervisor/configs
make[2]: Entering directory '/acrn/acrn-hypervisor/misc/hv_prebuild'
make[2]: *** No rule to make target '/acrn/acrn-hypervisor/build/hypervisor/configs/scenarios/hybrid_rt/vm_configurations.c', needed by 'default'.  Stop.
make[2]: Leaving directory '/acrn/acrn-hypervisor/misc/hv_prebuild'
make[1]: *** [Makefile:419: pre_build] Error 2
make[1]: Leaving directory '/acrn/acrn-hypervisor/hypervisor'
make: *** [Makefile:126: hypervisor] Error 2

Expected behavior Builds without issues

Remark Temporarily reverting 5bd3e964264b9476a5f123b96d62ae3672d2e837 and db1799229336a382c127cd678d5e41f85ee1548f fixes this issue. To continue my work on adding debianization, I will skip compiling hybrid_rt scenarios for know ;-)

[tttech-industrial-buchsbaum]

BTW, this also is true for nuc11tnbi5/shared, since it also uses VBDF configurations.

[junjiemao1]

Please use elementpath v2.4.0 instead because v2.5.0 has issues evaluating distinct-values on a union of node sets.

We also have a requirements.txt under misc/config_tools which includes the version requirements of the Python libraries.

[tttech-industrial-buchsbaum]

I can confirm that using elementpath 2.4.0 and xmlschema 1.9.2 (as stated in requirements.txt) solves this issue. Nevertheless the need for dedicated versions makes a packaged based build impossible for distributions already using newer version, like Debian Bookworm, Debian Sid and Ubuntu Jammy (which is already released!). They all use xmlschema 1.10.0, which somehow requires elementpath 2.5.0. For yocto based systems this is usually not a problem due to the flexibility of yocto (unless any other package on the system requires e.g. xmlschema >= 1.10.0). Even if we circumvent this problem at build time, we need these packages even when running the python based ACRN tools. This leads to conflicting situations on such distributions (providing the 'older' packages conflict with the distribution versions whit other packages depending on them). IMHO we should never required to pin to exact version but rather request a minimum version. This is the only way we can provide Debian packages with correct dependencies. Do you see any chance to switch to the new versions?

[jouillet]

Please use elementpath v2.4.0 instead because v2.5.0 has issues evaluating distinct-values on a union of node sets.

We also have a requirements.txt under misc/config_tools which includes the version requirements of the Python libraries.

The requirement of elementpath v2.4.0 effectively prevents the use of Ubuntu 22.04 (and later), and will be a problem for future releases of Debian as well. It looks to me like this has to be fixed at some point in the future anyway, so why not fix it now?

[junjiemao1]

A real fix has to go to the elementpath library which fails to evaluate a valid XPATH it formerly can. The only workaround I can make on our side is to catch the exception and warn that some assertions failed to evaluate rather than blocking the build.

[junjiemao1]

That said, I've cooked a patch to work this around at the cost of a slight increase in XML validation time. Building the updated version using elementpath 2.5.0 and xmlschema 1.10.0 works for me. Please verify if it is the case on your side as well. Thanks.