BUG: problems with unit test workflow

[prjemian]

While working on #303, noticed that some unit tests have been failing although the unit test workflow continued and reported success. This is a problem. Work on this came to overshadow #314 so that PR was split into two parts, #315 and this issue.

• [x] workflow should indicate if pytest had one or more failed tests
• [x] all failed tests should be resolved
• [x] ModuleNotFoundError: No module named 'suitcase'

---

Related:

• 303

• 314

• 315

[prjemian]

This is a common report in the tests:

2023-12-15T14:15:49.3143529Z             |   File "/home/runner/micromamba/envs/hklpy-test-py-3.11/lib/python3.11/site-packages/databroker/mongo_normalized.py", line 1277, in get_serializer
2023-12-15T14:15:49.3143721Z             |     from suitcase.mongo_normalized import Serializer
2023-12-15T14:15:49.3144061Z             | ModuleNotFoundError: No module named 'suitcase'

A solution for which was noted previously.

[prjemian]

These tests fail with the hklpy environment (databroker v2):

FAILED hkl/tests/test_save_restore_UB.py::test_fourc_orientation_save - assert 0 == 1
FAILED hkl/tests/test_save_restore_UB.py::test_fourc_run_orientation_info - TypeError: Type is not JSON serializable: _ResultTuple
FAILED hkl/tests/test_save_restore_UB.py::test_list_orientation_runs - TypeError: Type is not JSON serializable: _ResultTuple
FAILED hkl/tests/test_save_restore_UB.py::test_no_primary_stream - TypeError: Type is not JSON serializable: _ResultTuple
FAILED hkl/tests/test_save_restore_UB.py::test_missing_energy_key - TypeError: Type is not JSON serializable: _ResultTuple
FAILED hkl/tests/test_save_restore_UB.py::test_restore_orientation - TypeError: Type is not JSON serializable: _ResultTuple
FAILED hkl/tests/test_save_restore_UB.py::test_restore_sample - TypeError: Type is not JSON serializable: _ResultTuple
FAILED hkl/tests/test_util.py::test_issue215 - TypeError: Type is not JSON serializable: _ResultTuple

and succeed with bluesky_2024_1 environment (databroker 1.2):

hkl/tests/test_save_restore_UB.py::test_fourc_orientation_save PASSED                                                                          [ 12%]
hkl/tests/test_save_restore_UB.py::test_fourc_run_orientation_info PASSED                                                                      [ 25%]
hkl/tests/test_save_restore_UB.py::test_list_orientation_runs PASSED                                                                           [ 37%]
hkl/tests/test_save_restore_UB.py::test_no_primary_stream PASSED                                                                               [ 50%]
hkl/tests/test_save_restore_UB.py::test_missing_energy_key PASSED                                                                              [ 62%]
hkl/tests/test_save_restore_UB.py::test_restore_orientation PASSED                                                                             [ 75%]
hkl/tests/test_save_restore_UB.py::test_restore_sample PASSED                                                                                  [ 87%]
hkl/tests/test_util.py::test_issue215 PASSED                                                                                                   [100%]

[prjemian]

suitcase packages in the two environments are identical:

(bluesky_2024_1) prjemian@arf:~/.../Bluesky/hklpy$ conda list suitcase
# packages in environment at /home/prjemian/.conda/envs/bluesky_2024_1:
#
# Name                    Version                   Build  Channel
suitcase-jsonl            0.2.2              pyhd8ed1ab_0    conda-forge
suitcase-mongo            0.4.0              pyhd8ed1ab_0    conda-forge
suitcase-msgpack          0.3.0              pyhd8ed1ab_1    conda-forge
suitcase-utils            0.5.3              pyhd8ed1ab_1    conda-forge
(bluesky_2024_1) prjemian@arf:~/.../Bluesky/hklpy$ conda activate hklpy
(hklpy) prjemian@arf:~/.../Bluesky/hklpy$ conda list suitcase
# packages in environment at /home/prjemian/.conda/envs/hklpy:
#
# Name                    Version                   Build  Channel
suitcase-jsonl            0.2.2              pyhd8ed1ab_0    conda-forge
suitcase-mongo            0.4.0              pyhd8ed1ab_0    conda-forge
suitcase-msgpack          0.3.0              pyhd8ed1ab_1    conda-forge
suitcase-utils            0.5.4              pyhd8ed1ab_0    conda-forge

[prjemian]

The point of failure happens as data for a sample reflection is being serialized to JSON. Starting from the diffractometer's reflections attribute: https://github.com/bluesky/hklpy/blob/e0ba082db35120efa7d01b60af74fad419ce8d84/hkl/diffract.py#L179-L183 https://github.com/bluesky/hklpy/blob/e0ba082db35120efa7d01b60af74fad419ce8d84/hkl/sample.py#L321-L327

The reflections() property returns a list of libhkl.SampleReflection objects. A SampleReflection object is a ResultTuple (not defined in hklpy source) which renders such as this example: (h=1.0, k=2.0, l=3.0).

Can't convert this object to JSON (from tiled.utils.safe_json_dump()) TypeError: Type is not JSON serializable: _ResultTuple. The exception is raised by orjson.dumps() from this line in tiled.utils

return orjson.dumps(content, option=orjson.OPT_SERIALIZE_NUMPY, default=default)

We've seen this error before. But our attention was on the constraints information. This time, it is reflections.

What has been exceptionally problematic is that this exception did not stop the RunEngine, nor was it exposed to the user. Instead, the situation was discovered by applying safe_json_dump() to each of the diffractometer's configuration attributes. This seems to be an upstream problem which hklpy should not try to resolve.

[prjemian]

All tests pass locally with databroker v1.2 yet fail with databroker 2.0. This is a positive situation. The failures can be attributed to code changes related to databroker v2. This is understood to be a structural difference. With databroker v1.2, the documents are from intake while databroker v2 provides tiled Containers.

FAILED hkl/tests/test_util.py::test_issue215 - AssertionError: assert 'fourc' in {}
FAILED hkl/tests/test_util.py::test_RE_documents[None] - AssertionError: assert 'e4cv' in {}
FAILED hkl/tests/test_util.py::test_RE_documents[short] - AssertionError: assert 'e4cv' in {}
FAILED hkl/tests/test_util.py::test_RE_documents[long] - AssertionError: assert 'e4cv' in {}

The common point of failure is this code: https://github.com/bluesky/hklpy/blob/dfb7f1861a7462d30d3847f166e66361f90ad8a5/hkl/util.py#L329-L358