search

bluesky / hklpy

Diffractometer computation library with ophyd pseudopositioner support
https://blueskyproject.io/hklpy
BSD 3-Clause "New" or "Revised" License
4 stars 12 forks source link

relax the pinning of databroker=1.2 #268

Closed prjemian closed 10 months ago

prjemian commented 1 year ago

In the release process of the conda-forge package for hklpy, this pin was noted: https://github.com/bluesky/hklpy/blob/dd2cf5a0d32e73f04d8ec5a223faff7e6f9593a2/environment.yml#L17

Review and refactor to relax this pin, such that databroker v2 can be considered, in addition to databroker v1.2.

prjemian commented 1 year ago

Where is it used?

$ git grep import | grep databroker
docs/source/examples/notebooks/geo_e4cv.ipynb:    "import databroker\n",
docs/source/examples/notebooks/geo_e6c.ipynb:    "import databroker\n",
docs/source/examples/notebooks/geo_k4cv.ipynb:    "import databroker\n",
docs/source/examples/notebooks/tst_UB_in_descriptor_document.ipynb:    "import databroker\n",
docs/source/examples/notebooks/tst_UB_in_stream.ipynb:    "import databroker\n",
hkl/tests/test_save_restore_UB.py:import databroker
prjemian commented 1 year ago

Documentation build uses the development environment: https://github.com/bluesky/hklpy/blob/dd2cf5a0d32e73f04d8ec5a223faff7e6f9593a2/.github/workflows/publish-docs.yml#L34-L38

prjemian commented 1 year ago

The unit testing uses the packaging environment: https://github.com/bluesky/hklpy/blob/dd2cf5a0d32e73f04d8ec5a223faff7e6f9593a2/.github/workflows/conda_unit_test.yml#L60-L73

Change this to the development environment.

prjemian commented 1 year ago

Docs build with just databroker and no pin. Pytest stops with an internal problem with a test:

hkl/tests/test_save_restore_UB.py::test_fourc_orientation_save 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/main.py", line 271, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/main.py", line 325, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/main.py", line 350, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_callers.py", line 152, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/runner.py", line 114, in pytest_runtest_protocol
INTERNALERROR>     runtestprotocol(item, nextitem=nextitem)
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/runner.py", line 133, in runtestprotocol
INTERNALERROR>     reports.append(call_and_report(item, "call", log))
INTERNALERROR>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/runner.py", line 224, in call_and_report
INTERNALERROR>     report: TestReport = hook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_hooks.py", line 493, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_manager.py", line 115, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_callers.py", line 130, in _multicall
INTERNALERROR>     teardown[0].send(outcome)
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/skipping.py", line 266, in pytest_runtest_makereport
INTERNALERROR>     rep = outcome.get_result()
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_result.py", line 114, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/pluggy/_callers.py", line 77, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/runner.py", line 368, in pytest_runtest_makereport
INTERNALERROR>     return TestReport.from_item_and_call(item, call)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/reports.py", line 362, in from_item_and_call
INTERNALERROR>     longrepr = item.repr_failure(excinfo)
INTERNALERROR>                ^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/python.py", line 1833, in repr_failure
INTERNALERROR>     return self._repr_failure_py(excinfo, style=style)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/nodes.py", line 486, in _repr_failure_py
INTERNALERROR>     return excinfo.getrepr(
INTERNALERROR>            ^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/_code/code.py", line 701, in getrepr
INTERNALERROR>     return fmt.repr_excinfo(self)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/_code/code.py", line 989, in repr_excinfo
INTERNALERROR>     reprtraceback = self.repr_traceback(excinfo_)
INTERNALERROR>                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/_code/code.py", line 913, in repr_traceback
INTERNALERROR>     entries = [
INTERNALERROR>               ^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/_code/code.py", line 914, in <listcomp>
INTERNALERROR>     self.repr_traceback_entry(entry, excinfo if last == entry else None)
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/_code/code.py", line 860, in repr_traceback_entry
INTERNALERROR>     s = self.get_source(source, line_index, excinfo, short=short)
INTERNALERROR>         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/_code/code.py", line 794, in get_source
INTERNALERROR>     lines.extend(self.get_exconly(excinfo, indent=indent, markall=True))
INTERNALERROR>                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/_code/code.py", line 806, in get_exconly
INTERNALERROR>     exlines = excinfo.exconly(tryshort=True).split("\n")
INTERNALERROR>               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/_pytest/_code/code.py", line 605, in exconly
INTERNALERROR>     lines = format_exception_only(self.type, self.value)
INTERNALERROR>             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/traceback.py", line 159, in format_exception_only
INTERNALERROR>     te = TracebackException(type(value), value, None, compact=True)
INTERNALERROR>          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/site-packages/trio/_core/_multierror.py", line 393, in traceback_exception_init
INTERNALERROR>     traceback_exception_original_init(
INTERNALERROR>   File "/home/prjemian/.conda/envs/dev-hklpy/lib/python3.11/traceback.py", line 757, in __init__
INTERNALERROR>     context = TracebackException(
INTERNALERROR>               ^^^^^^^^^^^^^^^^^^^
INTERNALERROR> TypeError: traceback_exception_init() got an unexpected keyword argument 'max_group_width'
prjemian commented 1 year ago

locally, it fails when running RE(bp.count([det]))

ModuleNotFoundError: No module named 'suitcase.mongo_normalized'

and

(dev-hklpy) prjemian@arf:~/.../Bluesky/hklpy$ conda list suitcase
# packages in environment at /home/prjemian/.conda/envs/dev-hklpy:
#
# Name                    Version                   Build  Channel
suitcase                  0.12.0                   pypi_0    pypi
prjemian commented 1 year ago

Here's a condensed version that reproduces this failure:

import bluesky.plans as bp
import databroker
from bluesky import RunEngine
from ophyd.sim import hw

cat = databroker.temp().v2
RE = RunEngine()
RE.subscribe(cat.v1.insert)

det = hw().noisy_det

_uids = RE(bp.count([det]))
print(f"{_uids=}")

Note in environment with databroker==1.2, this code executes without error.

Is databroker 2 ready for this use?

prjemian commented 1 year ago

locally, it fails when running RE(bp.count([det]))

ModuleNotFoundError: No module named 'suitcase.mongo_normalized'

and

(dev-hklpy) prjemian@arf:~/.../Bluesky/hklpy$ conda list suitcase
# packages in environment at /home/prjemian/.conda/envs/dev-hklpy:
#
# Name                    Version                   Build  Channel
suitcase                  0.12.0                   pypi_0    pypi

Installed suitcase via pip and the original error persists.

prjemian commented 1 year ago

Not suitcase, but these conda-forge packages are needed:

$ conda list suitcase
# packages in environment at /home/prjemian/.conda/envs/dev-hklpy:
#
# Name                    Version                   Build  Channel
suitcase-jsonl            0.2.2              pyhd8ed1ab_0    conda-forge
suitcase-mongo            0.4.0              pyhd8ed1ab_0    conda-forge
suitcase-msgpack          0.3.0              pyhd8ed1ab_1    conda-forge
suitcase-utils            0.5.4              pyhd8ed1ab_0    conda-forge
prjemian commented 1 year ago

Now, getting same error: unexpected keyword argument 'max_group_width' but root cause is different: TypeError: Type is not JSON serializable: _ResultTuple and that tracks back to the diffractometer constraints being reported as a numpy array.

prjemian commented 1 year ago

The export_dict() method might be the way to resolve.

prjemian commented 1 year ago

Constraints are stored in a Python class and rendered for databroker. Here: https://github.com/bluesky/hklpy/blob/dd2cf5a0d32e73f04d8ec5a223faff7e6f9593a2/hkl/diffract.py#L514-L530

It is the ophyd.ArrayAttributeSignal that converts them into the incompatible numpy array:

fourc._constraints_for_databroker=[(-180.0, 180.0, 0.0, True), (-180.0, 180.0, 0.0, True), (-180.0, 180.0, 0.0, True), (-180.0, 180.0, 0.0, True)]
type(fourc._constraints_for_databroker)=<class 'list'>
fourc._constraints.get()=array([[-180.,  180.,    0.,    1.],
       [-180.,  180.,    0.,    1.],
       [-180.,  180.,    0.,    1.],
       [-180.,  180.,    0.,    1.]])
type(fourc._constraints)=<class 'ophyd.signal.ArrayAttributeSignal'>
type(fourc._constraints.get())=<class 'numpy.ndarray'>
padraic-shafer commented 1 year ago

Did using _constraints_for_databroker() solve this?

It seems like the right way to go. If I understand correctly the RE callback you've used, cat.v1.insert(), calls cat. post_document(), which in turn uses HTTP POST that expects the doc to be JSON serializable.

Somewhere in the stack (either in the ophyd signal get or in the run engine event processing) does the JSON serializer need to be customized for ndarray?

prjemian commented 12 months ago

It is ophyd.signal.ArrayAttributeSignal() that converts the structure [(float, float, float, float), (float, float, float, float)] into ndarray: https://github.com/bluesky/hklpy/blob/dd2cf5a0d32e73f04d8ec5a223faff7e6f9593a2/hkl/diffract.py#L217-L222

The RE is emitting a descriptor document that comes from DIFFRACTOMETER._constraints.read_configuration(). This document is published to cat.v1.insert(). As I understand it, one (or more) suitcase packages are involved in serializing the document stream. Somewhere, this document was not handled.

padraic-shafer commented 12 months ago

I thought that suitcase is no longer used in databroker v2, and that the v1 insert is just a wrapper around tiled.

Don't quote me on that, but that is my current (limited) understanding :)

prjemian commented 12 months ago

I believe you are correct about suitcase. However, suitcase-jsonl, et al. are in the databroker v2b requirements for testing, and then suitcase-utils is an included requirement. Without them, testing of hklpy stops, as reported above with the INTERNALERROR.

prjemian commented 12 months ago

It's not just the constraints, other content, such as the UB matrix, are also passed in the descriptor document as ndarray.

prjemian commented 12 months ago

Also reported on Mattermost.

@danielballan Can you advise?

prjemian commented 12 months ago

Reviewing c = from_uri("http://localhost:8000?api_key=secret"), leads me to

so having some success with:

from tiled.client import from_uri
URI = (
    "http://localhost:8020"
    "?"
    "api_key=abcdefghijklmonp1234567890"  # get from tiled server startup
)
client = from_uri(URI)
cat = client["training"]
RE = RunEngine()
# RE.subscribe(cat.v1.insert)
RE.subscribe(cat.post_document)

Is this the new direction to go?

prjemian commented 11 months ago

Note in #276, the pin was moved to an optional dependency. Still the work to relax the pin is needed.

prjemian commented 11 months ago

Discussion on Mattermost confirmed that tiled.client is (will become) the new SOP.

This snippet was added as a suggestion:

from tiled.client import from_uri
from databroker import Broker

# For security, set the API key by setting env var
# TILED_API_KEY rather than putting it in code.
client = from_uri("http://localhost:8020")
cat = client["training"]
db = Broker(cat)  # wrap in a databroker backward-compatible shim
RE.subscribe(db.insert)
prjemian commented 11 months ago

@danielballan adds: The earlier version (with cat.post_document) ...

has the advantage of avoiding a (slow) databroker import

prjemian commented 11 months ago

It is not necessary to switch the CI to use a tiled server at this time. The tests run with databroker v2.0.0b30 and this catalog setup: https://github.com/bluesky/hklpy/blob/f2d72632238ac51aa5a153a6bb7463ac07a54394/hkl/tests/test_util.py#L9-L13