dev -> main

[pperanich]

Summary of Changes

The following are a list of changes proposed for merging the dev branch into main:

Packaging changes

The src layout is a modern standard that isolates our source code from other project resources. This change minimizes the risk of accidental inclusion of unwanted files in the distribution, leads to cleaner packaging and install processes, and contributes to more reliable testing practices. For our developers and contributors, this translates to a more straightforward and error-resistant development workflow.

In conjunction with the migration to a src layout, ezmsg is also excited to convert to poetry for our packaging and dependency management needs. Poetry is a cutting-edge tool that offers several advantages over traditional packaging systems. Read about it here

Core Changes

• Bumped version to 3.3.4 for minor fixes and improvements.
• Improved debugging by adding log line for current log level in __init__.py.
• Enhanced logging for thread shutdown in backendprocess.py.
• Ensured graph_service initialization precedes shm_service in graphcontext.py.
• Enhanced subscriber notifications to include interface information in graphserver.py.
• Broadened exception catching in server.py to handle generic OSError.
• Added error handling for shared memory closure to shmserver.py.
• Simplified shared memory reattachment in subclient.py.
• Several refactors addressing graph context management, code readability, and FIXME comments in backend.py.
• BACKWARD INCOMPATIBILITY: Changes to underlying message logger format breaks loading old message logs.

Extensions (ezmsg-zmq)

• Version of ezmsg-zmq extension has been bumped to 1.1.5.
• Updated ezmsg-zmq with a wait_for_sub setting to control subscriber wait behavior in ZMQSenderUnit.
• Removed poller STATE variable from ZMQSenderUnit

Extensions (ezmsg-sigproc)

• Version of ezmsg-sigproc extension has been bumped to 1.2.3.
• Improvements to the ButterworthFilter to enforce type checking for message processing.
• Refactor of ewmfilter to support scale array broadcasting and customizable zero_offset.
• Enhancement of SampleTriggerMessage hashability by setting unsafe_hash to True.
• Adjustment in Spectrum class to rename output axis based on settings.
• Replaced Oscillator with Counter in test_downsample.py to better align with test requirements and added new assertions.
• Introduction of extensive unit tests for the Counter synthesizer.
• Introduction/reimplementation of almost every signal processing unit as a generator to allow for more comprehensive unit testing and better notebook experiences.
• Extensive unit testing (thank you @cboulay and Wyss!)

Utilities and Version Bumping

• Implemented message output rate limiting in MessageQueue through output_hz parameter.
• Fixed time jump handling in Rate utility.
• Added new transformation utility gen_to_unit for generator to async ezmsg Unit conversion.
• Additional generator utilities introduced into the ezmsg utility module.

Documentation and Examples

• Enhanced the README.md with installation instructions, a list of external extensions, and a new Publications section.
• Introduced an example script ezmsg_generator.py that showcases how to encapsulate computational logic within a generator and integrate it with ezmsg.

Acknowledgments

Thanks to the contributors who have played a role in enhancing the robustness and feature set of ezmsg.

[cboulay]

Some of my recent PRs broke the tests. I think it's just a missing import. I'll fix it tomorrow.

[cboulay]

Please merge #89 so the test runner can try again.

[cboulay]

One of my tests is still causing a problem. I will get to it when I'm done my current effort on gen_to_unit.

[cboulay]

Looking through the test failures, I can see that at least one of them is my fault. In some of the tests I wrote (e.g. modify_axis), I used typing.Optional[dict[key_type, value_type]] as a type. I guess this should be typing.Optional[typing.Dict[key_type, value_type]]. I can never remember what version of Python supports which plain types in type hints. Please let me know if you want me to make a pass over all of those, or if you want to do it.

P.S. I'd love to use int | None instead of typing.Optional[int]. But I guess requiring min py 3.10 leaves behind a little too much.

[pperanich]

I believe type hinting generics were introduced in Python 3.9 via PEP585. For now, I think we plan to continue Python 3.8 support. Maybe we can revisit once 3.8 hits EOL (October 2024).

[cboulay]

The most recent test failures are due to changes in MessageLogger. When reading back the logs, ezmsg.util.messagecodec message_log()... in the for loop the obj is always a dict now I guess, so the isinstance(obj, LogStart) is always False. This line can be changed to if isinstance(obj["obj"], LogStart) and all the tests pass. But I don't know if that's how the logger is intended to work.

[griffinmilsap]

Thanks for the heads up on message_log @cboulay. I think tests are passing now, so I think we're ready for review. I've also drafted a release tag in the releases section. I'm going to mark this PR as "Ready for Review".

FYI @pperanich and @cboulay -- There's a github action in this repository that uses a secret in the repo to update the PyPI packages on a new tagged version in main. If the internal version number of the new packages is not the same as the packages already there, it will upload new versions automatically and update the public package repository. Any developer or maintainer is able to push packages directly to PyPI. I'm happy to publish what's here now, but please feel free to review and I'll hit the button myself tomorrow afternoon if nobody else has hit it by then.

🍺 Cheers yall, every new release feels like quite a feat!

[griffinmilsap]

Alas, poor pull request. We hardly knew ye.

Actually scratch that, this PR's been open for months and got squished by https://github.com/isaacs/github/issues/361.

So it goes.

New PR #101 ahoy!