V3.3.0

[griffinmilsap]

ezmsg.core v3.3.0

• Better extension packaging (with consistent naming for single-file extensions)
• Better flow control with asyncio.sleep(0) after every publish. Note: this halves performance of the perf test, but maintains flow without constant backpressure warnings for systems that publish really fast
• New message utilities in ezmsg.util
  • ezmsg.util.messages.axisarray -- AxisArray fully fleshed out
    • Includes xarray-like sel and isel functionality for slices along dimensions
  • Finally removed StampedMessage
  • ezmsg.util.terminate -- TerminateOnTotal (messages received) and TerminateOnTimeout units introduced
  • ezmsg.util.messagecodec -- Refactored JSONEncoder and JSONDecoder for ezmsg.util.messagelogger. This codec now properly serializes/deserializes message dataclasses just like pickle, but with a human-readable plaintext message log format! NB. MessageEncoder and MessageDecoder moved here from ezmsg.util.messagelogger and the json file format has changed ever-so-slightly (and backward compatibly) to support this.
  • ezmsg.util.messagereplay -- Introduces a MessageReplay unit that can deserialize and publish messages from message files written by MessageLogger. Also introduces a MessageCollector that just saves all received messages to a list in the STATE; can be accessed after execution completes from a notebook if run in the main process.
  • ezmsg.util.messagequeue -- HUGE enhancement that allows ezmsg systems to run with "squishiness" and "leakiness" -- allowing your system to accumulate a backlog of messages (that leaks if/when you want) to prevent a slow downstream unit from backing up your whole system.
• ez.run
  • force_single_process keyword argument which tries to run provided components all in one process -- particularly useful for notebook operation
  • Can now pass multiple components to run, and connect them directly
• GraphServer and SHMServer Enhancements
  • Servers are now run in threads with dedicated event loops as opposed to separate processes. This dramatically speeds up startup on Windows
  • Servers will check for $EZMSG_GRAPHSERVER_ADDR and $EZMSG_SHMSERVER_ADDR (strings formatted as hostname:12345) environment variables -- and if defined, ezmsg will force a connection to these addresses.
  • Reserved ports for GraphServer, SHMServer are 25978 and 25979 respectively. If the environment variables are not defined, ezmsg will first check these reserved ports on 127.0.0.1 and try to connect to servers running there. the ezmsg start command starts servers on the default ports and is a handy way for multiple systems on the same PC to dynamically interact with each-other
  • If the environment variables are not defined and there are no servers running on the default ports, ezmsg will force-start new servers on unoccupied ports starting from 10000 (or $EZMSG_SERVER_PORT_START) -- but avoiding default ports for the services.
• Command Line Changes
  • ezmsg start now reports forked PID
  • ezmsg graphviz writes a DOT-formatted graph specification to stdout; can be piped to graphviz to visualize the current running graph.
• Fixed bugs
  • Moved __version__ attribute to ezmsg.version fix pip install
  • "Single-process-mode" isn't special anymore; main process is always used for execution of one group of units
  • ezmsg will always start a new threaded eventloop for tasks; because of @ez.main code
  • Better KeyboardInterrupt handling (especially on Windows)
  • Unit.Initialize() and creation of State is now called from the task thread (and asyncio synchronization primitives will automatically be associated with the correct event loop
  • Significantly cleaned up code pertaining to connections to the GraphServer and SHMServer; reduced redundant code between these services significantly as well.
  • No-longer forcing WindowsSelectorEventLoop on Windows for asyncio.
  • All leaked ResourceWarnings resolved; all asyncio operations now threadsafe-ly implemented.
  • Resolved ConnectionResetError and BrokenPipeError on shutdown.
  • Settings now decorated with @dataclass_transform and type-checking should resolve settings fields appropriately.
    • This required introducing ezmsg's FIRST non-stdlib dependency; typing_extensions which is required to backport this typing PEP to Python 3.8. typing_extensions has no external deps and is pure-python; unlikely to break in the future.
  • Settings classes no-longer automatically generate unsafe_hash functions. This behavior can be reintroduced by creating your own dataclass definition with unsafe_hash=True, frozen=True and inheriting from Settings.
  • Tasks/Publishers/Subscribers/Threads, etc.. now have attributes that are less likely to collide with user-code
  • Deprecated ezmsg.testing

Extensions

ezmsg.sigproc v1.2.0

• All signal processing modules rewritten to operate on AxisArrays
  • All units allow you to specify named axis to operate on (usually time axis)
• New spectrum (FFT) unit in ezmsg.sigproc.spectral
• Window now allows newaxis accumulation for batching
• TimeseriesMessage deleted
• TSMessage is now a function that returns a properly formatted AxisArray

ezmsg.websocket v1.1.1

• __version__ attribute
• Moved units from ezmsg.websocket to ezmsg.websocket.units
• Added a pyproject.toml

ezmsg.zmq v1.1.1

• __version__ attribute
• Moved units from ezmsg.zmq.sender and ezmsg.zmq.poller to ezmsg.zmq.units
• Added a pyproject.toml

[pperanich]

@griffinmilsap I'm starting to think that ezmsg shutdown should still be included with a confirmation step to make sure the user is doing so intentionally. Additionally, it may be possible to check how many graph connections are still active at that time and notify the user before confirmation prompt.

The alternative is that a user may still maliciously kill a running server with a kill -9 command, so I'm not sure exactly what we are gaining by removing this command entirely.