datetime object issue on VOLTTRON message bus?

[bbartling]

I ran into an issue testing with openleadr server to VOLTTRON VEN on when the open ADR signal gets printed to the message bus and then consumed again by another agent the datetime objects were None

For example this is the open ADR signal on the log from the open ADR VEN agent where the signal looks good coming from the VTN server: Logged inside the open ADR VEN agent before the event is published:

2022-07-07 20:00:50,869 (volttron_openadr_ven-1.0.1.dev5 3099009) __main__ INFO: [Bens VEN debug] - INCOMING ADR MESSAGE!!!!: {'event_descriptor': {'event_id': '1b6cbfef-98cb-410c-8c75-4069d6bde3f8', 'modification_number': 0, 'modification_date_time': datetime.datetime(2022, 7, 7, 20, 0, 50, 484266, tzinfo=datetime.timezone.utc), 'priority': 0, 'market_context': 'oadr://unknown.context', 'created_date_time': datetime.datetime(2022, 7, 7, 20, 0, 50, 484248, tzinfo=datetime.timezone.utc), 'event_status': 'far', 'test_event': False}, 'active_period': {'dtstart': datetime.datetime(2022, 8, 4, 12, 0, 6, tzinfo=datetime.timezone.utc), 'duration': datetime.timedelta(seconds=300)}, 'event_signals': [{'intervals': [{'dtstart': datetime.datetime(2022, 8, 4, 12, 0, 6, tzinfo=datetime.timezone.utc), 'duration': datetime.timedelta(seconds=300), 'uid': 0, 'signal_payload': 1.0}], 'signal_name': 'simple', 'signal_type': 'level', 'signal_id': '7d1c1c83-70aa-41fc-98b7-82e0f43b4731'}], 'targets': [{'ven_id': 'ven_id_geb_edge'}], 'targets_by_type': {'ven_id': ['ven_id_geb_edge']}, 'response_required': 'always'}

But then logging another debug statement on the HVAC agent consuming this message it comes through like this:

2022-07-07 20:00:50,871 (testthreeagentagent-0.1 3098409) __main__ INFO: [Simple DR Agent INFO] - INCOMING ADR MESSAGE!!!!: {'event_descriptor': {'event_id': '1b6cbfef-98cb-410c-8c75-4069d6bde3f8', 'modification_number': 0, 'modification_date_time': None, 'priority': 0, 'market_context': 'oadr://unknown.context', 'created_date_time': None, 'event_status': 'far', 'test_event': False}, **'active_period': {'dtstart': None, 'duration': 300}**, 'event_signals': [{'intervals': [{'dtstart': None, 'duration': 300, 'uid': 0, 'signal_payload': 1.0}], 'signal_name': 'simple', 'signal_type': 'level', 'signal_id': '7d1c1c83-70aa-41fc-98b7-82e0f43b4731'}], 'targets': [{'ven_id': 'ven_id_geb_edge'}], 'targets_by_type': {'ven_id': ['ven_id_geb_edge']}, 'response_required': 'always'}
2022-07-07 20:00:50,871 (listeneragent-3.3 2998418) __main__ INFO: Peer: pubsub, Sender: volttron_openadr_ven-1.0.1.dev5_1:, Bus: , Topic: openadr/event/geb_edge, Headers: {'TimeStamp': '2022-07-07T20:00:50.869963+00:00', 'min_compatible_version': '3.0', 'max_compatible_version': ''}, Message:
{'active_period': {'dtstart': None, 'duration': 300},
'event_descriptor': {'created_date_time': None,
                      'event_id': '1b6cbfef-98cb-410c-8c75-4069d6bde3f8',
                      'event_status': 'far',
                      'market_context': 'oadr://unknown.context',
                      'modification_date_time': None,
                      'modification_number': 0,
                      'priority': 0,
                      'test_event': False},
'event_signals': [{'intervals': **[{'dtstart': None,
                                   'duration': 300,**
                                   'signal_payload': 1.0,
                                   'uid': 0}],
                    'signal_id': '7d1c1c83-70aa-41fc-98b7-82e0f43b4731',
                    'signal_name': 'simple',
                    'signal_type': 'level'}],
'response_required': 'always',
'targets': [{'ven_id': 'ven_id_geb_edge'}]

[bbartling]

Apologize about the formatting between both code snips but the second code snip datetime objects are None for some reason?

[bonicim]

After a pair programming session with @bbartling and John Dyreby, the root cause of datetime objects showing as None on the message bus is that the OpenADRAgent is not properly serializing datetime objects in the message before it is published to the message bus. Currently, the OpenADRAgent serializes datetime objects into "None" as seen in this line of code.

Solution: The OpenADRAgent needs to properly serialize datetime objects in the incoming message from a VTN server.

Tasks:

• Refactor the _parse_event to properly serialize datetime objects
  • For help on refactoring, see this gist from @bbartling's gist.
  • Consider using JSONEncoder to handle the business logic of serializing objects that json.dumps can't serialize; see this example
• Redesign _parse_event to provide default serializing logic to allow users to manually update or add their own serializing logic.
• Update documentation

[bbartling]

Does the forwarding agent or listener serialize datetime objects or time stamps? For example when data is sent to the SQL agent from the forward agent, is there always a time stamp in volttron log? Like the 2022-07-07 20:00:50,871 in the volttron log snip:

2022-07-07 20:00:50,871 (testthreeagentagent-0.1 3098409) __main__ INFO: [Simple DR Agent INFO] - INCOMING ADR MESSAGE!!!!:

How does VOLTTRON typically handle time stamps on the message bus for passing data to a database?

[craig8]

Yes there is a utility function format_timestamp that it calls to put it in iso format.

[bbartling]

Cool should we use that? Any examples for this?

[craig8]

format_timestamp takes a normal datetime object and returns a normalized string representation of the datetime.

# available here in modular version
from volttron.utils import format_timestamp

# format timestamp in a normalized manner
 agent_timing_data[phase] = format_timestamp(get_aware_utc_now())

[bbartling]

Okay I think I get this Gist of this that the from volttron.utils import format_timestamp could be the permanent solution inside the VEN agent to replace this function work around.

[craig8]

The function mentioned has 2 responses

-> [int, str]

if timedelta then return total seconds else if datetime then return formatted string

what happens if it does something else?

[bbartling]

Mark would you know how to answer to Craig?

On Thu, Jul 14, 2022, 12:36 PM Craig @.***> wrote:

The function mentioned has 2 responses

-> [int, str]

if timedelta then return total seconds else if datetime then return formatted string

what happens if it does something else?

— Reply to this email directly, view it on GitHub https://github.com/VOLTTRON/volttron-openadr-ven/issues/35#issuecomment-1184722870, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHC4BHNSNVGGEZSA6OLAG7TVUBF3HANCNFSM53BG746Q . You are receiving this because you were mentioned.Message ID: @.***>

[bonicim]

@bbartling If I understand @craig8's question, I think he is asking what should happen if an object is neither a timedelta or a datetime. In that case, the 'default' function needs to add more logic on how to handle that. You could provide a default option of returning something such as 'None', the name of the object, or some other object. Does that make sense?

[bbartling]

Yeah I think so, that makes sense... I think returning None would be good if the data coming through from openleadr client is not a timedelta or datetime

[bbartling]

In the load shed HVAC agent, I could put a statement that if dstart or duration is None to disregard the event, like its a BAD event and do not load shed. Would the VOLTTRON VEN agent be updated at all? Or should I just work with this as is for what is in the Gist? Hoping to work with the best long term solution for our load shed research for anyone else in the community wanting and load shedding agent code...

[bonicim]

@bbartling I plan to update this agent with these and other changes. Will keep you posted.

[bbartling]

Great thanks @bonicim ! Also would the updated VEN Code include the empty class to include a non-IP Keys VTN Server?

(Like what we have been testing OpenLeadr with OpenLeadr....)

[bonicim]

Great thanks @bonicim ! Also would the updated VEN Code include the empty class to include a non-IP Keys VTN Server? (Like what we have been testing OpenLeadr with OpenLeadr....)

@bbartling To clarify, are you suggesting to include a class simliar to your gist?

[bbartling]

Well maybe something for a non IP Keys server....

For us it's open leadr client and server...

On Fri, Aug 12, 2022, 11:26 AM Mark Bonicillo @.***> wrote:

Great thanks @bonicim https://github.com/bonicim ! Also would the updated VEN Code include the empty class to include a non-IP Keys VTN Server? (Like what we have been testing OpenLeadr with OpenLeadr....)

@bbartling https://github.com/bbartling To clarify, are you suggesting to include a class simliar to your gist https://gist.github.com/bbartling/d2d719e58afb5f32227cc810a0c75414#file-main-py ?

— Reply to this email directly, view it on GitHub https://github.com/VOLTTRON/volttron-openadr-ven/issues/35#issuecomment-1213299254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHC4BHOSTB2SPA7DURQQGKLVYZ3KJANCNFSM53BG746Q . You are receiving this because you were mentioned.Message ID: @.***>

[bonicim]

@bbartling Just submitted #36 to address this issue.

[bonicim]

Great thanks @bonicim ! Also would the updated VEN Code include the empty class to include a non-IP Keys VTN Server? (Like what we have been testing OpenLeadr with OpenLeadr....)

@bbartling If you would like to see this in the next release, please submit a new issue on Github so we can track this request. Thanks in advance.

[bonicim]

In the load shed HVAC agent, I could put a statement that if dstart or duration is None to disregard the event, like its a BAD event and do not load shed. Would the VOLTTRON VEN agent be updated at all? Or should I just work with this as is for what is in the Gist? Hoping to work with the best long term solution for our load shed research for anyone else in the community wanting and load shedding agent code...

So in my #36, for objects that don't get serialized, we'll set it to None.