search

VOLTTRON / volttron

VOLTTRON Distributed Control System Platform
https://volttron.readthedocs.io/
Other
456 stars 216 forks source link

VIP bytestring automatic handling not supported. #2508

Open MachinesJesus opened 3 years ago

MachinesJesus commented 3 years ago

Description of Issue

VIP does not automagically handle bytestrings. Would be good to add this.

Sure I can handle this but I think volttron should / could detect bytestrings and support via VIP or just convert, so volttron does not break. Why? Encoding / Secrecy and Other packages may return bytestrings (aws).

Also, Protobuf seems to not care bytestring vs string. So if we get volttron to do an auto-conversion we should get a pretty good overall system / user friendliness.

Affected Version

Using 7.0 but can test whatever you need.

Screenshots

Expected

As expected.

Actual

Steps to Reproduce

Send bytestring over VIP message = data.encode('utf-8')

Additional Details

Error Log: 2020-11-02 11:28:23,969 (awssagent-0.1 8076) ERROR: Exception ignored in: <class 'SystemExit'> 2020-11-02 11:28:23,969 (awssagent-0.1 8076) ERROR: Traceback (most recent call last): 2020-11-02 11:28:23,969 (awssagent-0.1 8076) ERROR: File "/home/ubuntu/te/vtrn/env/lib/python3.8/site-packages/awscrt/mqtt.py", line 477, in callback_wrapper 2020-11-02 11:28:23,969 (awssagent-0.1 8076) ERROR: callback(topic=topic, payload=payload) 2020-11-02 11:28:23,969 (awssagent-0.1 8076) ERROR: File "/home/ubuntu/.volttron/agents/8f65a133-2c14-45b5-a701-fda9b7e956bc/awssagent-0.1/awss/agent.py", line 173, in _receive_awsiot_message 2020-11-02 11:28:23,969 (awssagent-0.1 8076) ERROR: self.vip.pubsub.publish('pubsub', topic=topic, message=payload) # See note below. 2020-11-02 11:28:23,969 (awssagent-0.1 8076) ERROR: File "/home/ubuntu/te/vtrn/volttron/platform/vip/agent/subsystems/pubsub.py", line 641, in publish 2020-11-02 11:28:23,969 (awssagent-0.1 8076) ERROR: self.vip_socket.send_vip('', 'pubsub', args, result.ident, copy=False) 2020-11-02 11:28:23,970 (awssagent-0.1 8076) ERROR: File "/home/ubuntu/te/vtrn/volttron/platform/vip/socket.py", line 471, in send_vip 2020-11-02 11:28:23,970 (awssagent-0.1 8076) ERROR: send(args, flags=flags, copy=copy, track=track) 2020-11-02 11:28:23,970 (awssagent-0.1 8076) ERROR: File "/home/ubuntu/te/vtrn/volttron/platform/vip/socket.py", line 415, in send_multipart 2020-11-02 11:28:23,970 (awssagent-0.1 8076) ERROR: parts = serialize_frames(msg_parts) 2020-11-02 11:28:23,970 (awssagent-0.1 8076) ERROR: File "/home/ubuntu/te/vtrn/volttron/utils/frame_serialization.py", line 109, in serialize_frames 2020-11-02 11:28:23,970 (awssagent-0.1 8076) ERROR: sys.exit(0) 2020-11-02 11:28:23,970 (awssagent-0.1 8076) ERROR: SystemExit: 0

craig8 commented 3 years ago

This should be fixed in the latest develop codebase.

One should be able to pass anything serializable through the bus.

MachinesJesus commented 3 years ago

Cool cool. OK will pull down the latest and test it.

craig8 commented 3 years ago

You could use main or develop...both should work

MachinesJesus commented 3 years ago

@craig8 tried 8.0rc1 and develop. Both fail to pass our Bytes. Would be great to get this working. The issue seems to be on the frame breakup.

"2020-11-11 10:55:07,725 (awssagent-0.1 13580) ERROR: File "/home/ubuntu/te/vtrn/volttron/utils/frame_serialization.py", line 109, in serialize_frames"

Thoughts: Do you breakup the messages? Appears to expect a List. Into what size? Because I could potentially build this into protobuff. Appears to have no size limits

Attempted to alter the fram_serialization to 'utf-8' this causes an error on re-install.

2020-11-11 11:02:40,223 () volttron.utils.frame_serialization ERROR: Unicode decode error: 'utf-8' codec can't decode byte 0xbf in position 2: invalid start byte

craig8 commented 3 years ago

@MachinesJesus can you reach out to me on slack so we can schedule a time...I would love to chat more about the protobufs you mention here. I have been looking into how I would utilize those for this type of issue. Maybe you have some insight for me?

MachinesJesus commented 3 years ago

@craig8 sent you a slack message.

For any users who want to use awsiot and protobuff.

Do this: