'buffer too small for requested bytes' during Connect()

[jackcarrozzo]

What am I doing wrong here?

jackc@wellgoddamn ~ $ python3  
Python 3.4.0 (default, Apr 11 2014, 13:05:11)
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from amqpy import Connection
>>> conn = Connection()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.4/dist-packages/amqpy/connection.py", line 131, in __init__
    self.connect()
  File "/usr/local/lib/python3.4/dist-packages/amqpy/connection.py", line 152, in connect
    self.wait(spec.Connection.Start)
  File "/usr/local/lib/python3.4/dist-packages/amqpy/abstract_channel.py", line 67, in wait
    m = self._wait_method([method])
  File "/usr/local/lib/python3.4/dist-packages/amqpy/abstract_channel.py", line 104, in _wait_method
    method = self.connection.method_reader.read_method()
  File "/usr/local/lib/python3.4/dist-packages/amqpy/method_io.py", line 167, in read_method
    return self._read_method()
  File "/usr/local/lib/python3.4/dist-packages/amqpy/concurrency.py", line 39, in wrapper
    retval = f(self, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/amqpy/method_io.py", line 150, in _read_method
    raise method
  File "/usr/local/lib/python3.4/dist-packages/amqpy/method_io.py", line 66, in _next_method
    frame = self.transport.read_frame()
  File "/usr/local/lib/python3.4/dist-packages/amqpy/concurrency.py", line 39, in wrapper
    retval = f(self, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/amqpy/transport.py", line 179, in read_frame
    payload = self.read(frame.payload_size)
  File "/usr/local/lib/python3.4/dist-packages/amqpy/transport.py", line 328, in read
    return self._read(n, initial, _errnos=(errno.EAGAIN, errno.EINTR))
  File "/usr/local/lib/python3.4/dist-packages/amqpy/transport.py", line 119, in _read
    bytes_read = self.sock.recv_into(mview, to_read)
ValueError: buffer too small for requested bytes
>>> 

root@wellgoddamn ~ # qpidd -v
qpidd (qpidc) version 0.16
root@wellgoddamn ~ # netstat -anp|grep qpidd
tcp        0      0 0.0.0.0:5672            0.0.0.0:*               LISTEN      1635/qpidd      
tcp6       0      0 :::5672                 :::*                    LISTEN      1635/qpidd      
unix  2      [ ]         DGRAM                    16134    1635/qpidd    

Packet log shows some chatter before the error:

root@wellgoddamn ~ # tcpdump -Ani lo port 5672
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
12:51:14.510843 IP 127.0.0.1.36665 > 127.0.0.1.5672: Flags [S], seq 2322166516, win 43690, options [mss 65495,sackOK,TS val 2808828482 ecr 0,nop,wscale 7], length 0
E..< .@.@............9.(.ir..........0.........
.kRB........
12:51:14.510863 IP 127.0.0.1.5672 > 127.0.0.1.36665: Flags [S.], seq 2217121944, ack 2322166517, win 43690, options [mss 65495,sackOK,TS val 2808828482 ecr 2808828482,nop,wscale 7], length 0
E..<..@.@.<..........(.9.&...ir......0.........
.kRB.kRB....
12:51:14.510874 IP 127.0.0.1.36665 > 127.0.0.1.5672: Flags [.], ack 1, win 342, options [nop,nop,TS val 2808828482 ecr 2808828482], length 0
E..4 .@.@............9.(.ir..&.....V.(.....
.kRB.kRB
12:51:14.510919 IP 127.0.0.1.36665 > 127.0.0.1.5672: Flags [P.], seq 1:9, ack 1, win 342, options [nop,nop,TS val 2808828482 ecr 2808828482], length 8
E..< .@.@............9.(.ir..&.....V.0.....
.kRB.kRBAMQP..  .
12:51:14.510935 IP 127.0.0.1.5672 > 127.0.0.1.36665: Flags [.], ack 9, win 342, options [nop,nop,TS val 2808828482 ecr 2808828482], length 0
E..4..@.@.q..........(.9.&...ir....V.(.....
.kRB.kRB
12:51:14.511016 IP 127.0.0.1.5672 > 127.0.0.1.36665: Flags [P.], seq 1:9, ack 9, win 342, options [nop,nop,TS val 2808828482 ecr 2808828482], length 8
E..<..@.@.q..........(.9.&...ir....V.0.....
.kRB.kRBAMQP...

12:51:14.511066 IP 127.0.0.1.5672 > 127.0.0.1.36665: Flags [F.], seq 9, ack 9, win 342, options [nop,nop,TS val 2808828482 ecr 2808828482], length 0
E..4..@.@.q..........(.9.&...ir....V.(.....
.kRB.kRB
12:51:14.511107 IP 127.0.0.1.36665 > 127.0.0.1.5672: Flags [.], ack 9, win 342, options [nop,nop,TS val 2808828482 ecr 2808828482], length 0
E..4 .@.@..     .........9.(.ir..&.....V.(.....
.kRB.kRB
12:51:14.549965 IP 127.0.0.1.36665 > 127.0.0.1.5672: Flags [.], ack 10, win 342, options [nop,nop,TS val 2808828492 ecr 2808828482], length 0
E..4 .@.@............9.(.ir..&.....V.(.....
.kRL.kRB
^C
9 packets captured
18 packets received by filter
0 packets dropped by kernel

[veegee]

It looks like you have a really old version of Qpid. The latest is 0.32 https://qpid.apache.org/download.html. I tested on a recent version of Qpid a while ago, and it seemed to work fine. I'll try again on the latest version tonight, as well as your version if I can get a hold of it.

I think you may be using the packaged version on Ubuntu or something. http://packages.ubuntu.com/vivid/qpidd

The way the protocol works is that it negotiates a frame size when connecting. Both the server and client must support a minimum frame size, and my library adheres to that specification. The version of Qpid that you have may not be respecting the AMQP 0.9.1 spec.

I'll take a look again tonight, since I use this library in production so I want to make sure there's no bugs whatsoever. In the meantime, in the amqpy.Connection() constructor, try passing in a larger value for frame_max=131072, which is the default. Try making it 1-10MB, but that's an extremely large frame size and shouldn't be required. The server breaks up payloads into multiple frames automatically anyway. In addition, set the python standard logger for "amqpy" to the logging.DEBUG level, and you'll see a lot of useful log messages related to what's going on with the connection.

It could also be that there's something wrong with the Qpid configuration.

You could also try using RabbitMQ to check if everything else is ok first: http://packages.ubuntu.com/vivid/rabbitmq-server. I'm using RabbitMQ in production with this library successfully.

[veegee]

http://amqpy.readthedocs.org/en/latest/

Apache Qpid is confirmed to work, but only with “anonymous” authentication.

Try setting the authentication settings appropriately. This could be due to the server trying to negotiate an unsupported auth mechanism.

[jackcarrozzo]

Built latest qpidd (qpidd (qpid-cpp) version 0.32) from source; started with --auth=no which, while not specifically mentioning "anonymous" authentication, does imply that it disables connection auth. Same thing.

Making the frame_size much larger didn't affect the result. I did add DEBUG logging, but unfortunately this is all I get prior to the exception:

DEBUG:amqpy:amqpy 0.11.0 Connection.__init__()

I'll do some more fooling with it tomorrow and see what I can track down.

[veegee]

https://qpid.apache.org/components/cpp-broker/index.html

After revisiting that page, it appears that the C++ broker only supports AMQP 0.10 and AMQP 1.0, which are not the most popular protocols. The Java broker supports AMQP 0.9.1.

So this is not a bug, just an incompatible AMQP version. All the big servers (RabbitMQ, Qpid Java, and others) all support 0.9.1.

[jackcarrozzo]

Ahh - you are correct, my fault. I mistakenly assumed that the cpp broker was compatible with both 0.9.1 and 1.0.

[veegee]

Yeah, AMQP version numbers were so poorly chosen. Decent protocol though.