search

smok-serwis / coolamqp

Pure Python pretty fast AMQP client with sane reconnects
MIT License
6 stars 1 forks source link

Bug spotted #23

Closed piotrmaslanka closed 6 years ago

piotrmaslanka commented 7 years ago 
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/smok4/core/tests/test_shci/test_drivers/test_otbox/test_drvobj.py", line 17, in test_get_parameters_curacao
    dev = create_nothingdevice(please='shci/otbox')
  File "/builds/smok4/core/tests/utils.py", line 49, in create_nothingdevice
    install(device_id, True, 'flow')
  File "/builds/smok4/core/shci/drivers/otbox/install.py", line 170, in install
    Predicate.create(device_id, 'opentherm', 'OT', {}).start()
  File "/builds/smok4/core/sai/alarms/predicate.py", line 153, in create
    start.run(device_id, new_id)
  File "/usr/local/lib/python2.7/dist-packages/loveseal/task.py", line 163, in run
    self.region.amqp.publish(msg, routing_key=self.region.deadletter_queue, tx=True).result()
  File "/builds/smok4/core/sai/basic/msgs.py", line 83, in publish
    return super(_AMQP, self).publish(msg, exchange, routing_key, tx=tx)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/clustering/cluster.py", line 144, in publish
    return (self.pub_tr if tx else self.pub_na).publish(message, exchange, routing_key)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/attaches/utils.py", line 231, in monitored
    return fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/attaches/publisher.py", line 218, in publish
    self._mode_cnpub_process_deliveries()
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/attaches/publisher.py", line 157, in _mode_cnpub_process_deliveries
    self.tagger.deposit(self.tagger.get_key(), FutureConfirmableRejectable(fut))
AttributeError: 'NoneType' object has no attribute 'deposit'
-------------------- >> begin captured logging << --------------------
coolamqp.uplink.connection.connection: DEBUG: TCP connection established, authentication in progress
sai.basic.msgs: INFO: Finished AMQP setup
coolamqp.attaches.publisher: DEBUG: Publish request, but not connected - dropping the message
coolamqp.uplink.connection.connection: DEBUG: Received connection.start
coolamqp.uplink.connection.connection: DEBUG: Received connection.tune
coolamqp.uplink.connection.connection: DEBUG: Received connection.open-ok
coolamqp.uplink.connection.connection: INFO: Connection ready.
coolamqp.uplink.connection.connection: DEBUG: Received channel.open-ok
coolamqp.uplink.connection.connection: DEBUG: Received channel.open-ok
coolamqp.attaches.publisher: DEBUG: Publisher on_setup, payload=<coolamqp.framing.definitions.ChannelOpenOk object at 0x7ff79c5c1410>
coolamqp.attaches.publisher: INFO: Publisher noack is up
coolamqp.uplink.connection.connection: DEBUG: Received channel.open-ok
coolamqp.attaches.publisher: DEBUG: Publisher on_setup, payload=<coolamqp.framing.definitions.ChannelOpenOk object at 0x7ff79c5c1410>
coolamqp.uplink.connection.connection: DEBUG: Received confirm.select-ok
coolamqp.attaches.publisher: DEBUG: Publisher on_setup, payload=<coolamqp.framing.definitions.ConfirmSelectOk object at 0x7ff79c536b10>
--------------------- >> end captured logging << ---------------------
piotrmaslanka commented 7 years ago

Additional sighting is:

Exception caught on 2017-05-17T11:43:26.563234:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/smok4_core-4.4.5-py2.7.egg/sai/basic/__init__.py", line 73, in run_with_except_hook
    run_old(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/uplink/listener/thread.py", line 30, in run
    self.listener.wait(timeout=1)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/uplink/listener/epoll_listener.py", line 98, in wait
    sock.on_fail()
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/uplink/listener/socket.py", line 47, in on_fail
    self._on_fail()
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/uplink/connection/connection.py", line 161, in on_fail
    self.finalize()
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/objects.py", line 33, in __call__
    callable(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/objects.py", line 33, in __call__
    callable(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/clustering/single.py", line 46, in _on_fail
    self.connect()
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/clustering/single.py", line 37, in connect
    self.attache_group.attach(self.connection)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/attaches/agroup.py", line 68, in attach
    attache.attach(connection)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/attaches/utils.py", line 230, in monitored
    return fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/attaches/publisher.py", line 99, in attach
    Channeler.attach(self, connection)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/attaches/channeler.py", line 79, in attach
    super(Channeler, self).attach(connection)
  File "/usr/local/lib/python2.7/dist-packages/coolamqp/attaches/channeler.py", line 39, in attach
    assert self.connection is None
AssertionError

* Stack trace, innermost first
** attach at /usr/local/lib/python2.7/dist-packages/coolamqp/attaches/channeler.py:39
*** connection: <coolamqp.uplink.connection.connection.Connection object at 0x7fd4866c2950>
*** self: <coolamqp.attaches.publisher.Publisher object at 0x7fd4a8414690>
** attach at /usr/local/lib/python2.7/dist-packages/coolamqp/attaches/channeler.py:79
*** connection: <coolamqp.uplink.connection.connection.Connection object at 0x7fd4866c2950>
*** self: <coolamqp.attaches.publisher.Publisher object at 0x7fd4a8414690>
** attach at /usr/local/lib/python2.7/dist-packages/coolamqp/attaches/publisher.py:99
*** connection: <coolamqp.uplink.connection.connection.Connection object at 0x7fd4866c2950>
*** self: <coolamqp.attaches.publisher.Publisher object at 0x7fd4a8414690>
** monitored at /usr/local/lib/python2.7/dist-packages/coolamqp/attaches/utils.py:230
*** fun: <function attach at 0x7fd4ab183d70>
*** args: (<coolamqp.attaches.publisher.Publisher object at 0x7fd4a8414690>, <coolamqp.uplink.connection.connection.Connection object at 0x7fd4866c2950>)
*** kwargs: {}
** attach at /usr/local/lib/python2.7/dist-packages/coolamqp/attaches/agroup.py:68
*** connection: <coolamqp.uplink.connection.connection.Connection object at 0x7fd4866c2950>
*** self: <coolamqp.attaches.agroup.AttacheGroup object at 0x7fd4a8414510>
*** attache: <coolamqp.attaches.publisher.Publisher object at 0x7fd4a8414690>
** connect at /usr/local/lib/python2.7/dist-packages/coolamqp/clustering/single.py:37
*** self: <coolamqp.clustering.single.SingleNodeReconnector object at 0x7fd4a8414610>
** _on_fail at /usr/local/lib/python2.7/dist-packages/coolamqp/clustering/single.py:46
*** self: <coolamqp.clustering.single.SingleNodeReconnector object at 0x7fd4a8414610>
** __call__ at /usr/local/lib/python2.7/dist-packages/coolamqp/objects.py:33
*** callable: <bound method SingleNodeReconnector._on_fail of <coolamqp.clustering.single.SingleNodeReconnector object at 0x7fd4a8414610>>
*** self: <coolamqp.objects.Callable object at 0x7fd4a8414650>
*** args: ()
*** kwargs: {}
** __call__ at /usr/local/lib/python2.7/dist-packages/coolamqp/objects.py:33
*** callable: <coolamqp.objects.Callable object at 0x7fd4a8414650>
*** self: <coolamqp.objects.Callable object at 0x7fd4a8414850>
*** args: ()
*** kwargs: {}
** on_fail at /usr/local/lib/python2.7/dist-packages/coolamqp/uplink/connection/connection.py:161
*** watchlist: deque([<coolamqp.uplink.connection.watches.MethodWatch object at 0x7fd4b14867d0>, <coolamqp.uplink.connection.watches.MethodWatch object at 0x7fd4a84243d0>])
*** watchlists: [deque([<coolamqp.uplink.connection.watches.MethodWatch object at 0x7fd4b14867d0>, <coolamqp.uplink.connection.watches.MethodWatch object at 0x7fd4a84243d0>])]
*** watch: <coolamqp.uplink.connection.watches.FailWatch object at 0x7fd4a84148d0>
*** self: <coolamqp.uplink.connection.connection.Connection object at 0x7fd4a84147d0>
*** channel: 0
** on_fail at /usr/local/lib/python2.7/dist-packages/coolamqp/uplink/listener/socket.py:47
*** self: <coolamqp.uplink.listener.epoll_listener.EpollSocket object at 0x7fd4a8414890>
** wait at /usr/local/lib/python2.7/dist-packages/coolamqp/uplink/listener/epoll_listener.py:98
*** mono: 792670.195234957
*** self: <coolamqp.uplink.listener.epoll_listener.EpollListener object at 0x7fd4a8414790>
*** sock: <coolamqp.uplink.listener.epoll_listener.EpollSocket object at 0x7fd4a8414890>
*** events: [(31, 1)]
*** fd: 31
*** timeout: 1
*** event: 1
** run at /usr/local/lib/python2.7/dist-packages/coolamqp/uplink/listener/thread.py:30
*** self: <ListenerThread(coolamqp/ListenerThread, started daemon 140551332542208)>
** run_with_except_hook at /usr/local/lib/python2.7/dist-packages/smok4_core-4.4.5-py2.7.egg/sai/basic/__init__.py:75
*** args: ()
*** kw: {}
*** run_old: <bound method ListenerThread.run of <ListenerThread(coolamqp/ListenerThread, started daemon 140551332542208)>>
** __bootstrap_inner at /usr/lib/python2.7/threading.py:801
*** self: <ListenerThread(coolamqp/ListenerThread, started daemon 140551332542208)>
** __bootstrap at /usr/lib/python2.7/threading.py:774
*** self: <ListenerThread(coolamqp/ListenerThread, started daemon 140551332542208)>
*************************************************

So this seems to be connected to race conditions during reconnects