search

agccie / ACI-EnhancedEndpointTracker

16 stars 15 forks source link

multiprocess no_wait queue is unreliable #23

Closed agccie closed 6 years ago

agccie commented 6 years ago

Example shows 1200 pending events for key where only 13 are found within the queue

### From manager
26889||CEST 2018-08-30 16:45:51.466||WARNING||ep_subscriber.py:(504)||key count(1201) exceeds max threshold(1200) for w(0):|addr:AC:16:2D:C2:0F:3D||type:mac||vnid:16154569|

### From corresponding worker
26935||CEST 2018-08-30 16:45:52.072||DEBUG||ep_worker.py:(1602)||removing all jobs from RxQ with key: {'type': 'mac', 'addr': 'AC:16:2D:C2:0F:3D', 'vnid': '16154569'}
26935||CEST 2018-08-30 16:45:52.085||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:270521
26935||CEST 2018-08-30 16:45:52.101||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:270308
26935||CEST 2018-08-30 16:45:52.111||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:270186
26935||CEST 2018-08-30 16:45:52.123||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:270035
26935||CEST 2018-08-30 16:45:52.135||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:269860
26935||CEST 2018-08-30 16:45:52.145||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:269711
26935||CEST 2018-08-30 16:45:52.163||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:269514
26935||CEST 2018-08-30 16:45:52.185||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:269240
26935||CEST 2018-08-30 16:45:52.252||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:268593
26935||CEST 2018-08-30 16:45:52.270||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:268427
26935||CEST 2018-08-30 16:45:52.318||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:267850
26935||CEST 2018-08-30 16:45:52.329||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:267700
26935||CEST 2018-08-30 16:45:52.346||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:267441
26935||CEST 2018-08-30 16:45:52.367||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:267157
26935||CEST 2018-08-30 16:45:52.380||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:267000
26935||CEST 2018-08-30 16:45:52.404||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:266744
26935||CEST 2018-08-30 16:45:52.422||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:266552
26935||CEST 2018-08-30 16:45:52.491||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:265655
26935||CEST 2018-08-30 16:45:52.500||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:265525
26935||CEST 2018-08-30 16:45:52.513||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:265348
26935||CEST 2018-08-30 16:45:52.523||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:265231
26935||CEST 2018-08-30 16:45:52.536||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:265059
26935||CEST 2018-08-30 16:45:52.548||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:264907
26935||CEST 2018-08-30 16:45:52.559||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:264749
26935||CEST 2018-08-30 16:45:52.571||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:264607
26935||CEST 2018-08-30 16:45:52.583||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:264409
26935||CEST 2018-08-30 16:45:52.669||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:263197
26935||CEST 2018-08-30 16:45:52.685||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:262985
26935||CEST 2018-08-30 16:45:52.695||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:262836
26935||CEST 2018-08-30 16:45:52.706||DEBUG||ep_worker.py:(1615)||no_wait triggered empty with rxQ:262675
26935||CEST 2018-08-30 16:45:52.706||WARNING||ep_worker.py:(1619)||stop waiting for get_nowait
26935||CEST 2018-08-30 16:45:52.707||DEBUG||ep_worker.py:(1625)||removed 13 jobs with key {'type': 'mac', 'addr': 'AC:16:2D:C2:0F:3D', 'vnid': '16154569'} from rxQ
26935||CEST 2018-08-30 16:45:52.707||DEBUG||ep_worker.py:(1626)||re-adding 8185 jobs back onto rxQ
26935||CEST 2018-08-30 16:45:52.736||DEBUG||ep_worker.py:(1631)||notify fail is not enabled
agccie commented 6 years ago

fixed in 2.0 - moved to redis for messaging queue