robinhood / faust

Python Stream Processing
Other
6.75k stars 534 forks source link

Repeated Timer Overlap Warning with SetTable #549

Open tapple opened 4 years ago

tapple commented 4 years ago

Checklist

Steps to reproduce

Run this demo worker like python bugdemo.py worker:

import faust
app = faust.App('example', broker='kafka://')
set_table_1 = app.SetTable(name='set_table_1')
set_table_2 = app.SetTable(name='set_table_2')
# set_table_3 = app.SetTable(name='set_table_3')
# set_table_4 = app.SetTable(name='set_table_4')
# set_table_5 = app.SetTable(name='set_table_5')

if __name__ == '__main__':
    app.main()

Expected behavior

The worker should do absolutely nothing. It has no tasks or agents

Actual behavior

I get this warning spam repeatedly in the logs:

[2020-03-12 11:14:29,644] [86917] [WARNING] Timer SetManager.flush is overlapping (interval=2.0 runtime=2.0014956910163164) 
[2020-03-12 11:14:33,644] [86917] [WARNING] Timer SetManager.flush is overlapping (interval=2.0 runtime=2.000451576896012) 
[2020-03-12 11:14:33,644] [86917] [WARNING] Timer SetManager.flush is overlapping (interval=2.0 runtime=2.0007298621349037) 

It continues until I kill the worker. I get the warning more frequently with more SetTables, and less frequently with only 1 SetTable, and never with 0 SetTables

Versions

tapple commented 4 years ago

SetManager.flush seems to be here: https://github.com/robinhood/faust/blob/master/faust/tables/objects.py#L145-L149

tapple commented 4 years ago

SetManager.flush seems to be here: https://github.com/robinhood/faust/blob/master/faust/tables/objects.py#L145-L149

Is that self.sleep(sleeptime) supposed to be there? I feel like there shouldn't be a sleep inside a timer

tapple commented 4 years ago

If I comment out await self.sleep(sleep_time), the warning goes away, and cpu usage is still mostly idle, so, I think this might be the correct fix

2phost commented 3 years ago

Any news regarding this?

bobh66 commented 3 years ago

This project appears to have been abandoned.

You might want to check out the fork of this project - https://github.com/faust-streaming/faust