faust-streaming / faust

Python Stream Processing. A Faust fork
https://faust-streaming.github.io/faust/
Other
1.65k stars 183 forks source link

`SetTable`s are missing `on_clear` #601

Closed cristianmatache closed 1 week ago

cristianmatache commented 10 months ago

mode has a method to clear all the elements in a set (https://github.com/faust-streaming/mode/blob/c0aa58181432402dca30aa5978179383863a185a/mode/utils/collections.py#L546-L548) which calls an .on_clear callback. This .on_clear callback method is empty by default (https://github.com/faust-streaming/mode/blob/c0aa58181432402dca30aa5978179383863a185a/mode/utils/collections.py#L535-L536)

However, faust does not implement on_clear at all.

This means that the in-memory data structure is cleared but the change is not reflected in the changelog topic. Upon recovery elements that were previously cleared may re-appear. I think the fix would be very simple:

bboggs-streambit commented 5 months ago

@cristianmatache @wbarnha , Since on_key_del is implemented for Table (https://github.com/faust-streaming/faust/blob/master/faust/tables/table.py#L86-L94) Could we simply (naively) implement this in terms of __delitem__?,

e.g. something like, in Table define

    def on_clear(self):
      for k in self.data.keys():
        del self[k]