Re-write filter logic in `Cleaner`

dennlinger / summaries

A toolkit for summarization analysis and aspect-based summarizers

MIT License

11 stars 0 forks source link

Re-write filter logic in `Cleaner` #47

Open dennlinger opened 1 year ago

dennlinger commented 1 year ago

Currently, the iteration of samples will cut to the next sample immediately once a filter condition is met. However, this could be re-structured to add only a boolean flag, which comes with different (dis-)advantages:

It is now possible to set a filtered flag in the sample, allowing a more fine-grained inspection experience.
Functions that are relatively costly to compute will still have to run for all samples (which were previously skipped)

dennlinger commented 1 year ago

A temporary workaround is that we can simply expect the "previous iteration sample". However, this adds quite a bit of unnecessary iteration logic to the code, and also still has a major shortcoming. In cases where a sample fulfills multiple filtering criteria, we will only consider the first encountered one. This means that a sample is not properly "classified" in that sense, and we might get a distorted view of why samples are filtered.