When reading from a source that contains duplicates, currently there's no way to remove duplicates, this PR add a new filtering based on a single item's field. Some considerations about the filter are
No multiple fields key are supported, if needed a transformation + filter_after could be used
Field with an empty or None key won't get filtered because it can not be determined deterministically if two items are duplicates based on that field's value.
When reading from a source that contains duplicates, currently there's no way to remove duplicates, this PR add a new filtering based on a single item's field. Some considerations about the filter are