Pinot batch ingestion duplicate row issue

[rahuldeshmukh81]

We are using pinot batch ingestion where we are receiving duplicate data in multiple files. Eg file1 has 5 record and few of them we Received in next file along with new data

We have tried primaryKeyColumns but it is inserting duplicate

[Jackie-Jiang]

Trying to understand the problem. Is the record duplicated in the input files? Do you want Pinot to help deduplicate the records? cc @swaminathanmanish

[rahuldeshmukh81]

Yes we have duplicate record in the file. And other use case is, we already persisted the record And Same record we received in the file along with new Data , here expectations is to only persist new one