It would be good to expose basic functionality on the Parsons table to allow people to deduplicate their data. Currently a lot of folks convert the Parsons table to a Pandas dataframe to accomplish this which seems wasteful both in terms of developer times and also could potentially impact performance:
I'm imagining something that looks like this:
parsons_table.deduplicate(keys, sort)
Where you can give no keys, one key, or several to deduplicate by, as well as indicating whether the data is already sorted or should be sorted (basically a wrapper on petl's presorted parameter.
Essentially it would be a wrapper around the petl function that accomplish this, ie:
from petl import etl
from parsons import table
people = people.to_petl()
result = etl.transform.dedup.distinct(people)
Table(result)
Medium priority - this seems like it would be very helpful but also it's not blocking anything.
It would be good to expose basic functionality on the Parsons table to allow people to deduplicate their data. Currently a lot of folks convert the Parsons table to a Pandas dataframe to accomplish this which seems wasteful both in terms of developer times and also could potentially impact performance:
I'm imagining something that looks like this:
parsons_table.deduplicate(keys, sort)
Where you can give no keys, one key, or several to deduplicate by, as well as indicating whether the data is already sorted or should be sorted (basically a wrapper on petl's
presorted
parameter.Essentially it would be a wrapper around the petl function that accomplish this, ie:
Medium priority - this seems like it would be very helpful but also it's not blocking anything.