Open Dzeiberg opened 10 months ago
only do filtering out in the write functions
@trberg , I added logic to filter out records right before I write to the sql table in commit 8c58a9e81b0fe6745457ad371e81634d8f9de937
I know log to the user how many rows are being skipped for each table.
This does cause Processor to take a long time to initialize, as I have to query the seq_hash,mutation pairs from all tables. If you're processing a large number of sets of mutations then this isn't significant, but for a single set of mutations this initialization time is quite long.
If not overwriting, filter out mutations that are present in all tables, additionally, add check to write_* that checks before adding.