search

tomnomnom / anew

A tool for adding new lines to files, skipping duplicates
MIT License
1.34k stars 147 forks source link

Performance Improvements, changed some design #3

Closed Ice3man543 closed 4 years ago

Ice3man543 commented 4 years ago

Major changes -

  1. The input file is used to eliminate duplicates. It is assumed input will be smaller than the file to append to, and hence to remove dupes, the input is read into a map. Output file is read line by line and any matches found to input map are removed. In the end, remaining elements are written to the output file.

  2. Used struct map instead of bool map as they occupy lesser space.

Ice3man543 commented 4 years ago

Closing as breaks the desired flow