search

ysf / anewer

anewer appends lines from stdin to a file if they don't already exist in the file. This is a rust version of https://github.com/tomnomnom/anew
https://github.com/ysf/anewer
GNU General Public License v3.0
57 stars 4 forks source link

Easily allow disregarding timestamps, etc. #3

Open ibotty opened 3 years ago

ibotty commented 3 years ago

Of course that can be done with filters before, but it would be easier to be able to specify an option to e.g., ignore changed numbers or timestamp-like data.

I assume simply regexp-changing the string to hash will allow that. It would only need to find nice regular expressions for some common filters.

ysf commented 3 years ago

You mean something like an integrated cut ? Do you have an example usecase? I don't know if I understand it correctly and don't want to agree or rule it out before that.

ibotty commented 3 years ago

I am mostly interested in log file analysis where I sometimes have timestamps in front or random IDs that are irrelevant for the bigger picture. Usually I use a sed 's/.../XXX/ | sort -u`, but that's not optimal.

ysf commented 3 years ago

Would it be helpful if anewer had a parameter that would skip the first x chars of the line? I think this could be done without loosing much performance.