Closed IanMagnusson closed 1 year ago
Adds a simple arg to always match hashes of whole paragraphs (so long as they are longer than min_ngram_size). This is something we frequently use for decontamination but have previously had to hack by setting a very large max_ngram_size.
Adds a simple arg to always match hashes of whole paragraphs (so long as they are longer than min_ngram_size). This is something we frequently use for decontamination but have previously had to hack by setting a very large max_ngram_size.