search

rtyley / bfg-repo-cleaner

Removes large or troublesome blobs like git-filter-branch does, but faster. And written in Scala
https://rtyley.github.io/bfg-repo-cleaner/
GNU General Public License v3.0
10.83k stars 535 forks source link

Unable to replace text in a large file #513

Closed sagerg closed 3 months ago

sagerg commented 3 months ago

When I run replace-text passwords.txt, I get expected results for all files in my git history except for 1 file which has around 700,000 lines of text in it. It is named foo.out and uses json formatting. I tried isolating the issue by copying the password in another file with about 20 lines of text with similar formatting. The replace text operation worked on that file, but not on the larger file. Any ideas? Thanks!

rtyley commented 3 months ago

If you run bfg with no command line arguments, it prints out help on all of it's command-line options - including this one:

  -fs, --filter-content-size-threshold <size>
                           only do file-content filtering on files smaller than <size> (default is 1048576 bytes)

Your 700,000 line file is probably bigger than 1MB, is that right?

sagerg commented 3 months ago

@rtyley hello! yes it is, thank you I didn’t notice that default value

sagerg commented 3 months ago

Works now! Ran bfg -rt keys.txt --no-blob-protection -fs 25000000B to take care of large json files that were littered with passwords. Thanks!