Generating Bloom Filter results in Error

[Stray216]

I'm getting an error message when trying to generate the bloom filter. Here's a timeline of my research so far:

The error message I'm getting looks like this: https://pastebin.com/MwkdBKzt

After asking ChatGPT about this, this was its first response: https://pastebin.com/gC3wP0r2

I pasted the whole of the BloomWorker() method in CreateBloomFilter.cs to ChatGPT and its response was this: https://pastebin.com/pqMYKhF7

I figured there must be lines shorter than 40 in the pwned passwords text file, so I had ChatGPT write me a python script to search for these. The output in the console was this: https://pastebin.com/N2SzUijC

Then I asked ChatGPT if line 124 could be modified so that it splits the line at the ":" symbol rather than taking the first 40 characters. Its response was this: https://pastebin.com/LjneGxfC

I hope any of this is helpful. I really want this bloom filter file as 34,9GB is really large.

[mihaifm]

Hi, interesting analysis :) Are you sure you have the correct input file? SHA1 hashes are always 40 characters so not sure why are you finding so many that aren't size 40.

[cristianst85]

@Stray216, can you describe what exactly you are trying to do? And how (step by step)?

Also, you might want to read https://github.com/mihaifm/HIBPOfflineCheck/issues/58 (start from the end).

[Stray216]

@Stray216, can you describe what exactly you are trying to do? And how (step by step)?

I am trying to get the bloom filter file by using the generator from this plugin. I click on the generate button and specify the pwned_passwords file and the output bloom filter file. When the generator is running I get this error: https://pastebin.com/MwkdBKzt

While I can choose to ignore the error message, the bloom filter won't generate any further after it.

Also, you might want to read https://github.com/mihaifm/HIBPOfflineCheck/issues/58 (start from the end).

I've read it and don't see how it can help me. If you know how this can help me, do tell. :D

[cristianst85]

What's your input file? What's the exact file size (in bytes)? How did you obtain it? Can you try to re-download it?

Also, try to create an MD5 hash sum for the HIBP password file that you have now to compare with the MD5 of the re-downloaded file; if the file sizes do not match, then skip comparing the MD5 hashes; repeat the download process until you obtain the same result/file, binary identical, at least 2 times out of 3. You can assume then with a high degree of confidence that the HIBP password file was successfully downloaded.

I am just assuming that your input file is/was botched during the download. That's why I've indicated to you to read https://github.com/mihaifm/HIBPOfflineCheck/issues/58.

[Stray216]

With the help of troyhunt and stebet from the PwnedPasswordsDownloader, I was able to download a, seemingly uncorrupted, pwned passwords file that has no lines shorther than 40 characters.

I will attempt to generate the bloom filter tomorrow evening.

[mihaifm]

I uploaded the latest bloom filter to the HIBPBloomFilter repository, feel free to download it from there.

I saw the issue you raised on PwnedPasswordsDownloader, indeed it seems like the input file had some wrong entries which were corrected. Nice job on figuring this out.

[Stray216]

I successfully generated the bloom filter without any errors. I tested the plugin with a few good/bad passwords using the bloom filter and everything seems to work fine. Thank you also for the new release at the HIBPBloomFilter repository.