search

mihaifm / HIBPOfflineCheck

Keepass plugin that performs offline and online checks against HaveIBeenPwned passwords
Other
315 stars 17 forks source link

Offline check mode has no effect. Always results in Secure, even for known breached entries. #58

Closed reclaimer5146 closed 1 year ago

reclaimer5146 commented 1 year ago

Duplicate of #48 but for offline mode.

Used the password downloader referenced in HIBP. Created a ~10gb file. Pointed the plugin at the offline file. All passwords are coming back secure, even known breached ones.

KeePass 2.53 HIBPOfflineCheck 1.7.9 MD5Sum of Offline passwords: 406CD8758AB00D597B9F350DC10C2FAF

cristianst85 commented 1 year ago

Which password file have you downloaded?

The required file is the one with format SHA-1 (ordered by hash).

The pwned-passwords-sha1-ordered-by-hash-v8.txt has about 34.7 GB.

L.E. To be honest, I did not test the HIBP Downloader, but I strongly recommend using the other method (i.e., the torrent method) to download the file.

reclaimer5146 commented 1 year ago

Which password file have you downloaded?

The required file is the one with format SHA-1 (ordered by hash).

The pwned-passwords-sha1-ordered-by-hash-v8.txt has about 34.7 GB.

L.E. To be honest, I did not test the HIBP Downloader, but I strongly recommend using the other method (i.e., the torrent method) to download the file.

The one in the list is outdated and has not been updated since 2021. Per Troy Hunt (owner of HIBP) and mihaifm:

The required file is the one with format SHA-1 (ordered by hash). Use the Pwned Passwords downloader ### (as of May 2022, this is the best way to get the most up-to-date passwords as suggested by the Have I Been Pwned project's author)

Per Troy from HIBP:

At present, the downloadable files are not updated with new entries from the ingestion pipeline, use the k-anonymity API if you'd like access to these.

I used the password downloader as specified. Not sure where you got the 34.7 GB file size from as the pwned-passwords-sha1-ordered-by-hash-v8.txt is only listed as 11.1GB.

I'll try to download again, but the password downloader (as recommended by Troy and mihaifm) doesn't have a way to download specific lists. It's all or nothing, at least according to my knowledge.

reclaimer5146 commented 1 year ago

Standby. I tried to download the passwords again and it said it finished but only downloaded 1gb. The password downloader goes through CloudFlare so I wonder if it thought I was trying to DDOS. I had 2048 threads running (download go brrrr). I dropped it down to 128 to see if it downloads more. Going to take about 30min per the ETA on the downloader.

2048 said 8 minutes so I stepped away and not sure if it actually reached 8 minutes but it was about a 10gb file. I figured with some deduplication, it wouldn't be out of the realm of possibilities for the filesize to have shrunk.

Will update with findings

cristianst85 commented 1 year ago

The pwned-passwords-sha1-ordered-by-hash-v8.7z has ~15.1 GB, but you need to un-zip it first to use it!

image

L.E. Corrected file size.

reclaimer5146 commented 1 year ago

The pwned-passwords-sha1-ordered-by-hash-v8.7z has ~ 15.6 GB, but you need to un-zip it first to use it!

image

That makes sense. Regardless, that list is out of date by over a year. Look at the date modified after you extracted. Therefore your solution doesn't apply and I'm downloading the latest using the downloader as mentioned by the developer and HIBP.

As mentioned, I'm downloading the passwords again using the recommended password downloader (not from the list because those are outdated). Says it's going to take 30 minutes with 128 threads.

cristianst85 commented 1 year ago

I know v8 might not be up-to-date. That fact is that I've updated (and not Mihai) the README.md file back in December (see here), but I did not test the HIBP Downloader. I assumed, based on information presented on the https://haveibeenpwned.com/Passwords site, that you can use it to download the equivalent of the file required by this plugin, which is the one having SHA-1 format (ordered by hash).

L.E. Edited for clarity.

reclaimer5146 commented 1 year ago

I know v8 might not be up-to-date. That fact is that I've updated the README.md file back in December (see here), but I did not test the HIBP Downloader. I assumed, based on information presented on the https://haveibeenpwned.com/ site, that you can download the equivalent of the file required by this plugin, But it seems it's not the case.

I'm still downloading the new password file. It may very well work. I only got a 10gb text file, NOT compressed. If it's supposed to be 34gb uncompressed, then I'm missing a TON of data. As mentioned, I think CloudFlare thought I was trying to DDOS with 2048 threads downloading the password list. I set it down to 128. Even though it's significantly slower, it hasn't stopped yet. It's saying 25min ETA but it's been bouncing around. I'll update once it's downloaded and I've tested again.

Update: Looks like it stopped again at 40% this time which was about a 12gb file. I've removed the additional thread processing and am now running it without any options. It is stating 50 minutes right now. Will update once I figured this out.

reclaimer5146 commented 1 year ago

This was entirely user error. The password file kept failing during the download but I hadn't realized it until the full file size was provided by Cristian. Naturally, the check didn't find the data that was missing (no way for it to know), so it was saying the passwords weren't found and the passwords were secure

If anyone else has this issue: The download kept failing for me so I stopped for a bit to clear any possible rate limiting. I tried it again, took about an hour to download. Full file size is 34.9 GB (37,512,290,443 bytes) at 100% downloaded with no errors reported during download. I downloaded it twice and the MD5sums matched.

MD5sum for 1/11/2023: BF840E28AC9B0618364F232FC7BF46AC