search

HaveIBeenPwned / PwnedPasswordsDownloader

A tool to download all Pwned Passwords hash ranges and save them offline so they can be used without a dependency on the k-anonymity API
BSD 3-Clause "New" or "Revised" License
701 stars 55 forks source link

Not able to download the complete dataset. #77

Open ranjoy001 opened 2 months ago

ranjoy001 commented 2 months ago

I'm trying to download all SHA1 hash passwords into a single file, but the process isn’t completing. I ran this command haveibeenpwned-downloader pwnedpasswords to download all the hashes into one text file, but only 14% of the data was downloaded. Then I tried another set of commands to download the hashes into a directory with multiple text files, but that process also didn't finish and only completed 8% of the total dataset. Can someone please help here?

ranjeet@MacBook-Pro-5 ~ %  haveibeenpwned-downloader pwnedpasswords
Hash ranges downloaded ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  14% 00:00:00

Finished downloading all hash ranges in 7,14,218ms (212.37 hashes per second).
We made 1,51,771 Cloudflare requests (avg response time: 170.97ms). Of those, Cloudflare had already cached 1,51,771 requests, and made 0 requests to the Have I Been Pwned origin server.

ranjeet@MacBook-Pro-5 ~ % haveibeenpwned-downloader hashes -s false -p 64
Hash ranges downloaded ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   8% 00:00:00

Finished downloading all hash ranges in 3,12,657ms (279.54 hashes per second).
We made 87,401 Cloudflare requests (avg response time: 208.67ms). Of those, Cloudflare had already cached 87,401 requests, and made 0 requests to the Have I Been Pwned origin server.
Borris70 commented 1 month ago

I had the same issue with .NET 8.0.403. The download finished without any error message bute the file was always to small (between 100 - 300 MB - different size with every attempt). After uninstalling .NET 8 und installing .NET 6 (6.0.427) the download was fine again.

stebet commented 4 weeks ago

I made an update to the application a few hours ago. If you can try this again with the latest SDK and report back, that would be awesome :)

Borris70 commented 4 weeks ago

I installed newest .Net 8 and updated the haveibeenpwned-downloader to Version 0.5.21. Now i get more errors.

errors
stebet commented 3 weeks ago

I installed newest .Net 8 and updated the haveibeenpwned-downloader to Version 0.5.21. Now i get more errors.

These errors are warnings and should be OK, as it retries the downloads for these files instead of just quitting. I should make these warning messages a little clearer :)

ranjoy001 commented 3 weeks ago

I have also given it a try and was able to download the complete dataset without any error, Thanks for looking into it!!

Borris70 commented 3 weeks ago

I tried again. Alot of warnings but the download seems to finish. I did the same download from my debian server. This download was without any warnings and the file is slightly larger.

dl02

I will use in future my debian server for this.