search

HaveIBeenPwned / PwnedPasswordsDownloader

A tool to download all Pwned Passwords hash ranges and save them offline so they can be used without a dependency on the k-anonymity API
BSD 3-Clause "New" or "Revised" License
629 stars 49 forks source link

It keeps breaking the download #32

Open worldsdream opened 1 year ago

worldsdream commented 1 year ago

It shows 6% for the hash ranges download, and the message appears:

Finished downloading all hash ranges in 2,364,478ms (48.52 hashes per second). We made 114,734 Cloudflare requests (avg response time: 207.52ms). Of those, Cloudflare had already cached 114,732 requests, and made 2 requests to the Have I Been Pwned origin server.

So it looks like it gets corrupted, and the download stops.

The message should now show that it's finished downloading because it's not.

Also, what can be done if you want to download such a big file? We used to have a torrent and that worked perfectly.

The command I run:

haveibeenpwned-downloader.exe -n pwnedpasswords_ntlm

troyhunt commented 1 year ago

The old model definitely wasn't prefect, it became a stuck-in-time frozen version of passwords that never got updates from the firehose of new passwords fed in by the FBI. We're looking into this issue, but you're far better off just consuming the API on demand for password checks rather than downloading the entire corpus of data.

worldsdream commented 1 year ago

I understand @troyhunt.

If there would be an option in the download to pause/resume or something that it doesn’t break and we have to start all over with the download would be great.

I tried 3x times and I still couldn’t download the file.

With the API is best. But the tool isn’t configured for that.

https://docs.lithnet.io/password-protection/installation/create-a-new-store

Maybe the application will get an update and use the API.

stebet commented 1 year ago

The latest release should be a lot more resilient. Please try updating to 0.3.12.

worldsdream commented 1 year ago

@stebet Thank you for the very fast update.

I am downloading it and will report what I will get. The internet speed (fiber-optic) is good, and I never have any hiccups.

Let's see if it works well now.

One last thing.

Is there also a command for "NTLM ordered by hash"? I always downloaded that one from "torrent".

Right now, I am running "haveibeenpwned-downloader.exe -n pwnedpasswords_ntlm". But that's not ordered by hash, right?

If so, is there command for it?

Thank you.

stebet commented 1 year ago

Right now, I am running "haveibeenpwned-downloader.exe -n pwnedpasswords_ntlm". But that's not ordered by hash, right?

If so, is there command for it?

Thank you.

That is ordered by hash :)

worldsdream commented 1 year ago

Thanks @stebet

After running the command, I left it overnight and checked it.

So this is how it looks:

info

At 49% it had an error and stopped.

It looks better with this code update. But unfortunately, not there yet.

stebet commented 1 year ago

Are you running the latest version? There should be richer exception detail.

stebet commented 1 year ago

Also, that looks like it actually finished making all 1.048.576 requests. How big is the resulting file?

worldsdream commented 1 year ago

I am running version: 0.3.12

The file is 28.5 GB.

It shows that it took 04:03:33 and failed. But, it started again, and within 00:00:00 it got finished (which doesn't make sense).

Let me know if you like me to test anything.

crypt0rr commented 1 year ago

Currently experiencing the same issue, will try to give it a go tomorrow again.

Running v0.3.12.

crypt0rr commented 1 year ago

Re-tested today, it finished resulting in a filesize of 36.633.142KB.

haveibeenpwned-downloader

stebet commented 1 year ago

Re-tested today, it finished resulting in a filesize of 36.633.142KB.

haveibeenpwned-downloader

That seems about right as a final file-size.