Large hashlist support

[fransla]

It seems that Phatcrack refuses to save large hashlists (> 10000 hashes), giving an error 500, "Failed to create hashlist".

Would it be possible to add file upload support or otherwise support larger hashlists please?

[lachlan2k]

Can you check with your administrator what the corresponding error is in the backend logs?

When you receive a 500, you should hopefully also get a UUID that can be grepped in the backend. The UUIDs aren't displayed everywhere yet though, unfortunately, so you might have to open your browser's network tab to find it.

Finally, just in case its something specific, which hash mode as you using?

Regardless though, I'll definitely get to work on supporting uploading hashlists as a text file instead, which will be part of wider work to support other formats (such as zip, keepass files, etc).

Thanks!

[JSmith-Aura]

Howdy,

We are running v0.1.3. Im not sure if this is the exact error that we're running in to as I've also opened another issue detailing another error I found in the logs today.

However it is the only one complaining about size:

2023/11/15 08:56:37 /app/api/internal/db/project.go:112 extended protocol limited to 65535 parameters
[322.567ms] [rows:1] INSERT INTO "hashlists" ("created_at","updated_at","deleted_at","project_id","name","version","hash_type") VALUES ('2023-11-15 08:56:37.663','2023-11-15 08:56:37.663',NULL,'127844f8-a6e9-483b-bcdf-259b35a7c099','large',1,1000) RETURNING "id"

Corresponding web response:

{"URI":"/api/v1/hashlist/create","content_length":"1165379","error":"extended protocol limited to 65535 parameters","error_id":"0166cedb-75b3-4b56-933c-021ee6bb1c7f","latency_ms":947,"level":"error","method":"POST","msg":"request error 0166cedb-75b3-4b56-933c-021ee6bb1c7f"}

[lachlan2k]

Ah, looks like Postgres has limit of 65,535 parameters in a parameterized query. Each inserted hash has 6 parameters, placing the limit of hashes inserted in a single query at just over 10k hashes.

I'll fix this by splitting the hashes into chunks of 1,000 and use several queries to insert them, wrapped in a transaction.

For sanity's sake though, I'll probably add a hard limit of 100k hashes (configurable by admins) to prevent resource exhaustion.

[JSmith-Aura]

Dang postgres