Seclists: not able to extract archived resources as rockyou.txt

[D3vil0p3r]

Describe the bug

Seclists has some common used resources, as rockyou.txt, that are compressed. I know that a user can copy and paste on home folder the archives, and then unzip them. But what I propose is to uncompress all .tar.gz files in the repository at building time of the package to make them immediately available for the users after install.

The .tar.gz files are:

seclists/Passwords/Leaked-Databases/rockyou-withcount.txt.tar.gz
seclists/Passwords/Leaked-Databases/rockyou.txt.tar.gz

Notify maintainers

@Tochiaha @Pamplemousse

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.6.41, Athena OS, 24.11 (Vicuna), 24.11pre655874.68c9ed8bbed9`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.5`
 - channels(root): `"nixos"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

---

Add a :+1: reaction to issues you find important.

[Pamplemousse]

Actually, rockyou is packaged...

$ nix-shell -p 'wordlists.override { lists = with (import <nixpkgs> {}); [ rockyou ]; }'
...

$ wordlists
/nix/store/vxwq7ba089nffxfv4ii335xzaxf4v7p3-wordlists-collection/share/wordlists
└── rockyou.txt -> /nix/store/66b599j6m4fx26d3cvh644zqpq3bnpvm-rockyou-2024.2/share/wordlists/rockyou.txt

1 directory, 1 file

$ head -n2 /nix/store/66b599j6m4fx26d3cvh644zqpq3bnpvm-rockyou-2024.2/share/wordlists/rockyou.txt
123456
12345
123456789
password
iloveyou

[D3vil0p3r]

Not sure what is the sense to have two separated packages for the same source repository, mostly when rockyou package provide one single file. What do you think to merge it to seclists and keep only this last one?

[Pamplemousse]

for the same source repository

Not sure what you mean by this.

SecLists is a "list aggregator", and we don't have control over what it adds or not: We don't know if it will keep the rockyou list, and the package doesn't allow any granular picking of the lists.

I don't see the harm of having rockyou being standalone. Even more: I would rather have the ability to get rockyou (134MB) without all the other lists that seclists are providing (2GB).

FYI, we added security lists to be used with the wordlist package. See #104712 for historical reference, and #260240.

Concerning the topic of the issue, you are welcome to add the archive extraction steps to the seclists package: https://github.com/nixos/nixpkgs/blob/master/pkgs/by-name/se/seclists/package.nix#L17-L25 . However, be mindful: the package already takes 2GB of space on disk. Extracting everything will probably make the whole package balloon. Probably you can rework the package to make extraction optional, and maybe even granular (allow user to only extract lists of their choice, instead of everything). If we had this, I guess we could get rid of the rockyou package.

[D3vil0p3r]

https://github.com/NixOS/nixpkgs/pull/329917