Missing word lists

[zzJZzz]

Details

• I created a ruby script to compare the repo.json file with the current wordlist.yml file and add anything that was missing.
• I triple checked the formatting was the same throughout. eg:

  joomla-themes:
  :url: https://raw.githubusercontent.com/danielmiessler/SecLists/master/Discovery/Web-Content/CMS/joomla-themes.fuzz.txt
  :summary: Discovery wordlist
  :categories:
  - discovery

  vs.

  alexa-top-1000:
  :url: https://github.com/urbanadventurer/WhatWeb/blob/master/plugin-development/alexa-top-1000.txt
  :summary: The Alexa Top 1000 domain names.
  :categories:
  - dns
  - domains

• When a list had a generic name eg: 13 I changed it to include where the list is from and what it contained like weakpass-wordlist. from the PR comments. The goal was to add more detail so it would be more apparent at first glance.

• Testing

• I also ran the rspec for the file and 5 green passing tests.
• I visually inspected each new wordlist for anything that stood out visually.

[postmodern]

@zzJZzz nice work on writing a script to import the entries!

I may also have to add more Category tags to better describe/group the wordlists.

[zzJZzz]

Ahh. The script caused some of those problems and then regex find and replace only caught some of the others. Like you said, there're so many new entries I go cross eyed 🤣.

I will get the spaces removed from the name.

Personally I like seclists-.

When you are searching for a word list do you know the name of it? Or go window shopping till you find one you like? if so then the prefix seems like the better way to go I think.

I will remove the operating platform URLs

I'll add the . To the summaries.

Thanks again for your comments and assistance!

On Tue, Jul 9, 2024, 5:52 PM Postmodern @.***> wrote:

@zzJZzz https://github.com/zzJZzz nice work on writing a script to import the entries!

I may also have to add more Category tags to better describe/group the wordlists.

— Reply to this email directly, view it on GitHub https://github.com/ronin-rb/ronin-wordlists/pull/40#issuecomment-2218793001, or unsubscribe https://github.com/notifications/unsubscribe-auth/AMYO64BK3R7SI4V3P5AR7ALZLRLQTAVCNFSM6AAAAABKTSZ3P6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJYG44TGMBQGE . You are receiving this because you were mentioned.Message ID: @.***>

[zzJZzz]

Hello. I addressed the changes above. with the exception of

This raises the question, should all wordlists which are downloaded from the SecLists repo, be prefixed with seclists-/SecLists-?

As I wasn't sure if there was an official decision. I can go back and change, or perhaps another issue could be created?

In addition:

I did remove a few wordlists that gave me trouble when I imported. Maybe about 40? The original repo had spaces in between which showed up on import. If that's how they are supposed to be then I can add in the %20, or if they should have the _ or -, I can do that too. When I tried to find those wordlists from the actual website and tried out some of the urls to see, I did not have success so I removed for now.

[postmodern]

The original repo had spaces in between which showed up on import. If that's how they are supposed to be then I can add in the %20, or if they should have the _ or -, I can do that too.

I would use hyphens, or whatever the filename of the wordlist is without the file extension (ex: foo-bar100.txt -> foo-bar100).

When I tried to find those wordlists from the actual website and tried out some of the urls to see, I did not have success so I removed for now.

This is a good point. We should check if the wordlist URLs do not return 404. I can probably add tests to do a HEAD request for each wordlist URL to check the status code.

[postmodern]

@zzJZzz I have now added a lint:wordlists rake task and hooked it up to GitHub Actions to run anytime data/wordlists.yml is changed. This will only lint the wordlists metadata in data/wordlists.yml once. We still do not actually check if the URL is still alive, due to web servers rate limiting the number of requests we can send. Definitely rebase against main to get the new GitHub Actions linting.