search

rverton / webanalyze

Port of Wappalyzer (uncovers technologies used on websites) to automate mass scanning.
MIT License
908 stars 134 forks source link

Update failure due to wappalyzer deletion #72

Closed NikosRig closed 7 months ago

NikosRig commented 10 months ago

Webanalyze downloads the technologies.json file from the project wappalyzer, but it no longer exists.

webanalyze -update

error: can not update apps file: json: cannot unmarshal number into Go value of type map[string]webanalyze.Category

rverton commented 10 months ago

Hi @NikosRig, yeah I saw they have deleted the repository. As I saw in a tweet, this is also intentional, so I don't think the repository will be reinstated. We can update the technologies.json file we have in this repository to the last public version. After this we need to see if there will be a new fork in the future of wappalyzer. Any other ideas?

amohar commented 10 months ago

@rverton Here's an idea, I don't know how legal it is due to licensing, so be warned... The idea is to pull the json files directly from the plugin itself. What could be done:

  1. Scrape the mozilla plugins page for wappalyzer to get a link to xpi file
  2. Xpi file is just a zip archive, so unzip, can be done in memory to be clean
  3. Extract the technologies folder, which now contains several jsons, one for each letter of the alphabet
  4. Merge jsons into a single technologies.json file
rverton commented 9 months ago

I thought of doing it this way, too, but I'm not sure if this will get us into any copyright/license problems. Maybe if we do not put the resulting fingerprints in the repository we are fine, because it's the user doing the extraction?

coj337 commented 8 months ago

Currently building from source works great, but go install is still on the old version. Now that this has been solved with the ethnec repo, @rverton can you do a release so the go install method works again? 😄

rverton commented 7 months ago

I just released a new version.