search

Cornayy / dofus-scraper

An open-source Dofus encyclopedia scraper.
MIT License
8 stars 3 forks source link

Error fetching #5

Closed Nondairy-Creamer closed 3 years ago

Nondairy-Creamer commented 3 years ago

When I run npm run new

I can select the type of information to download. However when I click something like 'equipment' I get the error: Error fetching: https://www.dofus.com/en/mmorpg/encyclopedia/equipment?size=96

I can visit the website on my browser. I'm on Windows 10.

Also, its unclear to me where it saves the data if I run npm run existing which is successful. It just says "saving to database" and I'd like a .json

Thanks for any help

Cornayy commented 3 years ago

Hey @Nondairy-Creamer , npm run new will scrape the latest data on the dofus website and save it into your configured database in your local .env file. If you want the data for every category in .json format, you can already find it in the src/data/json directory.

Nondairy-Creamer commented 3 years ago

Thanks for the help @Cornayy !

  1. The .json files in src/data/json seem to be old. If you look at Kaliptus Headband the recipe thinks it needs Maho Firefoux Ghost Fabric and Soryo Firefox Ghost Fabric which I don't believe are in the game. I think this was changed in 2.57. Is it possible to update those files? I'm trying to run npm run new so I can get updated .json files

  2. Any thoughts on why I get this error when running npm run new? Error fetching: https://www.dofus.com/en/mmorpg/encyclopedia/equipment?size=96

  3. I'm still confused on where things get stored. My .env is set for mongodb, where would I find the output if the code ran successfully?

Cornayy commented 3 years ago

It seems like the dofus website enabled cloudflare protection, so the GET requests for all the urls are getting blocked. You can still run npm run existing, as it will update your database with all the json files in the project. You can install a tool like MongoDB Compass to find all the saved data in your database.