search

wololodev / d2mt

Competitive coverage on the go!
37 stars 11 forks source link

Data Sources #1

Closed fareesh closed 10 years ago

fareesh commented 11 years ago

Does your server scrape the Gosugamers pages or do they have an open / closed API for the data?

wololodev commented 11 years ago

Scrape. I have APIs if you look around the source code carefully.

-Hamed

fareesh commented 11 years ago

Yeah I noticed the APIs, I was just curious as to whether the data was being obtained via scraping from the original source, or whether there was a more elegant data store. Does Gosugamers allow anyone to scrape their data, or had you obtained permission from them?

wololodev commented 11 years ago

They know I'm scrapping since their staff is using the application and collaborated with me to fix some issues. I initially never told them about it, I just did it. I was scrapping their data, let me know if you are interested in the PHP files that does that.

fareesh commented 11 years ago

I actually have a scraper which grabs data from gosugamers and joindota, that I made for personal use. I wanted to eventually migrate it into an API but then noticed this project so I was curious as to how you went about it.

Blasz commented 10 years ago

I'm interested in the scraping php files. Would you be able to put them in this github repo or a separate one? The "livein" values for gosugamer's upcoming matches isn't coming through. Probably just a div id rename or something.

wololodev commented 10 years ago

Yeh, you can even host it in your own hosting. Right now I'm serving half a TB per month and I'm maxing out.

I'll put the repo up in a couple of minutes

wololodev commented 10 years ago

there, I aded /api, just push your fixes and I'll pull it to production. Also, this is ancient php code running, please don't laugh.

The /v2 is being used by mobile versions of d2mt and some other websites. These also dump JSON key/values directly.

The other directories is what the Chrome extension is using (with the HTML dump). So you will have to update both I guess :sake:

Open an issue for any issues! GL&HF