search

openaustralia / morph

Take the hassle out of web scraping
https://morph.io
GNU Affero General Public License v3.0
461 stars 74 forks source link

Question: environmental variables #1182

Closed svict4 closed 6 years ago

svict4 commented 6 years ago

Hi folks, is there an environmental variable that I can tap to check if there already exists something in the Morph Database? It would help speed up the scraping process. Or is there a way that I can trigger the scraper, rather than the daily cronjob?

mlandauer commented 6 years ago

Hi @svict4, the help forum is the best place to ask questions. Github issues is really just for reporting bugs or requests for new features.

Now that you're here, I'll endeavour to answer your question but please next time go to the help forum. Thanks!

If you want to check if there is something in your database, just query it with a little bit of sql like select count(*) from data or use the scraperwiki library as shown in https://morph.io/documentation.

There is no api for triggering scrapers other than asking for the scraper to run once per day. This is by design. It allows us to spread the scraper load more effectively. Its the tradeoff with using a free service! ;-)