openaustralia / morph

Take the hassle out of web scraping
https://morph.io
GNU Affero General Public License v3.0
462 stars 74 forks source link

Changes to github repo descriptions do not migrate to Morph #207

Open handelaar opened 10 years ago

handelaar commented 10 years ago
  1. Create a new github scraper on Morph
  2. Change the description of the github repo
  3. Watch as nothing happens at the Morph end
mlandauer commented 10 years ago

There are a few different options on how this could be implemented:

  1. Ideally GitHub would used webhooks to tell Morph that the description has been updated and update it immediately. Unfortunately, at a glance of what the GitHub webhooks support this doesn't look to be possible
  2. Have a refresh button next to the description on Morph (see #33)
  3. Update the description when the scraper is run

I guess my preference would be for 1 if it was possible and then 3.

@handelaar what do you think?

handelaar commented 10 years ago

Likewise, though 3 may tip the balance for you the wrong way with remote function calls and latency? I'm also grand with 2 but I can see why that's suboptimal.

mlandauer commented 10 years ago

Another possibility is just to run a background job once per day or once per hour that updates things.