search

open-contracting-archive / kingfisher-vagrant

Abandoned as not kept up-to-date with Kingfisher components
BSD 3-Clause "New" or "Revised" License
5 stars 5 forks source link

[1.1] [SPIKE - 3 days] Familiarisation with Scrapy + scrapyd deployment #273

Closed robredpath closed 5 years ago

jpmckinney commented 5 years ago

As part of this, let's check if there are any popular scrapy web UIs besides scrapyd, and share the results of that check in this issue.

rhiaro commented 5 years ago

This scrapyd web UI looks new, active, and shiny: https://github.com/my8100/scrapydweb (usual caveats: one developer, not official Scrapy code, so long-term maintenance and who it's currently used by uncertain).

Screenshots of this one look nice: https://github.com/DormyMo/SpiderKeeper. More contributors but less recent activity. Doesn't support individual scraper stats yet.

This is a fork of the previous one, with individual scraper stats, and lots of pretty charts: https://github.com/fliot/ScrapyKeeper

This one is simpler, more of a nice skin over existing functionality. No individual crawler stats: https://github.com/jxltom/scrapymon

jpmckinney commented 5 years ago

Do we have a list of requirements for the UI?

If the list of requirements is short, simple and stable over time, then the main concern is whether the project stays up-to-date with its requirements (mainly with Scrapy). Otherwise, it's fine for a functional UI to have low activity.

How big are the codebases? If they are small, then it's easier to maintain a fork if needed.

robredpath commented 5 years ago

@BibianaC to review @rhiaro 's deploy instructions as part of this

jpmckinney commented 5 years ago

"Familiarisation with Scrapy + scrapyd deployment" seems complete, though there might be follow-up issues.