Expand the crawler that can crawl a list of mastodon instances and extracts public profiles with #for-hire tags.
Details
For the intermediate product "for hire search", we need to extend the ScraPy spider to crawl across multiple mastodon instances. Currently it only crawls one instance in a Proof of concept. TODO: release this scraper proof of concept in a flockingbird repo.
"Intermediate search" is explained in #37.
Deliverable
Given a list (in JSON, text or STDIN), we crawl each instance on that list for public profiles
As in the Proof of concept scraper, we only index public data.
As in the Proof of concept scraper, we adhere to noindex, robots.txt, etc.
It returns a JSON document either per instance, or of all instances, structural similar to the proof of concept scraper.
Expand the crawler that can crawl a list of mastodon instances and extracts public profiles with #for-hire tags.
Details
For the intermediate product "for hire search", we need to extend the ScraPy spider to crawl across multiple mastodon instances. Currently it only crawls one instance in a Proof of concept. TODO: release this scraper proof of concept in a flockingbird repo.
"Intermediate search" is explained in #37.
Deliverable