Open Chrisbattarbee opened 8 months ago
Statusphere currently has 657 companies in its database.
Of those, 298 are not indexed curl 'https://statusphere.metoro.io/api/v1/statusPages' | jq '.statusPages[] | select(.isIndexed == false) | length' | wc -l
curl 'https://statusphere.metoro.io/api/v1/statusPages' | jq '.statusPages[] | select(.isIndexed == false) | length' | wc -l
This is because the scrapers are unable to parse them, so we need more providers.
We should figure out which providers to prioritize based on the number of pages it would enable the scraping of.
From a random sampling of around 30 pages. It seems like the most beneficial would be
Statusphere currently has 657 companies in its database.
Of those, 298 are not indexed
curl 'https://statusphere.metoro.io/api/v1/statusPages' | jq '.statusPages[] | select(.isIndexed == false) | length' | wc -l
This is because the scrapers are unable to parse them, so we need more providers.
We should figure out which providers to prioritize based on the number of pages it would enable the scraping of.