Harris county tx

[sglyon]

Summary

Adds hopsital and ICU bed usage for harris county TX.

This probably needs a bit more work, but I wanted to touch base with maintainers here to see if I'm on the right direction.

What it does:

• Adds a new export to fetch/index.js that allows users to specify a url and a callback to be called in the context of the opened puppeteer Page instance
• Uses the new exported function to open a PowerBI dashbaord from a southeast texas council
• Navigates through that dashboard to a page that contains summary stats on covid patients in general and ICU beds.
• Scrapes this data for Harris county (Houston). There are other counties there that we could scrape, but for now I'm only focusing on Harris county.
• Returns hospitalized_current and icu_current for process.env.SCRAPE_DATE. The scraper has data for previous two weeks, so it is a timeseries

Things I'm a little uncertain on are:

• Is adding this additional exported function to the fetchlib ok?
• How could/should we handle cache for this scraper? I'm not sure on the details of how caching is handled here. For now I am completely ignoring it.
• I only return those two current hospital bed usage data points, I don't have more fundamental results like cases, deaths, reported, etc. Those are aggregate level TX scraper. Is there a strategy for merging multiple scraper outputs so we have all the info?

Thanks!

[jzohrab]

Hi @sglyon - thanks for the PR! Have been busy over here so haven't had time to get to it.

The callback stuff looks to be mostly ok. We're moving to a new repo (Li) in this same org, which has a completely different crawl and scrape model.

Questions and Answers:

Is adding this additional exported function to the fetchlib ok?

It should be, but there are few changes to this so I'll need to check!

How could/should we handle cache for this scraper? I'm not sure on the details of how caching is handled here. For now I am completely ignoring it.

Yes, good question. Ultimately your page does const data = await page.waitForXPath("//div[@aria-label='Grid']").then(getDataFromPivotTable);, but this is totally different from our usual methods. Tough stuff!

I only return those two current hospital bed usage data points, I don't have more fundamental results like cases, deaths, reported, etc. Those are aggregate level TX scraper. Is there a strategy for merging multiple scraper outputs so we have all the info?

Normally, we actually fetch things within one scraper and combine them in there. Ideally, the data would be normalized when inserted into the db or document, and then some other process would join them ... but that's not how it's done (yet)!

Great work, this is a non-trivial problem.

[jzohrab]

I opened https://github.com/valorumdata/coronadatascraper/pull/1 into this branch from my repo, it has some substantial revisions, but respects caching etc which we must follow. Take a look and let me know, thank you!

[jzohrab]

Closing in favor of https://github.com/covidatlas/coronadatascraper/pull/1040, which updates the work you started to follow this project's conventions.

Thank you for the PR! I hope 1040 gets merged soon so we can see how it behaves in prod ... at the moment I'm not sure how it will work.

[sglyon]

Thanks @jzohrab -- hopefully future contributions don't require so much hands on help from the core team!