search

open-austin / azure-indigent-defense

4 stars 2 forks source link

Scraper function writes to blob storage #2

Closed zimzoom closed 1 year ago

zimzoom commented 1 year ago

These changes add the ability for the 'scraper' function to write the html files to blob storage. Also adds a minimal README. (note that we did deploy this version of scraper and it tested successfully on Azure, but we disabled it because we have not parallelized it yet and we want to stay under the 15 minute limit)

newswim commented 1 year ago

One question we had was whether the requirements.txt in the /scraper folder was necessary or not.

newswim commented 1 year ago

@zimzoom do you have a preference between merging this now and then doing the bytestream refactor, or would you rather do that as part of this PR? I would say I have a slight preference to merge as is.

zimzoom commented 1 year ago

I am fine with it either way.

normaljosh commented 1 year ago

Let's merge as is and I can do the refactor sometime this week

zimzoom commented 1 year ago

Thank you josh!