Scraping

[chadbrewbaker]

Could you write a short blurb how you scraped the Utah code into this repo? I would love to do it for Iowa. Assuming you already tackled a lot of problems along the way and could provide useful advise.

[chadbrewbaker]

Ah. I see you did include the scraping scripts, my bad. Still I bet you have some advise on putting these legal documents into a normal form.

[divegeek]

Hi Chad,

Yeah, the best resource for how I did the scraping is the scripts. Beyond that, I don't know that I have a lot of advice to offer. The problem is that there is no standard and no consistency in how the code is provided. I've been looking into how to scrape the Colorado code (I live in CO now), and it's hosted by LexisNexis in a way that makes scraping very difficult. It also has some copyright notices on it, which have to be legally invalid, but which make me nervous.

What would really be ideal is nicely marked-up XML with CSS to provide all presentation. If you can't find that, an HTML version might be easy enough to clean up -- or not, depending. Plain text is the easiest to work with. You'll notice that the Utah code is managed and distributed in WordPerfect (seriously!) format. That makes sense historically, since Utah was one of the first states to move to electronic management of its code, long before the web came into prominence, and perhaps before it came into existence, and WordPerfect was not only a de-facto standard at the time it was a Utah company. Anyway, if it weren't for the existence of open source WordPerfect file conversion tools, I'd have been dead in the water. Utah also provides an HTML version, but it's pretty ugly HTML. Looks okay on screen, but would be a pain to parse.

Bottom line: You just have to look at the formats that Iowa makes available and figure out which one is the easiest for you to work with.