Have a script we can use to do a side-by-side comparison of our JSON court lists with the newest info from masscourts

[nonprofittechy]

Note that there are a lot of errors on the webpage (or least there were ~ 2 years ago). So this is just to have a way for us to highlight things that look different for manual inspection.

No rush on this--likely a student can work on it this summer (if we have a summer student) or in the Fall.

[nonprofittechy]

There's some JSON file embedded in this page: https://www.mass.gov/orgs/massachusetts-court-system/locations. Can check it as well, although I think sometimes it differs from what you get when you visit each link manually.

See this for the specific element on the page that might have some data: https://github.com/GBLS/docassemble-MACourts/blob/6a5da00ddefbeec39aa5b1f140921de52d7faf80/docassemble/MACourts/macourts.py#L17

Note: it looks like the structure of this page changed and it's no longer a JSON file embedded with everything. You need to click next.

[nonprofittechy]

Here is the list of data we want to collect from scraping the court list:

1. Court URL on masscourts.gov, if any
2. Name
3. Any special notes. E.g.: https://www.mass.gov/locations/southeast-housing-court-barnstable-session says not to mail or file anything at this session. May overlap with the description below.
4. Description (with all description text from MassCourts. Usually this is a list of locations covered by the court)
5. fax
6. phone number of the clerk's office
7. If there are multiple phone numbers, we could grab those, but usually there's only one clerk's office number
8. Mailing Address
9. Physical address
10. ADA coordinators
11. Operating hours

[plocket]

Note that https://www.mass.gov/guides/courthouses-by-county seems to have the links to all the court pages [on one page]. Since we do actually need to visit the page to get this info, this is probably better than the paginated list linked above.

[plocket]

Someone should do a cursory check of the google spreadsheet I created as a first attempt. I'm not sure what anomalies to look for.

Artifacts:

• Repo with the raw generated csv. For now it's got a couple csv column obobs going on that create extra columns. It creates a log file if you run it locally.
• Google spreadsheet. Manually deleted the extra generated columns mentioned above.
• Initial brainstorming "discussion" in the repo. Contains sample HTML [of various relevant sections] from a few pages [I used as references that I linked in the discussion]. They were the first pages I opened and they had a fair amount of variety between them, so I suspect there are plenty of other inconsistencies on other pages. I tried to be flexible about how I captured information, so I may have gotten too much, but I suspect the code probably missed some as well.

Problems:

• As mentioned, an extra set of phone and fax columns from an obob I can take care of on the next iteration.
• Couldn't figure out where to find the mailing address in a way that didn't need to be parsed out of a "note".
• Wasn't sure where to find notes not related to the address (as per the provided example page)
• ADA coordinator names have some funky slashes and quotes
• Not sure what column labels would be most useful. Happy to adjust them as desired.

Notes:

• Max number of phone numbers found was 10.
• There's a lot of variability in the DOM. This code probably missed a bunch of info. If I get feedback on what it missed, I can adjust and run it again to save folks some time.
• I think it'd be good for you folks to have hold of this code in some way so you can have it in the future. I'm happy to transfer this repo to Suffolk or to put it somewhere in this repo if that would be useful.

[plocket]

Court URL on masscourts.gov, if any

Just noticed this. Do we expect some courts to not have pages? Would that be detectable on https://www.mass.gov/guides/courthouses-by-county or does that only list courts that do have pages? If not, I'm not sure how to include those in the csv.

[nonprofittechy]

Yeah, there will be a few courts that map on to an existing court (e.g., a session of the housing court) that we have a separate entry for in our database but never had a dedicated page on mass.gov created for them. In theory we could link to a page that was higher up in the hierarchy in that case.

I wouldn't worry about catching that kind of issue through scraping. All we can do is get the data that's on mass.gov, and assume ours includes more in some cases.

[plocket]

Already noticed some of the numbers are missing extensions. All the ones I've seen, though, are ones that are in the sidebar, not in the main phone number section (at the top of the page). Do those sidebar numbers matter? Example: row 46 linking for https://www.mass.gov/locations/lawrence-district-court.

[nonprofittechy]

No, extensions change. The main phone tree should get people to the right place still.