openstates / openstates-scrapers

source for Open States scrapers
https://openstates.org
GNU General Public License v3.0
845 stars 464 forks source link

HI "carried over" bills bug? #1196

Closed pqnelson closed 7 years ago

pqnelson commented 7 years ago

I just found out this fascinating situation where HI can carry over bills from a regular session in an odd year (e.g., 2013) to the next regular session in an even year (e.g., 2014). The bills have the same bill number (e.g., HB882).

But Openstates appears to be making duplicate bills, e.g., the 2013 version [internal id HIB00009746] and the 2014 version [id HIB00014752].

I'm not certain what the "correct" or "best" solution is, whether it should be (i) to continue using the 2013 version of the bill and append to its actions the 2014 actions, or (ii) create 2 bill records representing the same bill but have some field to indicate that it's a "carry over" of HIB00009746?

Either way, I thought I should flag it to someone's attention.

(I have checked the other states which can carry-over bills and have 1-year-long session records, namely ME, VA, WV, WI --- the rest of them seem fine, it's just a HI issue.)

showerst commented 7 years ago

Carryover laws vary by state, in some states it's just the same text and bill number, but the actions and stages are considered 'reset', in others everything carries over and it's picked up as-is.

I'm not sure what the best way to handle this is from a database perspective, or a scraper standpoint either.

Does the state site provide any information we could use to link the bill back?

jamesturk commented 7 years ago

We actually do have the concept of a cross-session link between bills, but they aren't used for much of anything right now.

They were added for situations like this, for a number of reasons we want to keep the two bills distinct, but we would like to ideally provide a link between them.

On Tue, Dec 20, 2016 at 1:44 PM, showerst notifications@github.com wrote:

Carryover laws vary by state, in some states it's just the same text and bill number, but the actions and stages are considered 'reset', in others everything carries over and it's picked up as-is.

I'm not sure what the best way to handle this is from a database perspective, or a scraper standpoint either.

Does the state site provide any information we could use to link the bill back?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openstates/openstates/issues/1196#issuecomment-268323491, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAfYgK82d65Tp7KLVL4fJcWwVY-kxvdks5rKCIfgaJpZM4LSJHR .

paultag commented 7 years ago

Some states even show the bill ID from last session(s), which is handy when scraping bill pages

pqnelson commented 7 years ago

@showerst

Regarding the carryover practices, true, it varies state-to-state. (Part of my job, alas, is to find out how it varies.) The legislator handbook for Hawaii states:

One important feature of a carry-over bill, that is a bill not enacted during the first year of a session which carries over into the second year with the same status it held at the end of the first session, is that the carryover bill must pass at least one reading in the house in which it was introduced during the second carryover session. Thus, a Senate bill that passed three readings in the Senate and was transmitted to the House in the first year must pass an additional reading in the Senate during the second year even if it is returned by the House in unamended form.

Since 1969, the bill numbers (for the carried over bills) are likewise carried over. Apparently they had a constitutional convention in '68, and wrote it into the thing as article 3 section 15, which specifically states "Any bill pending at the final adjournment of a regular session in an odd-numbered year shall carry over with the same status to the next regular session".

For Hawaii, it seems that implicitly the bill numbers are the same, i.e., the two bills are one-in-the-same (as they are not reintroduced within the biennium, hence the clerks do not assign new numbers to "carried over" bills).

pqnelson commented 7 years ago

@jamesturk

That linkage might also be useful for "companion bills", e.g., HI's HB882-2013 and SB1113-2013.

But, for HI at least, the carried-over bills are just the original bills "kept alive" for another year...as opposed to a distinct "clone" of the original.

jamesturk commented 7 years ago

yep, in fact Bill.add_companion is indeed intended to be used for both cases in the billy model

https://github.com/openstates/billy/blob/7763b7d705c0e120c157fe86dc5a72924f2c3277/billy/scrape/bills.py#L202

It looks like right now it is used in MN, NJ, and NY. We don't distinguish between the types right now, but at least the relation will exist.

& In the OCD schema we're actually specifically attaching types to bill relationships:

https://github.com/opencivicdata/python-opencivicdata-django/blob/master/opencivicdata/common.py#L99

jamesturk commented 7 years ago

solution to this is to use add_companion in HI bills scraper and specify the prior session

probably a fairly light lift if someone is working in HI