search

usgpo / bill-status

Information about Bill Status XML Bulk Data including the XML User Guide.
https://www.govinfo.gov/bulkdata/BILLSTATUS
159 stars 47 forks source link

Missing xml url data in BILLSTATUS'es #219

Closed BorodaUA closed 8 months ago

BorodaUA commented 1 year ago

Hello,

I work with @aih on public projects related to bill and legislative data. In analyzing the GPO API, we have found packages that are missing certain information. The purpose of this issue is to document those packages where some information (typically textVersions) is missing from the GPO API.

Here are items where BILLSTATUS doesn't have a url for the package in textVersions. To confirm, open BILLSTATUS URL from the table below and search(Ctrl+F) for "textVersions" to see that the url is missing. BILLSTATUS ID BILLSTATUS URL Expected Package URL Actual Package URL Notes
115hr5896 115hr5896 115hr5896rh None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr6762 115hr6762 115hr6762ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr6389 115hr6389 115hr6389ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr6342 115hr6342 115hr6342ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr6176 115hr6176 115hr6176ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr6063 115hr6063 115hr6063ih None The url for this version is missing from BILLSTATUS in textVersions>formats
113hr1067 113hr1067 113hr1067enr None The url for this version is missing from BILLSTATUS in textVersions>formats
113hr1068 113hr1068 113hr1068enr None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr3241 117hr3241 117hr3241rh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr3239 117hr3239 117hr3239rh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr3241 117hr3241 117hr3241rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr3241 117hr3241 117hr3241eh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr3239 117hr3239 117hr3239rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr3239 117hr3239 117hr3239eh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5695 117hr5695 117hr5695ih None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5677 117hr5677 117hr5677ih None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5705 117hr5705 117hr5705ih None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5679 117hr5679 117hr5679ih None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5718 117hr5718 117hr5718ih None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5677 117hr5677 117hr5677rh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5695 117hr5695 117hr5695rh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5679 117hr5679 117hr5679eh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5695 117hr5695 117hr5695eh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5705 117hr5705 117hr5705eh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5677 117hr5677 117hr5677eh None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5679 117hr5679 117hr5679rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5695 117hr5695 117hr5695rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5677 117hr5677 117hr5677rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
117hr5705 117hr5705 117hr5705rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
116hres755 116hres755 116hres755eh None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr5295 115hr5295 115hr5295ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr5293 115hr5293 115hr5293ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr5297 115hr5297 115hr5297ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr5296 115hr5296 115hr5296ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr5344 115hr5344 115hr5344ih None The url for this version is missing from BILLSTATUS in textVersions>formats
115hr1625 115hr1625 115hr1625eah None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2832 114hr2832 114hr2832ih None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2834 114hr2834 114hr2834ih None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2831 114hr2831 114hr2831ih None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2830 114hr2830 114hr2830ih None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2832 114hr2832 114hr2832eh None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2831 114hr2831 114hr2831eh None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2830 114hr2830 114hr2830eh None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2831 114hr2831 114hr2831rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2830 114hr2830 114hr2830rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
114hr2832 114hr2832 114hr2832rfs None The url for this version is missing from BILLSTATUS in textVersions>formats
118sres77 118sres77 118sres77ats None The url for this version is missing from BILLSTATUS in textVersions>formats
118hres383 118hres383 118hres383eh None The url for this version is missing from BILLSTATUS in textVersions>formats
jonquandt commented 1 year ago

I am looking into this - not that at least for the first item in the table, that BILLS package does not have an available XML version.

https://api.govinfo.gov/packages/BILLS-115hr5896rh/summary?api_key=DEMO_KEY

I did see that there is at least one BILLSTATUS XML that does not include text version URLs, so checking on that as well.

https://www.govinfo.gov/bulkdata/BILLSTATUS/114/hr/BILLSTATUS-114hr2830.xml

jonquandt commented 1 year ago

Good afternoon --

I have done some checking and can confirm that there is no XML available for any of the BILLS packages in your list. This means that GovInfo did not receive XML at the time of ingest of those packages. I don't have any specific reasons why those bills may not have XML, but that may be something that House Staff can help determine.

The BILLS ID field links to the API package summary, where you can see the available downloadable content.

BILLSTATUS ID BILLS ID XML Exists? Congress Bill Type Bill Number Bill Version Title
BILLSTATUS-115hr5896 BILLS-115hr5896rh No 115 hr 5896 rh To amend title 5, United States Code, to modify the authority for pay and work schedules of border patrol agents, and for other purposes.
BILLSTATUS-115hr6762 BILLS-115hr6762ih No 115 hr 6762 ih H.R. 6762 (IH) - Introduced in House
BILLSTATUS-115hr6389 BILLS-115hr6389ih No 115 hr 6389 ih H.R. 6389 (IH) - Introduced in House
BILLSTATUS-115hr6342 BILLS-115hr6342ih No 115 hr 6342 ih H.R. 6342 (IH) - Introduced in House
BILLSTATUS-115hr6176 BILLS-115hr6176ih No 115 hr 6176 ih H.R. 6176 (IH) - Introduced in House
BILLSTATUS-115hr6063 BILLS-115hr6063ih No 115 hr 6063 ih H.R. 6063 (IH) - Introduced in House
BILLSTATUS-113hr1067 BILLS-113hr1067enr No 113 hr 1067 enr An act h.R. 1067 (ENR) - Enrolled Bill"
BILLSTATUS-113hr1068 BILLS-113hr1068enr No 113 hr 1068 enr An act h.R. 1068 (ENR) - Enrolled Bill"
BILLSTATUS-117hr3241 BILLS-117hr3241rh No 117 hr 3241 rh To make improvements in the enactment of title 54, United States Code, into a positive law title and to improve the Code.
BILLSTATUS-117hr3239 BILLS-117hr3239rh No 117 hr 3239 rh To make improvements in the enactment of title 41, United States Code, into a positive law title and to improve the Code.
BILLSTATUS-117hr3241 BILLS-117hr3241rfs No 117 hr 3241 rfs An act h.R. 3241 (RFS) - Referred in Senate"
BILLSTATUS-117hr3241 BILLS-117hr3241eh No 117 hr 3241 eh An act to make improvements in the enactment of title 54, United States Code, into a positive law title and to improve the Code.
BILLSTATUS-117hr3239 BILLS-117hr3239rfs No 117 hr 3239 rfs An act h.R. 3239 (RFS) - Referred in Senate"
BILLSTATUS-117hr3239 BILLS-117hr3239eh No 117 hr 3239 eh An act to make improvements in the enactment of title 41, United States Code, into a positive law title and to improve the Code.
BILLSTATUS-117hr5695 BILLS-117hr5695ih No 117 hr 5695 ih H.R. 5695 (IH) - Introduced in House
BILLSTATUS-117hr5677 BILLS-117hr5677ih No 117 hr 5677 ih H.R. 5677 (IH) - Introduced in House
BILLSTATUS-117hr5705 BILLS-117hr5705ih No 117 hr 5705 ih H.R. 5705 (IH) - Introduced in House
BILLSTATUS-117hr5679 BILLS-117hr5679ih No 117 hr 5679 ih H.R. 5679 (IH) - Introduced in House
BILLSTATUS-117hr5718 BILLS-117hr5718ih No 117 hr 5718 ih Prince Congressional Gold Medal Act"
BILLSTATUS-117hr5677 BILLS-117hr5677rh No 117 hr 5677 rh To make technical amendments to update statutory references to certain provisions classified to title 2, United States Code, title 50, United States Code, and title 52, United States Code.
BILLSTATUS-117hr5695 BILLS-117hr5695rh No 117 hr 5695 rh To make technical amendments to update statutory references to certain provisions which were formerly classified to chapters 14 and 19 of title 25, United States Code.
BILLSTATUS-117hr5679 BILLS-117hr5679eh No 117 hr 5679 eh An act to make technical amendments to update statutory references to certain provisions classified to title 7, title 20, and title 43, United States Code.
BILLSTATUS-117hr5695 BILLS-117hr5695eh No 117 hr 5695 eh An act to make technical amendments to update statutory references to certain provisions which were formerly classified to chapters 14 and 19 of title 25, United States Code.
BILLSTATUS-117hr5705 BILLS-117hr5705eh No 117 hr 5705 eh An act to make technical amendments to update statutory references to provisions reclassified to title 34, United States Code.
BILLSTATUS-117hr5677 BILLS-117hr5677eh No 117 hr 5677 eh An act to make technical amendments to update statutory references to certain provisions classified to title 2, United States Code, title 50, United States Code, and title 52, United States Code.
BILLSTATUS-117hr5679 BILLS-117hr5679rfs No 117 hr 5679 rfs An act h.R. 5679 (RFS) - Referred in Senate"
BILLSTATUS-117hr5695 BILLS-117hr5695rfs No 117 hr 5695 rfs An act h.R. 5695 (RFS) - Referred in Senate"
BILLSTATUS-117hr5677 BILLS-117hr5677rfs No 117 hr 5677 rfs An act h.R. 5677 (RFS) - Referred in Senate"
BILLSTATUS-117hr5705 BILLS-117hr5705rfs No 117 hr 5705 rfs An act h.R. 5705 (RFS) - Referred in Senate"
BILLSTATUS-116hres755 BILLS-116hres755eh No 116 hres 755 eh H. RES. 755 (EH) - Engrossed in House"
BILLSTATUS-115hr5295 BILLS-115hr5295ih No 115 hr 5295 ih H.R. 5295 (IH) - Introduced in House
BILLSTATUS-115hr5293 BILLS-115hr5293ih No 115 hr 5293 ih H.R. 5293 (IH) - Introduced in House
BILLSTATUS-115hr5297 BILLS-115hr5297ih No 115 hr 5297 ih H.R. 5297 (IH) - Introduced in House
BILLSTATUS-115hr5296 BILLS-115hr5296ih No 115 hr 5296 ih H.R. 5296 (IH) - Introduced in House
BILLSTATUS-115hr5344 BILLS-115hr5344ih No 115 hr 5344 ih H.R. 5344 (IH) - Introduced in House
BILLSTATUS-115hr1625 BILLS-115hr1625eah No 115 hr 1625 eah An Act to amend the State Department Basic Authorities Act of 1956 to include severe forms of trafficking in persons within the definition of transnational organized crime for purposes of the rewards program of the Department of State, and for other purposes.
BILLSTATUS-114hr2832 BILLS-114hr2832ih No 114 hr 2832 ih H.R. 2832 (IH) - Introduced in House
BILLSTATUS-114hr2834 BILLS-114hr2834ih No 114 hr 2834 ih H.R. 2834 (IH) - Introduced in House
BILLSTATUS-114hr2831 BILLS-114hr2831ih No 114 hr 2831 ih H.R. 2831 (IH) - Introduced in House
BILLSTATUS-114hr2830 BILLS-114hr2830ih No 114 hr 2830 ih To make technical amendments to update statutory references to certain provisions classified to title 2, United States Code
BILLSTATUS-114hr2832 BILLS-114hr2832eh No 114 hr 2832 eh An act to make technical amendments to update statutory references to certain provisions classified to title 52, United States Code.
BILLSTATUS-114hr2831 BILLS-114hr2831eh No 114 hr 2831 eh An act to make technical amendments to update statutory references to provisions classified to chapters 44, 45, 46, and 47 of title 50, United States Code.
BILLSTATUS-114hr2830 BILLS-114hr2830eh No 114 hr 2830 eh An act to make technical amendments to update statutory references to certain provisions classified to title 2, United States Code.
BILLSTATUS-114hr2831 BILLS-114hr2831rfs No 114 hr 2831 rfs An act h.R. 2831 (RFS) - Referred in Senate"
BILLSTATUS-114hr2830 BILLS-114hr2830rfs No 114 hr 2830 rfs An act h.R. 2830 (RFS) - Referred in Senate"
BILLSTATUS-114hr2832 BILLS-114hr2832rfs No 114 hr 2832 rfs An act h.R. 2832 (RFS) - Referred in Senate"
BILLSTATUS-118sres77 BILLS-118sres77ats No 118 sres 77 ats S. RES. 77 (ATS) - Agreed to Senate"
BILLSTATUS-118hres383 BILLS-118hres383eh No 118 hres 383 eh H. RES. 383 (EH) - Engrossed in House"
aih commented 1 year ago

Thank you so much for undertaking this review, @jonquandt! For downstream tools, and users it is very helpful to confirm that the XML is missing for particular documents. We'll do some investigation about these to see, if there is a pattern for these (e.g. some appear to be technical amendments).

jonquandt commented 1 year ago

@aih - happy to help!

A potential alternative would be to use the related endpoint against given BILLSTATUS package IDs to determine which bill text versions exist and then grab the xmlLink if available. I assume the html/text wouldn't be useful for you, but that could be an alternative format.

That presumably is less efficient from a request perspective than just getting the BILLSTATUS xml and pull text versions, but then you would be sure that the xml is actually available or not.

That being said, the upstream Congress.gov API that our BILLSTATUS xml relies on uses our API to grab bill information, so if an xml textVersion doesn't exist for a bill, that will generally be because it does not exist on GovInfo.

aih commented 1 year ago

I think this is the sequence we used to follow, but the Billstatus -> textversion sequence is so much more efficient (I think that was one of the later improvements in the API). We now have a number of years of examples (above) that you've confirmed that the missing textVersion means the xml is missing. I think we'll take that as the indicator for now (but we'll still make some inquiries why these bills are missing xml).

jonquandt commented 1 year ago

Totally fair.

Thinking about it more, If any other textVesion formats exist - e.g. pdf or html, that's a good indicator that the package exists on GovInfo, but that we don't have the XML.