usgpo / link-service

The link service is used to create links to content and metadata on govinfo
Other
89 stars 41 forks source link

Retrieving a list of valid years for a given section #23

Closed cgodwin1 closed 1 year ago

cgodwin1 commented 1 year ago

Hello, I am a developer on the eRegs project at https://github.com/CMSgov/cmcs-eregulations. We are trying to utilize your link service to retrieve historical PDF versions of regulation sections programmatically, however there does not seem to be a way to find out what years have valid responses.

For example, if a user wants to view all previous versions of 42 CFR 433.50, we would need a list of years that, when inserted into https://www.govinfo.gov/link/cfr/42/433?sectionnum=50&year={YEAR} would give a valid response.

Is there some way to do this with your service, or can you point us in the right direction? Thank you!

cgodwin1 commented 1 year ago

We estimate that to do this programmatically ourselves, we would need to verify a 200 status code from approximately 23,000 link service URLs ahead of time.

brittag commented 1 year ago

Hi! I work with @cgodwin1 on this project. As an alternative that would need fewer queries, I'd also be interested in giving our users links to historical PDFs at the subpart level. For example:

But the link service doesn't have subpart as a documented option (just title, part, section). Is that a possibility?

Thank you!

brittag commented 1 year ago

Hi @jonquandt! Just curious if there may be plans to add this kind of functionality in the future; we're probably going to assume we need to design this feature in a different way, which is okay too.

jonquandt commented 1 year ago

@brittag - this (providing a list of valid years) isn't functionality that I currently anticipate us adding as part of the link-service in the future, but we will consider it.

Are you trying to provide link service links on your site that users could then call, or are you trying to ingest historical PDFs into your system?

My suggestion would be to consider looking at the GovInfo API and then grabbing the mods for each CFR package. The MODS metadata includes information on the whole volume for a given title as well as its components, including down to the section level. You could then use this to identify which years have a given section, and then you could create the appropriate link-service calls.

Regarding subpart numbers, I believe that we can provide this already. It looks as though our link service documentation needs to be updated to reflect this. You can use subpartnum and appendix parameters to help with this.

example for 10 CFR Subpart A Appendix A https://www.govinfo.gov/link/cfr/10/51?appendix=a&subpartnum=A

42 CFR 433 Subpart A in 2021

brittag commented 1 year ago

Thank you! Yes, the idea is to provide link-service links that users could call. Using the MODS files is an interesting idea.

When I click your 42 CFR 433 Subpart A in 2021 link (https://www.govinfo.gov/link/cfr/42/433?subpartnum=A&year=2021), I get "No Results were found for the provided search criteria."

Good to know about the appendix parameter - this works: https://www.govinfo.gov/link/cfr/42/441?appendix=f

enlightenup commented 1 year ago

1974

brittag commented 1 year ago

OK to close this ticket; a while back my teammates figured out a way to implement the section-level feature we wanted. :)

Screenshot 2023-07-21 at 2 49 03 PM