API Timeout issues

[jeremycrumb]

We have encountered some instability recently. Have there been any changes to the timeout recently or do you have suggestions for how to prevent future timeouts?

Info:

'HTTPSConnectionPool(host='api.congress.gov', port=443): Read timed out. (read timeout=30)' instead of data Samples: ~Sep 6, 2023, 4:11:16 AM USA , request was https://api.congress.gov/v3/bill/118/hr/2057?format=json + token

Similar issue occurred ~Sep1 8:13 pm USA , request: https://api.congress.gov/v3/bill/118/hr/254?format=json + token

[mnewatloc]

We've been unable to reproduce any instability. There may be an occasional unplanned outage, but the system should recover within minutes. We do sometimes plan short outages during release deployments, but we try to ensure the API is available.

[jeremycrumb]

Hi Mike,

We are still having occasional issues with cloudflare. Any ideas of what we could try?

See below from our developers:

1.Looks like APi enabled "new Cloudflare protection with some limitations", "some time" API blocks our/CAS API requests and return Error 5** message instead of data,

2.See sample screenshot file with error message - c1_err.png. Request was https://api.congress.gov/v3/bill/118/hr/2616?format=json&api_key=....

@.*** Thank you for your help!

Sincerely,

Jeremy Crumb Manager, Source Acquisition & Management Legal Editorial Operations Thomson Reuters Phone: 763-326-5506 Mobile: 612-258-3391

@.**@.>

@.***

From: Mike Newman @.> Sent: Monday, September 11, 2023 1:32 PM To: LibraryOfCongress/api.congress.gov @.> Cc: Crumb, Jeremy (Content Operations) @.>; Author @.> Subject: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

External Email: Use caution with links and attachments.

We've been unable to reproduce any instability. There may be an occasional unplanned outage, but the system should recover within minutes. We do sometimes plan short outages during release deployments, but we try to ensure the API is available.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/LibraryOfCongress/api.congress.gov/issues/147*issuecomment-1714384724__;Iw!!GFN0sa3rsbfR8OLyAw!ZuIFzA2Klhr1a2n6c9omQ4_r2DXxwrXOJk26Nd07C6W7L0mOsHEeuwaaJDqO-b1ikT-6Bh-PU66ZSG-WUWh928sq0mkgvuTh-A$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A56GAVCRWCT2QX4ZAA5C7U3XZ5KIRANCNFSM6AAAAAA4PJF2NI__;!!GFN0sa3rsbfR8OLyAw!ZuIFzA2Klhr1a2n6c9omQ4_r2DXxwrXOJk26Nd07C6W7L0mOsHEeuwaaJDqO-b1ikT-6Bh-PU66ZSG-WUWh928sq0mnnFXnUFw$. You are receiving this because you authored the thread.Message ID: @.**@.>>

[jeremycrumb]

Good morning,

Just following up on this. We still are experiencing sporadic issues with this.

Thank you!

Jeremy Crumb Manager, Source Acquisition & Management Legal Editorial Operations Thomson Reuters Phone: 763-326-5506 Mobile: 612-258-3391

@.**@.>

@.***

From: Crumb, Jeremy (Content Operations) Sent: Friday, October 13, 2023 9:10 AM To: LibraryOfCongress/api.congress.gov @.>; LibraryOfCongress/api.congress.gov @.> Cc: Author @.***> Subject: RE: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

Hi Mike,

We are still having occasional issues with cloudflare. Any ideas of what we could try?

See below from our developers:

1.Looks like APi enabled "new Cloudflare protection with some limitations", "some time" API blocks our/CAS API requests and return Error 5** message instead of data,

2.See sample screenshot file with error message - c1_err.png. Request was https://api.congress.gov/v3/bill/118/hr/2616?format=json&api_key=....

@.*** Thank you for your help!

Sincerely,

Jeremy Crumb Manager, Source Acquisition & Management Legal Editorial Operations Thomson Reuters Phone: 763-326-5506 Mobile: 612-258-3391

@.**@.>

@.***

From: Mike Newman @.**@.>> Sent: Monday, September 11, 2023 1:32 PM To: LibraryOfCongress/api.congress.gov @.**@.>> Cc: Crumb, Jeremy (Content Operations) @.**@.>>; Author @.**@.>> Subject: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

External Email: Use caution with links and attachments.

We've been unable to reproduce any instability. There may be an occasional unplanned outage, but the system should recover within minutes. We do sometimes plan short outages during release deployments, but we try to ensure the API is available.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/LibraryOfCongress/api.congress.gov/issues/147*issuecomment-1714384724__;Iw!!GFN0sa3rsbfR8OLyAw!ZuIFzA2Klhr1a2n6c9omQ4_r2DXxwrXOJk26Nd07C6W7L0mOsHEeuwaaJDqO-b1ikT-6Bh-PU66ZSG-WUWh928sq0mkgvuTh-A$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A56GAVCRWCT2QX4ZAA5C7U3XZ5KIRANCNFSM6AAAAAA4PJF2NI__;!!GFN0sa3rsbfR8OLyAw!ZuIFzA2Klhr1a2n6c9omQ4_r2DXxwrXOJk26Nd07C6W7L0mOsHEeuwaaJDqO-b1ikT-6Bh-PU66ZSG-WUWh928sq0mnnFXnUFw$. You are receiving this because you authored the thread.Message ID: @.**@.>>

[ryparker]

Possibly related: https://github.com/LibraryOfCongress/api.congress.gov/issues/163

[jeremycrumb]

Good afternoon,

I’m checking in on this again. I’ve got a bit of additional information from our developers below.

1. Got fresh timeout error today from API : API command was: https://api.congress.gov/v3/bill/118/s/1260?format=json&api%5C_key=OUR_API_KEY Error response (+ we got html instead of json) INC5702766_dev_2.png

2. Cloudflare tool is occasionally blocking our API request? How can we prevent? @.***

Jeremy Crumb Manager, Source Acquisition & Management Legal Editorial Operations Thomson Reuters Phone: 763-326-5506 Mobile: 612-258-3391

@.**@.>

@.***

From: Crumb, Jeremy (Content Operations) Sent: Tuesday, November 7, 2023 9:31 AM To: 'LibraryOfCongress/api.congress.gov' @.>; 'LibraryOfCongress/api.congress.gov' @.> Cc: 'Author' @.***> Subject: RE: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

Good morning,

Just following up on this. We still are experiencing sporadic issues with this.

Thank you!

Jeremy Crumb Manager, Source Acquisition & Management Legal Editorial Operations Thomson Reuters Phone: 763-326-5506 Mobile: 612-258-3391

@.**@.>

@.***

From: Crumb, Jeremy (Content Operations) Sent: Friday, October 13, 2023 9:10 AM To: LibraryOfCongress/api.congress.gov @.**@.>>; LibraryOfCongress/api.congress.gov @.**@.>> Cc: Author @.**@.>> Subject: RE: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

Hi Mike,

We are still having occasional issues with cloudflare. Any ideas of what we could try?

See below from our developers:

1.Looks like APi enabled "new Cloudflare protection with some limitations", "some time" API blocks our/CAS API requests and return Error 5** message instead of data,

2.See sample screenshot file with error message - c1_err.png. Request was https://api.congress.gov/v3/bill/118/hr/2616?format=json&api_key=....

@.*** Thank you for your help!

Sincerely,

Jeremy Crumb Manager, Source Acquisition & Management Legal Editorial Operations Thomson Reuters Phone: 763-326-5506 Mobile: 612-258-3391

@.**@.>

@.***

From: Mike Newman @.**@.>> Sent: Monday, September 11, 2023 1:32 PM To: LibraryOfCongress/api.congress.gov @.**@.>> Cc: Crumb, Jeremy (Content Operations) @.**@.>>; Author @.**@.>> Subject: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

External Email: Use caution with links and attachments.

We've been unable to reproduce any instability. There may be an occasional unplanned outage, but the system should recover within minutes. We do sometimes plan short outages during release deployments, but we try to ensure the API is available.

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/LibraryOfCongress/api.congress.gov/issues/147*issuecomment-1714384724__;Iw!!GFN0sa3rsbfR8OLyAw!ZuIFzA2Klhr1a2n6c9omQ4_r2DXxwrXOJk26Nd07C6W7L0mOsHEeuwaaJDqO-b1ikT-6Bh-PU66ZSG-WUWh928sq0mkgvuTh-A$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A56GAVCRWCT2QX4ZAA5C7U3XZ5KIRANCNFSM6AAAAAA4PJF2NI__;!!GFN0sa3rsbfR8OLyAw!ZuIFzA2Klhr1a2n6c9omQ4_r2DXxwrXOJk26Nd07C6W7L0mOsHEeuwaaJDqO-b1ikT-6Bh-PU66ZSG-WUWh928sq0mnnFXnUFw$. You are receiving this because you authored the thread.Message ID: @.**@.>>

[mnewatloc]

We believe that the issues related to Cloudflare have been fixed. Can you please confirm if you have had Cloudflare issues between the 10th and 26th of November?

[jeremycrumb]

Hi Mike,

I want to introduce myself. I'm Anna Albrecht, from Thomson Reuters. I will be taking over for Jeremy Crumb as he will be leaving Thomson Reuters next week. Thank you for your quick reply. Unfortunately, we are still having some issues with the API. We are intermittently still getting "Connection timed out: Error Code 522". See the example error message with the time stamp below. Please let us know if you have any suggestions that we could try on our end. We really appreciate your assistance with this. Any help is greatly appreciated!

Thank you,

Anna

@.*** Anna Albrecht Operations Content - USA Tel: 651-253-7677

@. @. @.***

From: Crumb, Jeremy (Content Operations) @.**@.>> Sent: Thursday, December 7, 2023 8:29 AM To: Albrecht, Anna (Content Operations) @.**@.>> Subject: FW: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

Jeremy Crumb Manager, Source Acquisition & Management Legal Editorial Operations Thomson Reuters Phone: 763-326-5506 Mobile: 612-258-3391

@.**@.>

@.***

From: Mike Newman @.**@.>> Sent: Monday, December 4, 2023 9:08 AM To: LibraryOfCongress/api.congress.gov @.**@.>> Cc: Crumb, Jeremy (Content Operations) @.**@.>>; Author @.**@.>> Subject: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

External Email: Use caution with links and attachments.

We believe that the issues related to Cloudflare have been fixed. Can you please confirm if you have had Cloudflare issues between the 10th and 26th of November?

- Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/LibraryOfCongress/api.congress.gov/issues/147*issuecomment-1838842791__;Iw!!GFN0sa3rsbfR8OLyAw!Z4uPtGdMA9T_hN2N2FhLQe-crHHufSdYKH7Ll9bsKIdnGKhYhTvOXPRGOWLZq2JJLz8n4-b1bstBRWVx67HkgeA3TCVcIuggcA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A56GAVDPUSUY2AP3ZFSMHL3YHXRMVAVCNFSM6AAAAAA4PJF2NKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZYHA2DENZZGE__;!!GFN0sa3rsbfR8OLyAw!Z4uPtGdMA9T_hN2N2FhLQe-crHHufSdYKH7Ll9bsKIdnGKhYhTvOXPRGOWLZq2JJLz8n4-b1bstBRWVx67HkgeA3TCV5-Mg2Fw$. You are receiving this because you authored the thread.Message ID: @.**@.>>

[mnewatloc]

We've created a ticket to investigate this issue.

[rbram]

Hello. Would you please let us know if you are still experiencing this issue? Thanks.

[jeremycrumb]

Happy New Year Mike!

I have an additional question from our Tech team as we are still experiencing issues with the API. We are wondering if your website is updated via latest action date?

Thanks

Anna

@.*** Anna Albrecht Operations Content - USA Tel: 651-253-7677

From: Albrecht, Anna (Content Operations) @.> Sent: Friday, December 8, 2023 7:46 PM To: LibraryOfCongress/api.congress.gov @.>; LibraryOfCongress/api.congress.gov @.> Cc: Author @.>; Crumb, Jeremy (Content Operations) @.***> Subject: RE: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

Hi Mike,

I want to introduce myself. I'm Anna Albrecht, from Thomson Reuters. I will be taking over for Jeremy Crumb as he will be leaving Thomson Reuters next week. Thank you for your quick reply. Unfortunately, we are still having some issues with the API. We are intermittently still getting "Connection timed out: Error Code 522". See the example error message with the time stamp below. Please let us know if you have any suggestions that we could try on our end. We really appreciate your assistance with this. Any help is greatly appreciated!

Thank you,

Anna

@.*** Anna Albrecht Operations Content - USA Tel: 651-253-7677

@. @. @.***

From: Crumb, Jeremy (Content Operations) @.**@.>> Sent: Thursday, December 7, 2023 8:29 AM To: Albrecht, Anna (Content Operations) @.**@.>> Subject: FW: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

Jeremy Crumb Manager, Source Acquisition & Management Legal Editorial Operations Thomson Reuters Phone: 763-326-5506 Mobile: 612-258-3391

@.**@.>

@.***

From: Mike Newman @.**@.>> Sent: Monday, December 4, 2023 9:08 AM To: LibraryOfCongress/api.congress.gov @.**@.>> Cc: Crumb, Jeremy (Content Operations) @.**@.>>; Author @.**@.>> Subject: [EXT] Re: [LibraryOfCongress/api.congress.gov] API Timeout issues (Issue #147)

External Email: Use caution with links and attachments.

We believe that the issues related to Cloudflare have been fixed. Can you please confirm if you have had Cloudflare issues between the 10th and 26th of November?

- Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/LibraryOfCongress/api.congress.gov/issues/147*issuecomment-1838842791__;Iw!!GFN0sa3rsbfR8OLyAw!Z4uPtGdMA9T_hN2N2FhLQe-crHHufSdYKH7Ll9bsKIdnGKhYhTvOXPRGOWLZq2JJLz8n4-b1bstBRWVx67HkgeA3TCVcIuggcA$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/A56GAVDPUSUY2AP3ZFSMHL3YHXRMVAVCNFSM6AAAAAA4PJF2NKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZYHA2DENZZGE__;!!GFN0sa3rsbfR8OLyAw!Z4uPtGdMA9T_hN2N2FhLQe-crHHufSdYKH7Ll9bsKIdnGKhYhTvOXPRGOWLZq2JJLz8n4-b1bstBRWVx67HkgeA3TCV5-Mg2Fw$. You are receiving this because you authored the thread.Message ID: @.**@.>>

[mnewatloc]

Hi Anna,

We check with our data partners every 15 minutes and update if there is new data. So it depends on our data partners when we get new data. Not sure if that answers your question.

[AnnaAlbrechtTR]

@mnewatloc - We are still occasionally receiving API error/issues “Connection timed out: Error Code 522”, so just wondering if you have any suggestions or could we provide more information to help troubleshoot? Also, our developers noticed the API "push" response is similar to HTML (with 522 error code message) instead of pushing the expected JSON (with error message), do you know why that might be or think it might have anything to do with the issues we are experiencing? Thank you for your help and any suggestions you might have. -Anna

[rbram]

Thanks. We have a development ticket in process to fix this issue.

[rbram]

Hello. Could you please attach screenshots of your error messages? Thank you.

[AnnaAlbrechtTR]

Hello. Could you please attach screenshots of your error messages? Thank you.

Thomson Reuters Error code.docx

[rbram]

Hello. We are continuing to investigate these issues but did want to check in with you about rate limits. By default, the API returns 20 results starting with the first record. The 20 results limit can be adjusted up to 250 results. If the limit is adjusted to be greater than 250 results, only 250 results will be returned. We wanted to provide this information in case it was the rate limit causing the error messages. https://community.cloudflare.com/t/community-tip-fixing-error-522-connection-timed-out/42325

[AnnaAlbrechtTR]

Hello. We are continuing to investigate these issues but did want to check in with you about rate limits. By default, the API returns 20 results starting with the first record. The 20 results limit can be adjusted up to 250 results. If the limit is adjusted to be greater than 250 results, only 250 results will be returned. We wanted to provide this information in case it was the rate limit causing the error messages. https://community.cloudflare.com/t/community-tip-fixing-error-522-connection-timed-out/42325

Thank you @rbram , we will check into whether the rate limits are playing a role in the issue and follow up.

[AnnaAlbrechtTR]

Hello. We are continuing to investigate these issues but did want to check in with you about rate limits. By default, the API returns 20 results starting with the first record. The 20 results limit can be adjusted up to 250 results. If the limit is adjusted to be greater than 250 results, only 250 results will be returned. We wanted to provide this information in case it was the rate limit causing the error messages. https://community.cloudflare.com/t/community-tip-fixing-error-522-connection-timed-out/42325

Thank you @rbram , we will check into whether the rate limits are playing a role in the issue and follow up.

Hi @mnewatloc, following up, @rbram had suggested we consider the "limit" value of how many results are on a page, but the developers on my team said there is not an issue with the "bills_search" action using congress.gov/bill/{congress}/{billType} API. Also, the link @rbram shared above about fixing error 522 is for Cloudfare tool users, but we are not Cloudfare users, so I am not sure how that information would be helpful for our team?

Finally, we are still experiencing occasional issues with Cloudfare. Sometimes our data scraping fails (API return 522 error code) when we try to download all searched "detailed bills" information, >12000. bills from congress.gov API by "/bill/{congress}/{billType}/{billNumber}". Could the support team review our most recent failure? We also want to note that after the 522 failure, we got a new error after 60 seconds. I will send an email the with the error screenshot details

[mnewatloc]

We are still investigating raising the rate limit. I discussed this issue with our developers and we're not sure what else we can do.