Note how it attempts to include the entire qurl in the header link. I believe that this value is way too long because of all the domains in my giant query, throwing an error within requests when it is trying to be parsed. We don't actually use that link header for paging; in all the code I've seen we've use the x-resume-token that is included as a separate header just above. Since this is an internal API we could just remove this header, or add code to ensure that it is not too long.
I suggest making a change either way and deploying it to the staging release. It can be tested by creating a tunnel to the staging deployment and using the following proposed test case for mediacloud-news-client locally on a dev machine (which is how I triggered the error):
def test_header_too_long(self):
self._api.TIMEOUT_SECS = 500
with open(os.path.join(os.path.dirname(__file__), 'data', 'Collection-38379429-sources-20240611133536.csv')) as f:
reader = csv.DictReader(f)
all_domains = set([row['name'] for row in reader])
query = '"new trial" AND canonical_domain:({})'.format(" OR ".join(all_domains))
start_date = dt.datetime(2024, 1, 1)
end_date = dt.datetime(2024, 6, 16)
page1, next_token1 = self._api.paged_articles(query, start_date, end_date)
assert len(page1) > 0
If I run a query that includes lots of domains, I get an error:
This prevents me from downloading the story list CSV in the search tools, and even making API queries doesn't work. Looking at this code, I believe the critical line causing the error is this one: https://github.com/mediacloud/news-search-api/blob/0a25568dafafc5fbb53706e7d20d23ffad6350e1/api.py#L512
Note how it attempts to include the entire
qurl
in the header link. I believe that this value is way too long because of all the domains in my giant query, throwing an error withinrequests
when it is trying to be parsed. We don't actually use thatlink
header for paging; in all the code I've seen we've use thex-resume-token
that is included as a separate header just above. Since this is an internal API we could just remove this header, or add code to ensure that it is not too long.I suggest making a change either way and deploying it to the staging release. It can be tested by creating a tunnel to the staging deployment and using the following proposed test case for
mediacloud-news-client
locally on a dev machine (which is how I triggered the error):(Collection-38379429-sources-20240611133536.csv)