Inconsistent results in web endpoint

[mtrojan-ub]

Downloading the same URL 10 times in a row results in about 2 correct and 8 empty results

#!/bin/bash
#
for i in `seq 1 10`;
do
        curl -d 'http://journals.sagepub.com/doi/abs/10.1177/0004865818786763?ai=2b4&mi=ehikzz&af=R' -H 'Content-Type: text/plain' http://127.0.0.1:1969/web
done;

2 times

[{"key":"BF48MMAQ","version":0,"itemType":"journalArticle","creators":[{"firstName":"Suzanna","lastName":"Fay","creatorType":"author"},{"firstName":"Robert","lastName":"Crutchfield","creatorType":"author"}],"tags":[],"title":"Perceptions of “others,” risk, and counter terrorism-related informal social control","date":"August 8, 2018","DOI":"10.1177/0004865818786763","publicationTitle":"Australian & New Zealand Journal of Criminology","journalAbbreviation":"Australian & New Zealand Journal of Criminology","pages":"0004865818786763","abstractNote":"Anti-terrorism messages associate immigration and minorities with terrorism even if this link is not explicit. The consequence is the potential for racial profiling of minorities as threats to national security. Recent experiences or threats of domestic terrorism, in Australia, the US, and other industrialized countries, have led policy makers to encourage informal social control in terrorism prevention efforts by appealing to citizens to report suspicious behavior to authorities. The linking of ethnically different “others”—members of Australia’s population who are or are perceived to be outsiders to the mainstream—and terrorism is important because who is seen as threatening effects whether individuals engage in informal social control; the willingness of residents to recognize, intervene, and report suspicious behavior. However, the concept of “others” in relation to informal social control is more complicated than just immigration status and ethnic identity alone. This study examines whether perceptions of “others” are related to perceptions of terrorism risk and perceptions of informal social control in reporting national security threats.","ISSN":"0004-8658","url":"https://doi.org/10.1177/0004865818786763","language":"en","libraryCatalog":"SAGE Journals","accessDate":"2018-08-24T11:37:45Z"}]

8 times

[]

Even if you use different urls, we noticed that occasionaly some results are empty and some are not, but i think the same url is a good test case. Sequence is not always the same, sometimes e.g. only attempt 3 and 7 are correct, sometimes only attempt 5, ...

/edit: This might be URL specific, using e.g. https://www.nytimes.com/2018/06/11/technology/net-neutrality-repeal.html instead leads to 10 of 10 successful results

[adomasven]

@dstillman is it at all possible that the old translator-server maintains a global session with related cookies?

[dstillman]

@adomasven: Yes, I imagine the old server persisted cookies for sites at least as long as it was running. (The Docker version would've lost those cookies on next startup, while a non-Docker install would preserve them forever.)

I think not preserving cookies beyond a single save will generally lead to better results, since it will avoid paywalls that trigger after repeated visits. Google sites will probably be more inclined to block, but I suspect they're the exception.

How do you think that's contributing to this issue?