Internal server error for a NYTimes URL

[dhimmel]

Running the following command, I'm getting status code 500 for Internal Server Error

curl --verbose \
  --header "Content-Type: text/plain" \
  --data 'https://nyti.ms/1NuB0WJ' \
  'https://translate.manubot.org/web'

Full output:

* Connected to translate.manubot.org (35.221.11.188) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=translate.manubot.org
*  start date: Sep 23 15:02:01 2021 GMT
*  expire date: Dec 22 15:02:00 2021 GMT
*  subjectAltName: host "translate.manubot.org" matched cert's "translate.manubot.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
> POST /web HTTP/1.1
> Host: translate.manubot.org
> User-Agent: curl/7.74.0
> Accept: */*
> Content-Type: text/plain
> Content-Length: 23
> 
* upload completely sent off: 23 out of 23 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Server: nginx/1.14.0 (Ubuntu)
< Date: Sat, 06 Nov 2021 21:00:58 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< Connection: keep-alive
< 
* Connection #0 to host translate.manubot.org left intact

translate.manubot.org is a public translation server instance we host that is up to date with 9831fc31c5b24aa5f0a2de49091ec57d44b5e508.

I'll try to get the internal server logs when we have this error.

[dhimmel]

@dongbohu how do we see the logs for the translation-server process that is managed with supervisor?

[dongbohu]

@dhimmel: According to /etc/supervisor/conf.d/translation-server.conf:

• standard output is in /var/log/supervisor/translation-server.log
• standard error is in /var/log/supervisor/translation-server.err

[dstillman]

Works for us.

https://zbib.org/?q=https://nyti.ms/1NuB0WJ

[dstillman]

Now failing after a few tries from AWS. Still working for a local install. Possible they're rate-limiting.

TypeError: Cannot read property 'replace' of null

    TypeError: Cannot read property 'replace' of null
        at addHighwireMetadata (eval at <anonymous> (/var/task/src/translation/sandboxManager.js:70:4), <anonymous>:471:59)
        at completeItem (eval at <anonymous> (/var/task/src/translation/sandboxManager.js:70:4), <anonymous>:214:2)
        at eval (eval at <anonymous> (/var/task/src/translation/sandboxManager.js:70:4), <anonymous>:343:4)
        at /var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:384:8
        at Zotero.Translate.Import._runHandler (/var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:1128:32)
        at run (/var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:188:23)
        at /var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:250:6
        at new Promise (<anonymous>)
        at Object._itemDone (/var/task/modules/zotero/chrome/content/zotero/xpcom/translation/translate.js:248:11)
        at Object._itemDone (/var/task/src/translation/sandboxManager.js:96:17)

[dstillman]

I suspect they're serving something different in the cases where it's failing, but this looks like a bug in the Embedded Metadata translator — if I skip the call to addHighwireMetadata() it succeeds. We'll investigate.

[dstillman]

They're definitely also rate-limiting, though. After enough requests the server starts getting a 403 from nytimes.com, and then it returns 500. So it's possible that's all you're seeing.

[dstillman]

OK, rate-limiting and page differences aside, there was a regression in the Embedded Metadata translator from a couple weeks ago. I've pushed a fix, so if you were seeing the Cannot read property 'replace' of null error above, pull the latest translators and it should be fixed. If you're getting a 403, nothing we can do about that.

Thanks for reporting.

[dhimmel]

Thanks a lot @dstillman for https://github.com/zotero/translators/commit/0d435d8a952639d4e7489263b3a40c89377ecd31! Our CI builds are now back to 🟢.

pull the latest translators and it should be fixed

I did this using git submodule update --remote --merge which updated the submodules beyond the commits specified by zotero/translation-server currently for these submodules:

Submodule path 'modules/translate': merged in 'a9308c0e8632846ca2dc069a1b72db0a33f99ca6'
Submodule path 'modules/translators': merged in '0d435d8a952639d4e7489263b3a40c89377ecd31'
Submodule path 'modules/zotero-schema': merged in '97e0a8efa2cb2cf6c9853ceca334ec56180a9df0'

Is that okay, or is it best to just fast-forward modules/translators since the other two should be updated in lock-step with zotero/translation-server?

[dhimmel]

BTW we actually still get some CI failures:

But since it only happened in the later jobs, I bet it's rate limiting like @dstillman mentioned. One reason we should look into caching.

If you're getting a 403, nothing we can do about that.

I'm not actually sure how to see that. In the stderr logs for translation-server, this is what the failure looks like:

2021-11-07 10:31:22,151: 
  InternalServerError: An error occurred retrieving the document
      at Object.throw (/home/translate/translation-server/node_modules/koa/lib/context.js:97:11)
      at module.exports.WebSession.handleURL (/home/translate/translation-server/src/webSession.js:219:19)
      at <anonymous>
      at process._tickDomainCallback (internal/process/next_tick.js:228:7)

@dstillman where'd you see the TypeError: Cannot read property 'replace' of null log?

[dstillman]

I wouldn't use NYT for CI, since they rate-limit. Use something that will work reliably.

Is that okay, or is it best to just fast-forward modules/translators since the other two should be updated in lock-step with zotero/translation-server?

Yeah, definitely don't update the others. Just use git pull origin master in the translators submodule.

where'd you see the TypeError: Cannot read property 'replace' of null log?

It's just in the stdout from the server, which includes Zotero debug output (with lines beginning with, e.g., (3)(+0000010):).