Some requests to https://api.audnex.us/ failing with 504 gateway time out error

rloomans commented 2 years ago

Some requests to https://api.audnex.us/ are failing with a 504 error...

% curl -v https://api.audnex.us/books/B083QSLMGD
*   Trying 104.21.52.190:443...
* Connected to api.audnex.us (104.21.52.190) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Aug  9 00:00:00 2022 GMT
*  expire date: Aug  9 23:59:59 2023 GMT
*  subjectAltName: host "api.audnex.us" matched cert's "*.audnex.us"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd57700ca00)
> GET /books/B083QSLMGD HTTP/2
> Host: api.audnex.us
> user-agent: curl/7.79.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 504 
< date: Wed, 07 Sep 2022 03:52:53 GMT
< content-type: text/html
< cf-cache-status: BYPASS
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=XbNWX7BJKnf2FzhMoHk%2FNjPNHNTHYR7cAQhgGUAHmmVMiCZCPRbPh%2B1iGdiFdfYjKJmEn%2FoOQDgygl11zXZmjxItjhxEGo2cw59fJ0vg89Ikir53GUP9ZfraX9SDB35t"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 746c7d1059878986-SIN
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
< 
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host api.audnex.us left intact

While others are fine...

% curl -v https://api.audnex.us/books/0241548705
*   Trying 172.67.202.222:443...
* Connected to api.audnex.us (172.67.202.222) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Aug  9 00:00:00 2022 GMT
*  expire date: Aug  9 23:59:59 2023 GMT
*  subjectAltName: host "api.audnex.us" matched cert's "*.audnex.us"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe9b7811400)
> GET /books/0241548705 HTTP/2
> Host: api.audnex.us
> user-agent: curl/7.79.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Wed, 07 Sep 2022 03:54:24 GMT
< content-type: application/json; charset=utf-8
< content-length: 2100
< vary: Origin
< last-modified: Fri, 02 Sep 2022 03:02:47 GMT
< cache-control: max-age=432000
< cf-cache-status: HIT
< accept-ranges: bytes
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=OmAZbsZahiSZNkqjgFKNkh%2Ft2zKdukVvXjTPUC%2FM1pf3VvhXXrovO8ofHAuK7he70Em5mrDkIICP3W4JOC2y7S6Ne6kpd%2FhyqpI74%2FqrqDvk%2FVytxJlamiAnv%2FljUgzv"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 746c80bff87187d1-SIN
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
< 
* Connection #0 to host api.audnex.us left intact
{"asin":"0241548705","authors":[{"asin":"B000AQ74C6","name":"Philip Pullman"}],"description":"Lyra finds herself in a shimmering, haunted otherworld - Cittàgazze, where soul-eating Spectres stalk the streets and wingbeats of distant angels sound against the sky. But she is not without allies....","formatType":"unabridged","image":"https://m.media-amazon.com/images/I/A1hzSma-3gS.jpg","language":"english","narrators":[{"name":"Philip Pullman"},{"name":"full cast"}],"publisherName":"Penguin Audio","rating":"4.9","releaseDate":"2021-06-24T00:00:00.000Z","runtimeLengthMin":533,"subtitle":"His Dark Materials, Book 2","summary":"<p><b>Brought to you by Penguin.</b></p> <p><b>From the world of Philip Pullman's </b><b><i>His Dark Materials - </i></b><b>now a major critically acclaimed BBC series.</b><br /> <br /> <b><i>She had asked: what is he? A friend or an enemy?</i></b><br /> <b><i>The alethiometer answered: he is a murderer.</i></b><br /> <b><i>When she saw the answer, she relaxed at once.</i></b></p> <p>Lyra finds herself in a shimmering, haunted otherworld - Cittàgazze, where soul-eating Spectres stalk the streets and wingbeats of distant angels sound against the sky.</p> <p>But she is not without allies: 12-year-old Will Parry, fleeing for his life after taking another's, has also stumbled into this strange new realm.</p> <p>On a perilous journey from world to world, Lyra and Will uncover a deadly secret: an object of extraordinary and devastating power.</p> <p>And with every step, they move closer to an even greater threat - and the shattering truth of their own destiny.</p> <p><b>Read by Philip Pullman and a full cast of narrators.</b></p>","title":"The Subtle Knife","genres":[{"asin":"18580715011","name":"Teen & Young Adult","type":"genre"},{"asin":"18581048011","name":"Science Fiction & Fantasy","type":"genre"},{"asin":"18581054011","name":"Dark Fantasy","type":"tag"},{"asin":"18581055011","name":"Epic","type":"tag"},{"asin":"18581057011","name":"Magical Realism","type":"tag"}],"seriesPrimary":{"asin":"B006K1ML9G","name":"His Dark Materials","position":"2"}}

As an additional datapoint, I'm currently running a local container with laxamentumtech/audnexus:develop and it's retrieving the same ASINs fine.

% curl -v https://audnexus.example.com/books/B083QSLMGD
*   Trying 192.168.10.6:443...
* Connected to audnexus.example.com (192.168.10.6) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.example.com
*  start date: Aug 14 16:00:27 2022 GMT
*  expire date: Nov 12 16:00:26 2022 GMT
*  subjectAltName: host "audnexus.example.com" matched cert's "*.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fab06811400)
> GET /books/B083QSLMGD HTTP/2
> Host: audnexus.example.com
> user-agent: curl/7.79.1
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx/1.18.0
< date: Wed, 07 Sep 2022 04:04:28 GMT
< content-type: application/json; charset=utf-8
< content-length: 2222
< vary: Origin
< x-robots-tag: noindex, nofollow, nosnippet, noarchive
<
* Connection #0 to host audnexus.example.com left intact
{"asin":"B083QSLMGD","authors":[{"asin":"B07C9483MP","name":"David Reiss"}],"description":"His reputation had been secure. He was the world's most feared supervillain! But then he saved the world....","formatType":"unabridged","genres":[{"asin":"18580606011","name":"Science Fiction & Fantasy","type":"genre"},{"asin":"18580607011","name":"Fantasy","type":"tag"},{"asin":"18580626011","name":"Superhero","type":"tag"},{"asin":"18580628011","name":"Science Fiction","type":"tag"},{"asin":"18580629011","name":"Adventure","type":"tag"}],"image":"https://m.media-amazon.com/images/I/810t6JP87aL.jpg","language":"english","narrators":[{"name":"George Napier"}],"publisherName":"Atian Press","rating":"4.8","releaseDate":"2020-01-11T00:00:00.000Z","runtimeLengthMin":576,"seriesPrimary":{"asin":"B07R3QSR7S","name":"The Chronicles of Fid","position":"2"},"subtitle":"The Chronicles of Fid, Book 2","summary":"<p><b>His reputation had been secure. </b></p> <p><b>He was the world's most feared supervillain! </b></p> <p><b><i>But then he saved the world....</i></b></p> <p>News cameras had captured every moment of the battle in which Doctor Fid single-handedly averted an alien invasion. As details emerge, the public discovers how close the Earth had come to inescapable subjugation...or to complete annihilation. </p> <p>In the aftermath, there are many who wonder if the veteran supervillain has changed his ways. There are many who think that Doctor Fid may not be a monster after all. </p> <p>Notoriety is important to many of Doctor Fid's long-term plans to punish the unworthy, and this shift in public perception threatens to undermine decades' worth of effort.... But it also presents a tempting opportunity. </p> <p>New dangers arise, and through it all Doctor Fid must struggle to decide what role he will play. Can the notorious supervillain set aside his endless quest? Can Doctor Fid become the hero that the world needs? </p> <p>Or will he remain the villain that the world's heroes deserve? </p> <p><b>Any fan of the superhero genre will love this supervillainous novel. Read book two in the series that critics have called innovative, snarky, and ridiculously fun!</b></p>","title":"Behind Distant Stars"}

rloomans commented 2 years ago

As of about 5 minutes ago, it seems to be ok again

djdembeck commented 2 years ago

There was an issue with the server that should be resolved now. I am keeping an eye on it for a while to make sure it won't happen again.

Thanks for the report!

rloomans commented 2 years ago

There are a bunch that are now giving me a 500 error instead.

eg

% curl -v https://api.audnex.us/books/B109400077
*   Trying 172.67.202.222:443...
* Connected to api.audnex.us (172.67.202.222) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Aug  9 00:00:00 2022 GMT
*  expire date: Aug  9 23:59:59 2023 GMT
*  subjectAltName: host "api.audnex.us" matched cert's "*.audnex.us"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb83d80d400)
> GET /books/B109400077 HTTP/2
> Host: api.audnex.us
> user-agent: curl/7.79.1
> accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 500 
< date: Wed, 07 Sep 2022 05:35:37 GMT
< content-type: application/json; charset=utf-8
< content-length: 114
< vary: Origin
< cf-cache-status: BYPASS
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=zF1eT34B%2FqZD6RDa8qXk8sM2BST6ONP%2BK5p9izNg4TdzczMHARJwL9IypFGNf%2Bdqp13ZcixVt1CMqPP8RH%2Fhm%2B20p23aT2knMlXExekD1idr%2FBKUQ8WMK9UtcIFJbn80"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 746d14f51e158802-SIN
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
< 
* Connection #0 to host api.audnex.us left intact
{"statusCode":500,"error":"Internal Server Error","message":"Required keys not found in API response: B109400077"}

The problematic ASINs I've seen so far are: B109400077, B172133685, B177230695, B177424023, B177424247, B179990641, B198263383, B198267584

djdembeck commented 2 years ago

The one with the error does not have data on the api:

{"statusCode":500,"error":"Internal Server Error","message":"Required key 'authors' does not exist in Audible API response for ASIN B109400077"}

are these audible.com asins?

trustosas commented 2 years ago

I'm facing the same issue on Neurrone's beet-audible:

!beet import "/content/Animal Farm"

/usr/local/lib/python3.7/dist-packages/beets/mediafile.py:19: UserWarning: beets.mediafile is deprecated; use mediafile instead
  warnings.warn("beets.mediafile is deprecated; use mediafile instead")
Error: MusicBrainz not reachable in release search with query {'release': 'animal farm', 'artist': 'george orwell', 'tracks': '1'}
Error while requesting https://api.audnex.us/books/B002V5H6F4/chapters, attempt 1/3: status code 504, Gateway Timeout
Error while requesting https://api.audnex.us/books/B002V5H6F4/chapters, attempt 2/3: status code 504, Gateway Timeout
Error while requesting https://api.audnex.us/books/B002V5H6F4/chapters, attempt 3/3: status code 504, Gateway Timeout
audible: Error while fetching book information from Audnex
Traceback (most recent call last):
  File "/content/beets-audible/audible.py", line 291, in get_albums
    return [self.get_album_info(p["asin"]) for p in products_without_unreleased_entries]
  File "/content/beets-audible/audible.py", line 291, in <listcomp>
    return [self.get_album_info(p["asin"]) for p in products_without_unreleased_entries]
  File "/content/beets-audible/audible.py", line 300, in get_album_info
    (book, chapters) = get_book_info(asin)
  File "/content/beets-audible/api.py", line 36, in get_book_info
    chapter_response = json.loads(make_request(f"{AUDNEX_ENDPOINT}/books/{asin}/chapters"))
  File "/content/beets-audible/api.py", line 61, in make_request
    raise e
  File "/content/beets-audible/api.py", line 53, in make_request
    with request.urlopen(req) as response:
  File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 504: Gateway Timeout

/content/Animal Farm (1 items)
No matching release found for 1 tracks.
For help, see: https://beets.readthedocs.org/en/latest/faq.html#nomatch
[S]kip, Use as-is, as Tracks, Group albums, Enter search, enter Id, aBort,
eDit?

djdembeck commented 2 years ago

Please keep an eye on this once the above feature makes it into the main branch (and thus gets deployed to public instance). Looks like the public server is already behaving better now with some other tweaks I made.

rloomans commented 2 years ago

I pulled them from the logs for the Plex Audnexus plugin. I assumed they were valid.

I’ll have another look at the logs.

On Wed, 7 Sep 2022 at 17:11, David Dembeck @.***> wrote:

The one with the error does not have data on the api:

{"statusCode":500,"error":"Internal Server Error","message":"Required key 'authors' does not exist in Audible API response for ASIN B109400077"}

are these audible.com asins?

— Reply to this email directly, view it on GitHub https://github.com/laxamentumtech/audnexus/issues/479#issuecomment-1238997257, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAWPZL4AS7C4DBS6NWXA23V5A5ZFANCNFSM6AAAAAAQGMBXPM . You are receiving this because you authored the thread.Message ID: @.***>

-- Robert Loomans @.***

lduesing commented 2 years ago

I'm running in the same 500 problem, but with a perfectly legal audible asin: DDF - und der Super-Papagei curl https://api.audnex.us/books/B0B1QKF361 {"statusCode":500,"error":"Internal Server Error","message":"Required key 'authors' does not exist in Audible API response for ASIN B0B1QKF361"}

djdembeck commented 2 years ago

I'm running in the same 500 problem, but with a perfectly legal audible asin: DDF - und der Super-Papagei curl https://api.audnex.us/books/B0B1QKF361 {"statusCode":500,"error":"Internal Server Error","message":"Required key 'authors' does not exist in Audible API response for ASIN B0B1QKF361"}

This is an audible.de ASIN which is not implemented

lduesing commented 2 years ago

Oh, sorry. Wasn't aware of a difference here.

laxamentumtech / audnexus

Some requests to https://api.audnex.us/ failing with 504 gateway time out error #479