504 on recently played api call

[kevincornish]

Just switched from running internally only to subdomain and noticed that the api call to fetch songs is 504'ing. If i reboot the containers or relog to spotify it will catch the missing songs but i think that'll only work if there are less than 50 played in that period.

Still using mongo 4.4.8 as the newer versions wont work on my system.

[info]  Refreshing songs for XXX
[error]  Retrying crashed promise, 1/10 AxiosError: Request failed with status code 504
    at settle (/app/node_modules/axios/dist/node/axios.cjs:1900:12)
    at Unzip.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:2952:11)
    at Unzip.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1358:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'ERR_BAD_RESPONSE',
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [ 'xhr', 'http' ],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: { FormData: [Function], Blob: null },
    validateStatus: [Function: validateStatus],
    headers: AxiosHeaders {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json',
      Authorization: 'Bearer XXX',
      'User-Agent': 'axios/1.3.4',
      'Accept-Encoding': 'gzip, compress, deflate, br'
    },
    baseURL: 'https://api.spotify.com/v1',
    method: 'get',
    url: '/me/player/recently-played?after=1689328005876',
    data: undefined
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      finish: [Function: requestOnFinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: true,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    maxRequestsOnConnectionReached: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: false,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    strictContentLength: false,
    _contentLength: 0,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: true,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'api.spotify.com',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 9,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'api.spotify.com',
      _closeAfterHandlingError: false,
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: null,
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [Circular *1],
      write: [Function: writeAfterFIN],
      [Symbol(res)]: null,
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 2352,
      [Symbol(kHandle)]: null,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: false,
      [Symbol(kSetKeepAlive)]: true,
      [Symbol(kSetKeepAliveInitialDelay)]: 60,
      [Symbol(kBytesRead)]: 853,
      [Symbol(kBytesWritten)]: 544,
      [Symbol(connect-options)]: [Object],
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'GET /v1/me/player/recently-played?after=1689328005876 HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Content-Type: application/json\r\n' +
      'Authorization: Bearer XXX' +
      'User-Agent: axios/1.3.4\r\n' +
      'Accept-Encoding: gzip, compress, deflate, br\r\n' +
      'Host: api.spotify.com\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 443,
      protocol: 'https:',
      options: [Object: null prototype],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype] {},
      freeSockets: [Object: null prototype] {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 0,
      maxCachedSessions: 100,
      _sessionCache: [Object],
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'GET',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/v1/me/player/recently-played?after=1689328005876',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 504,
      statusMessage: 'Gateway Timeout',
      client: [TLSSocket],
      _consuming: true,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'https://api.spotify.com/v1/me/player/recently-played?after=1689328005876',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 30,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'api.spotify.com',
    protocol: 'https:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'https://api.spotify.com/v1/me/player/recently-played?after=1689328005876',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kBytesWritten)]: 0,
    [Symbol(kEndCalled)]: true,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'content-type': [Array],
      authorization: [Array],
      'user-agent': [Array],
      'accept-encoding': [Array],
      host: [Array]
    },
    [Symbol(kUniqueHeaders)]: null
  },
  response: {
    status: 504,
    statusText: 'Gateway Timeout',
    headers: AxiosHeaders {
      'cache-control': 'private, max-age=0',
      'access-control-allow-origin': '*',
      'access-control-allow-headers': 'Accept, App-Platform, Authorization, Content-Type, Origin, Retry-After, Spotify-App-Version, X-Cloud-Trace-Context, client-token, content-access-token',
      'access-control-allow-methods': 'GET, POST, OPTIONS, PUT, DELETE, PATCH',
      'access-control-allow-credentials': 'true',
      'access-control-max-age': '604800',
      'strict-transport-security': 'max-age=31536000',
      'x-content-type-options': 'nosniff',
      date: 'Sat, 15 Jul 2023 12:15:31 GMT',
      server: 'envoy',
      via: 'HTTP/2 edgeproxy, 1.1 google',
      'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000',
      connection: 'close',
      'transfer-encoding': 'chunked'
    },
    config: {
      transitional: [Object],
      adapter: [Array],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      env: [Object],
      validateStatus: [Function: validateStatus],
      headers: [AxiosHeaders],
      baseURL: 'https://api.spotify.com/v1',
      method: 'get',
      url: '/me/player/recently-played?after=1689328005876',
      data: undefined
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'GET /v1/me/player/recently-played?after=1689328005876 HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'Authorization: Bearer XXX\r\n' +
        'User-Agent: axios/1.3.4\r\n' +
        'Accept-Encoding: gzip, compress, deflate, br\r\n' +
        'Host: api.spotify.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/v1/me/player/recently-played?after=1689328005876',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'api.spotify.com',
      protocol: 'https:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kEndCalled)]: true,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(kUniqueHeaders)]: null
    },
    data: { error: [Object] }
  }
}

my compose

version: "3"

services:
  server:
    image: yooooomi/your_spotify_server
    restart: always
    ports:
      - "8886:8080"
    links:
      - mongo
    depends_on:
      - mongo
    environment:
      API_ENDPOINT: https://mapi.xxx.xxx
      CLIENT_ENDPOINT: https://music.xxx.xxx
      SPOTIFY_PUBLIC: xxx
      SPOTIFY_SECRET: xx

  mongo:
    container_name: mongo
    image: mongo:4.4.8
    restart: always
    volumes:
      - /srv/xxx/Mongo:/data/db

  web:
    image: yooooomi/your_spotify_client
    restart: always
    ports:
      - "6447:3000"
    environment:
      API_ENDPOINT: https://mapi.xxx.xxx

[Yooooomi]

Hello! What reverse proxy are you using? It seems to be something related to your architecture.

[kevincornish]

Thanks for replying!

I did think it might be partly to-do with the reverse proxy not being able to go external, but why would it be able to fetch info if i force it to relog to spotify?

I'm using swag with the following config's for the server and client:

client

server {
    listen 443 ssl;

    server_name music.xxx.xxx;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    location / {
        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        proxy_pass http://192.168.1.82:6447;
    }
}
# Redirect HTTP requests to HTTPS
server {  
    listen 80;
    server_name music.xxx.xxx;
    return 301 https://$host$request_uri;
}

server

server {
    listen 443 ssl;

    server_name mapi.xxx.xxx;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    location / {
        include /config/nginx/proxy.conf;
        resolver 127.0.0.11 valid=30s;
        proxy_pass http://192.168.1.82:8886;
    }
}
# Redirect HTTP requests to HTTPS
server {  
    listen 80;
    server_name mapi.xxx.xxx;
    return 301 https://$host$request_uri;
}

If this doesn't look like an issue, I could try attaching the 3 containers to the same docker network as swag.

[Yooooomi]

The client_max_body_size 0 seems weird to me. But in my opinion you should go step by step on rebuilding your architecture and see where it hangs. I don't why you sometimes get info sometimes not.

[kevincornish]

Ok, just had tried putting the containers onto the same docker network with no luck. Removed the client_max_body_size too.

Will attach another log, i'll try deleting the containers + data (keeping the mongo data)

_spotifystats-server-1_logs(3).txt

[kevincornish]

_spotifystats-server-1_logs(4).txt

I have just rebuilt the containers with no change. I do have 3 users but unable to compare their stats using affinity. They were fresh signups after i moved from internal only.

[kevincornish]

Checked out a few more things this morning and also tried replicating the request, if i manually make a request to https://api.spotify.com/v1/me/player/recently-played?after=1689422404563 using the same bearer token, i get the same reply:

{
  "error": {
    "status": 504,
    "message": "Service didn't reply before timeout"
  }
}

Found a couple other people with the same issue trying to hit the same endpoint - here and here

@Yooooomi, do you have multiple users on your server?

[Yooooomi]

Hello. It seems really weird. I do have like a dozen of users on my instance and everything works flawlessly. Can you try requesting from another network like phone data or something? We got to investigate wether it's caused by your account or your router.

[kevincornish]

Hmm interesting. Ok just tested the same request with new bearer token via mobile network and got the same 504 timeout. I did notice that i didn't add the two additional users to the app on the dev portal so i've done that, rebuilt + asked them to relog to spotify but no difference there.

[Yooooomi]

At this point you can maybe try contacting the support.

[kevincornish]

I've replied to the open threads i've found on the spotify forum and referenced this issue. I also noticed that it looks like it's only the 2 users that signed up getting the 504 response, my account is syncing ok. They have nothing on their accounts since they joined yesterday and have listened to spotify since.

[Yooooomi]

I'm sorry I can't do anything for you about this. I really hope they come up with an explanation or fix for this. I will also check if my instance has any issues like this.

[kevincornish]

Hi @Yooooomi, figured it out. The 2 new users supplied me with emails that were different from the ones they had linked to on spotify. Might be worth chucking a catch in for the 504 error on the /v1/me/player/recently-played api call? Happy to commit one if needed, if you don't like that approach then we can close off this issue!

[Yooooomi]

Oh I wouldn't have thought of that. Yes we could catch something off of that. If you still have some logs it could be great as I don't know if it's easily reproducible. Glad you found out!

[kevin-kraus]

@kevincornish I don't know if you could solve the problem already, but i stumbled over the same problem today.

For me it did the trick because my Developer app was still in the "development mode". Therefore you need to add the users permitted to use your app in the spotify developer portal. Otherwise you'll get those weird 504 errors.

You can add them here:

[kevincornish]

@kevincornish I don't know if you could solve the problem already, but i stumbled over the same problem today.

For me it did the trick because my Developer app was still in the "development mode". Therefore you need to add the users permitted to use your app in the spotify developer portal. Otherwise you'll get those weird 504 errors.

You can add them here:

Haha yeah, same issue. The two users supplied with me different emails than the ones linked to their Spotify accounts. I answered it somewhere along this issue. I didn't close it out as I was going to push a PR with an updated catch but completely forgot!!