Spotify Rate Limit - Githubissues

KellyFromSales commented 5 months ago

Describe the bug

I'm trying to import my extended listening history going back to 2010. I was able to import around 65k without issue, but now it always fails

looking through my docker logs, after the 10th retry an error is displayed and it looks like i'm hitting a rate limit.

I've tried making a file only a few entries, the 12 in the example above, but it made no difference. I've tried leaving it overnight with no difference. I've also tried importing a file i know worked previously, but I'm encountering the same issue.

Looking at the docs, it looks like they work on a rolling 30 second api rate limit, so i'm confused what's going on.

Expected behavior

Running an import finishes successfully

Additional context

response: {
    status: 429,
    statusText: 'Too Many Requests',
    headers: Object [AxiosHeaders] {
      'cache-control': 'private, max-age=0',
      'retry-after': '7293',
      'access-control-allow-origin': '*',
      'access-control-allow-headers': 'Accept, App-Platform, Authorization, Content-Type, Origin, Retry-After, Spotify-App-Version, X-Cloud-Trace-Context, client-token, content-access-token',
      'access-control-allow-methods': 'GET, POST, OPTIONS, PUT, DELETE, PATCH',
      'access-control-allow-credentials': 'true',
      'access-control-max-age': '604800',
      'strict-transport-security': 'max-age=31536000',
      'x-content-type-options': 'nosniff',
      date: 'Wed, 19 Jun 2024 12:30:22 GMT',
      server: 'envoy',
      via: 'HTTP/2 edgeproxy, 1.1 google',
      'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000',
      'transfer-encoding': 'chunked'
    },
    config: {
      transitional: [Object],
      adapter: [Array],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      env: [Object],
      validateStatus: [Function: validateStatus],
      headers: [Object [AxiosHeaders]],
      baseURL: 'https://api.spotify.com/v1',
      method: 'get',
      url: 'https://api.spotify.com/v1/tracks?ids=0mmVS3xvOVcTP4blGNb9sE,6L55D19gnwlodoMRH60VJL',
      data: undefined
    },

not sure what other logs might be useful, happy to provide

KellyFromSales commented 5 months ago

seems to be working now, must have just had to wait!

samip5 commented 3 months ago

It's funny, I tried that too and got rate limited. :D

End result is that even the API calls to fetch currently playing is not working.

TaylorBurnham commented 3 months ago

I hit this last night after starting an import of 400k~ plays going back to 2014.

Spotify has their rate limit API sorta documented. In this they include a Retry-After header that ideally should be used in a backoff/retry strategy. However, I suspect we have found an undocumented hard limit for number of calls in a 24 hour period.

Here's the sequence of events I see:

Long term streaming history data was loaded in and started at 3:55 PM
249,443 of 429,129 records (69%) complete before hitting the first "crashed promise".
Finally the 429 returns with the timestamp of 5:44 PM telling me to retry after 79,892 seconds.
Restarting the service this morning at 9:37 AM it threw the same 429 at with retry after value of 22708.

Adding the the number of seconds to those two timestamps gives us approximately 3:55 PM, which is 24 hours after the import first started. Odds are I just need to retry the import after I pass that 24 hour window, which is probably why it worked for @KellyFromSales later in the day they posted this issue. Their Retry-After header only had about 2 hours left at the time of posting.

Logs


server-1  | [info]  Adding  from data
server-1  | [error]  Retrying crashed promise, 1/10, retrying in 30 seconds...
server-1  | [info]  [DbLoop] starting for 1 users
server-1  | [info]  []: refreshing...
server-1  | [info]  No missing tracks, passing...
server-1  | [info]  []: 0 tracks, 0 albums, 0 artists
server-1  | [error]  Retrying crashed promise, 2/10, retrying in 30 seconds...
server-1  | [error]  Retrying crashed promise, 3/10, retrying in 30 seconds...
server-1  | [error]  Retrying crashed promise, 4/10, retrying in 30 seconds...
server-1  | [error]  Retrying crashed promise, 5/10, retrying in 30 seconds...
server-1  | [info]  [DbLoop] starting for 1 users
server-1  | [info]  []: refreshing...
server-1  | [info]  No missing tracks, passing...
server-1  | [info]  []: 0 tracks, 0 albums, 0 artists
server-1  | [error]  Retrying crashed promise, 6/10, retrying in 30 seconds...
server-1  | [error]  Retrying crashed promise, 7/10, retrying in 30 seconds...
server-1  | [error]  Retrying crashed promise, 8/10, retrying in 30 seconds...
server-1  | [error]  Retrying crashed promise, 9/10, retrying in 30 seconds...
server-1  | [info]  [DbLoop] starting for 1 users
server-1  | [info]  []: refreshing...
server-1  | [info]  No missing tracks, passing...
server-1  | [info]  []: 0 tracks, 0 albums, 0 artists
server-1  | [error]  Retrying crashed promise, 10/10
server-1  | [error]  AxiosError: Request failed with status code 429
server-1  |     at settle (/app/node_modules/axios/dist/node/axios.cjs:1967:12)
server-1  |     at Unzip.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3066:11)
server-1  |     at Unzip.emit (node:events:530:35)
server-1  |     at endReadableNT (node:internal/streams/readable:1696:12)
server-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
server-1  |     at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:3877:41)
server-1  |     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
server-1  |   code: 'ERR_BAD_REQUEST',
server-1  |   config: {
server-1  |     transitional: {
server-1  |       silentJSONParsing: true,
server-1  |       forcedJSONParsing: true,
server-1  |       clarifyTimeoutError: false
server-1  |     },
server-1  |     adapter: [ 'xhr', 'http' ],
server-1  |     transformRequest: [ [Function: transformRequest] ],
server-1  |     transformResponse: [ [Function: transformResponse] ],
server-1  |     timeout: 0,
server-1  |     xsrfCookieName: 'XSRF-TOKEN',
server-1  |     xsrfHeaderName: 'X-XSRF-TOKEN',
server-1  |     maxContentLength: -1,
server-1  |     maxBodyLength: -1,
server-1  |     env: { FormData: [Function], Blob: [class Blob] },
server-1  |     validateStatus: [Function: validateStatus],
server-1  |     headers: Object [AxiosHeaders] {
server-1  |       Accept: 'application/json, text/plain, */*',
server-1  |       'Content-Type': 'application/json',
server-1  |       Authorization: 'Bearer ',
server-1  |       'User-Agent': 'axios/1.6.7',
server-1  |       'Accept-Encoding': 'gzip, compress, deflate, br'
server-1  |     },
server-1  |     baseURL: 'https://api.spotify.com/v1',
server-1  |     method: 'get',
server-1  |     url: 'https://api.spotify.com/v1/tracks?ids=1OJM0F5Qm4HsB4sM2V44Ql,0EIb7XkputeTjwQ8I8nwHk,6ErFraw24w4dsws22tmhQp,3HlK8txWAdtKMrbsqX40pl,0usxQA2FN1YMjDU3VLBkdJ,1b5H3zxRuFcalPIhl5di62,5cuwIuGhMvdxuGgakdV2l1,64LidMt0VfCyt0Op180BEq,5Cra5G974PUHma5AJJrtNK,25t9f3jxPiblWalMFJ8yU1,6PvrRK8gYyIZNirN9hVdh6,77QqkYm8HV0ttONtStqMZc,110rywo98Evull4beppQ7S,74wCdvPRGpzMCuwxMxr2k4,74wCdvPRGpzMCuwxMxr2k4,0Ndvi7Y4BZps8SqtZmpISB,5dVv5kCMDdBwgNIuWiV8jq,70ZmxdXo2JxT3K4MYBb4N9,5jzeuuTaGqe55AM2jCByZW,1L9Oiyyx3BjlYagjhMEez1',
server-1  |     data: undefined
server-1  |   },
server-1  |   request:  ClientRequest {
server-1  |     _events: [Object: null prototype] {
server-1  |       abort: [Function (anonymous)],
server-1  |       aborted: [Function (anonymous)],
server-1  |       connect: [Function (anonymous)],
server-1  |       error: [Function (anonymous)],
server-1  |       socket: [Function (anonymous)],
server-1  |       timeout: [Function (anonymous)],
server-1  |       finish: [Function: requestOnFinish]
server-1  |     },
server-1  |     _eventsCount: 7,
server-1  |     _maxListeners: undefined,
server-1  |     outputData: [],
server-1  |     outputSize: 0,
server-1  |     writable: true,
server-1  |     destroyed: true,
server-1  |     _last: true,
server-1  |     chunkedEncoding: false,
server-1  |     shouldKeepAlive: true,
server-1  |     maxRequestsOnConnectionReached: false,
server-1  |     _defaultKeepAlive: true,
server-1  |     useChunkedEncodingByDefault: false,
server-1  |     sendDate: false,
server-1  |     _removedConnection: false,
server-1  |     _removedContLen: false,
server-1  |     _removedTE: false,
server-1  |     strictContentLength: false,
server-1  |     _contentLength: 0,
server-1  |     _hasBody: true,
server-1  |     _trailer: '',
server-1  |     finished: true,
server-1  |     _headerSent: true,
server-1  |     _closed: true,
server-1  |     socket: TLSSocket {
server-1  |       _tlsOptions: [Object],
server-1  |       _secureEstablished: true,
server-1  |       _securePending: false,
server-1  |       _newSessionPending: false,
server-1  |       _controlReleased: true,
server-1  |       secureConnecting: false,
server-1  |       _SNICallback: null,
server-1  |       servername: 'api.spotify.com',
server-1  |       alpnProtocol: false,
server-1  |       authorized: true,
server-1  |       authorizationError: null,
server-1  |       encrypted: true,
server-1  |       _events: [Object: null prototype],
server-1  |       _eventsCount: 9,
server-1  |       connecting: false,
server-1  |       _hadError: false,
server-1  |       _parent: null,
server-1  |       _host: 'api.spotify.com',
server-1  |       _closeAfterHandlingError: false,
server-1  |       _readableState: [ReadableState],
server-1  |       _writableState: [WritableState],
server-1  |       allowHalfOpen: false,
server-1  |       _maxListeners: undefined,
server-1  |       _sockname: null,
server-1  |       _pendingData: null,
server-1  |       _pendingEncoding: '',
server-1  |       server: undefined,
server-1  |       _server: null,
server-1  |       ssl: [TLSWrap],
server-1  |       _requestCert: true,
server-1  |       _rejectUnauthorized: true,
server-1  |       timeout: 5000,
server-1  |       parser: null,
server-1  |       _httpMessage: null,
server-1  |       autoSelectFamilyAttemptedAddresses: [Array],
server-1  |       [Symbol(alpncallback)]: null,
server-1  |       [Symbol(res)]: [TLSWrap],
server-1  |       [Symbol(verified)]: true,
server-1  |       [Symbol(pendingSession)]: null,
server-1  |       [Symbol(async_id_symbol)]: -1,
server-1  |       [Symbol(kHandle)]: [TLSWrap],
server-1  |       [Symbol(lastWriteQueueSize)]: 0,
server-1  |       [Symbol(timeout)]: Timeout {
server-1  |         _idleTimeout: 5000,
server-1  |         _idlePrev: [TimersList],
server-1  |         _idleNext: [TimersList],
server-1  |         _idleStart: 6602200,
server-1  |         _onTimeout: [Function: bound ],
server-1  |         _timerArgs: undefined,
server-1  |         _repeat: null,
server-1  |         _destroyed: false,
server-1  |         [Symbol(refed)]: false,
server-1  |         [Symbol(kHasPrimitive)]: false,
server-1  |         [Symbol(asyncId)]: 6016149,
server-1  |         [Symbol(triggerId)]: 6016147
server-1  |       },
server-1  |       [Symbol(kBuffer)]: null,
server-1  |       [Symbol(kBufferCb)]: null,
server-1  |       [Symbol(kBufferGen)]: null,
server-1  |       [Symbol(shapeMode)]: true,
server-1  |       [Symbol(kCapture)]: false,
server-1  |       [Symbol(kSetNoDelay)]: false,
server-1  |       [Symbol(kSetKeepAlive)]: true,
server-1  |       [Symbol(kSetKeepAliveInitialDelay)]: 1,
server-1  |       [Symbol(kBytesRead)]: 0,
server-1  |       [Symbol(kBytesWritten)]: 0,
server-1  |       [Symbol(connect-options)]: [Object]
server-1  |     },
server-1  |     _header: 'GET /v1/tracks?ids=1OJM0F5Qm4HsB4sM2V44Ql,0EIb7XkputeTjwQ8I8nwHk,6ErFraw24w4dsws22tmhQp,3HlK8txWAdtKMrbsqX40pl,0usxQA2FN1YMjDU3VLBkdJ,1b5H3zxRuFcalPIhl5di62,5cuwIuGhMvdxuGgakdV2l1,64LidMt0VfCyt0Op180BEq,5Cra5G974PUHma5AJJrtNK,25t9f3jxPiblWalMFJ8yU1,6PvrRK8gYyIZNirN9hVdh6,77QqkYm8HV0ttONtStqMZc,110rywo98Evull4beppQ7S,74wCdvPRGpzMCuwxMxr2k4,74wCdvPRGpzMCuwxMxr2k4,0Ndvi7Y4BZps8SqtZmpISB,5dVv5kCMDdBwgNIuWiV8jq,70ZmxdXo2JxT3K4MYBb4N9,5jzeuuTaGqe55AM2jCByZW,1L9Oiyyx3BjlYagjhMEez1 HTTP/1.1\r\n' +
server-1  |       'Accept: application/json, text/plain, */*\r\n' +
server-1  |       'Content-Type: application/json\r\n' +
server-1  |       'Authorization: Bearer ' +
server-1  |       'User-Agent: axios/1.6.7\r\n' +
server-1  |       'Accept-Encoding: gzip, compress, deflate, br\r\n' +
server-1  |       'Host: api.spotify.com\r\n' +
server-1  |       'Connection: keep-alive\r\n' +
server-1  |       '\r\n',
server-1  |     _keepAliveTimeout: 0,
server-1  |     _onPendingData: [Function: nop],
server-1  |     agent: Agent {
server-1  |       _events: [Object: null prototype],
server-1  |       _eventsCount: 2,
server-1  |       _maxListeners: undefined,
server-1  |       defaultPort: 443,
server-1  |       protocol: 'https:',
server-1  |       options: [Object: null prototype],
server-1  |       requests: [Object: null prototype] {},
server-1  |       sockets: [Object: null prototype] {},
server-1  |       freeSockets: [Object: null prototype],
server-1  |       keepAliveMsecs: 1000,
server-1  |       keepAlive: true,
server-1  |       maxSockets: Infinity,
server-1  |       maxFreeSockets: 256,
server-1  |       scheduling: 'lifo',
server-1  |       maxTotalSockets: Infinity,
server-1  |       totalSocketCount: 1,
server-1  |       maxCachedSessions: 100,
server-1  |       _sessionCache: [Object],
server-1  |       [Symbol(shapeMode)]: false,
server-1  |       [Symbol(kCapture)]: false
server-1  |     },
server-1  |     socketPath: undefined,
server-1  |     method: 'GET',
server-1  |     maxHeaderSize: undefined,
server-1  |     insecureHTTPParser: undefined,
server-1  |     joinDuplicateHeaders: undefined,
server-1  |     path: '/v1/tracks?ids=1OJM0F5Qm4HsB4sM2V44Ql,0EIb7XkputeTjwQ8I8nwHk,6ErFraw24w4dsws22tmhQp,3HlK8txWAdtKMrbsqX40pl,0usxQA2FN1YMjDU3VLBkdJ,1b5H3zxRuFcalPIhl5di62,5cuwIuGhMvdxuGgakdV2l1,64LidMt0VfCyt0Op180BEq,5Cra5G974PUHma5AJJrtNK,25t9f3jxPiblWalMFJ8yU1,6PvrRK8gYyIZNirN9hVdh6,77QqkYm8HV0ttONtStqMZc,110rywo98Evull4beppQ7S,74wCdvPRGpzMCuwxMxr2k4,74wCdvPRGpzMCuwxMxr2k4,0Ndvi7Y4BZps8SqtZmpISB,5dVv5kCMDdBwgNIuWiV8jq,70ZmxdXo2JxT3K4MYBb4N9,5jzeuuTaGqe55AM2jCByZW,1L9Oiyyx3BjlYagjhMEez1',
server-1  |     _ended: true,
server-1  |     res: IncomingMessage {
server-1  |       _events: [Object],
server-1  |       _readableState: [ReadableState],
server-1  |       _maxListeners: undefined,
server-1  |       socket: null,
server-1  |       httpVersionMajor: 1,
server-1  |       httpVersionMinor: 1,
server-1  |       httpVersion: '1.1',
server-1  |       complete: true,
server-1  |       rawHeaders: [Array],
server-1  |       rawTrailers: [],
server-1  |       joinDuplicateHeaders: undefined,
server-1  |       aborted: false,
server-1  |       upgrade: false,
server-1  |       url: '',
server-1  |       method: null,
server-1  |       statusCode: 429,
server-1  |       statusMessage: 'Too Many Requests',
server-1  |       client: [TLSSocket],
server-1  |       _consuming: true,
server-1  |       _dumped: false,
server-1  |       req: [Circular *1],
server-1  |       _eventsCount: 4,
server-1  |       responseUrl: 'https://api.spotify.com/v1/tracks?ids=1OJM0F5Qm4HsB4sM2V44Ql,0EIb7XkputeTjwQ8I8nwHk,6ErFraw24w4dsws22tmhQp,3HlK8txWAdtKMrbsqX40pl,0usxQA2FN1YMjDU3VLBkdJ,1b5H3zxRuFcalPIhl5di62,5cuwIuGhMvdxuGgakdV2l1,64LidMt0VfCyt0Op180BEq,5Cra5G974PUHma5AJJrtNK,25t9f3jxPiblWalMFJ8yU1,6PvrRK8gYyIZNirN9hVdh6,77QqkYm8HV0ttONtStqMZc,110rywo98Evull4beppQ7S,74wCdvPRGpzMCuwxMxr2k4,74wCdvPRGpzMCuwxMxr2k4,0Ndvi7Y4BZps8SqtZmpISB,5dVv5kCMDdBwgNIuWiV8jq,70ZmxdXo2JxT3K4MYBb4N9,5jzeuuTaGqe55AM2jCByZW,1L9Oiyyx3BjlYagjhMEez1',
server-1  |       redirects: [],
server-1  |       [Symbol(shapeMode)]: true,
server-1  |       [Symbol(kCapture)]: false,
server-1  |       [Symbol(kHeaders)]: [Object],
server-1  |       [Symbol(kHeadersCount)]: 30,
server-1  |       [Symbol(kTrailers)]: null,
server-1  |       [Symbol(kTrailersCount)]: 0
server-1  |     },
server-1  |     aborted: false,
server-1  |     timeoutCb: null,
server-1  |     upgradeOrConnect: false,
server-1  |     parser: null,
server-1  |     maxHeadersCount: null,
server-1  |     reusedSocket: true,
server-1  |     host: 'api.spotify.com',
server-1  |     protocol: 'https:',
server-1  |     _redirectable: Writable {
server-1  |       _events: [Object],
server-1  |       _writableState: [WritableState],
server-1  |       _maxListeners: undefined,
server-1  |       _options: [Object],
server-1  |       _ended: true,
server-1  |       _ending: true,
server-1  |       _redirectCount: 0,
server-1  |       _redirects: [],
server-1  |       _requestBodyLength: 0,
server-1  |       _requestBodyBuffers: [],
server-1  |       _eventsCount: 3,
server-1  |       _onNativeResponse: [Function (anonymous)],
server-1  |       _currentRequest: [Circular *1],
server-1  |       _currentUrl: 'https://api.spotify.com/v1/tracks?ids=1OJM0F5Qm4HsB4sM2V44Ql,0EIb7XkputeTjwQ8I8nwHk,6ErFraw24w4dsws22tmhQp,3HlK8txWAdtKMrbsqX40pl,0usxQA2FN1YMjDU3VLBkdJ,1b5H3zxRuFcalPIhl5di62,5cuwIuGhMvdxuGgakdV2l1,64LidMt0VfCyt0Op180BEq,5Cra5G974PUHma5AJJrtNK,25t9f3jxPiblWalMFJ8yU1,6PvrRK8gYyIZNirN9hVdh6,77QqkYm8HV0ttONtStqMZc,110rywo98Evull4beppQ7S,74wCdvPRGpzMCuwxMxr2k4,74wCdvPRGpzMCuwxMxr2k4,0Ndvi7Y4BZps8SqtZmpISB,5dVv5kCMDdBwgNIuWiV8jq,70ZmxdXo2JxT3K4MYBb4N9,5jzeuuTaGqe55AM2jCByZW,1L9Oiyyx3BjlYagjhMEez1',
server-1  |       [Symbol(shapeMode)]: true,
server-1  |       [Symbol(kCapture)]: false
server-1  |     },
server-1  |     [Symbol(shapeMode)]: false,
server-1  |     [Symbol(kCapture)]: false,
server-1  |     [Symbol(kBytesWritten)]: 0,
server-1  |     [Symbol(kNeedDrain)]: false,
server-1  |     [Symbol(corked)]: 0,
server-1  |     [Symbol(kOutHeaders)]: [Object: null prototype] {
server-1  |       accept: [Array],
server-1  |       'content-type': [Array],
server-1  |       authorization: [Array],
server-1  |       'user-agent': [Array],
server-1  |       'accept-encoding': [Array],
server-1  |       host: [Array]
server-1  |     },
server-1  |     [Symbol(errored)]: null,
server-1  |     [Symbol(kHighWaterMark)]: 16384,
server-1  |     [Symbol(kRejectNonStandardBodyWrites)]: false,
server-1  |     [Symbol(kUniqueHeaders)]: null
server-1  |   },
server-1  |   response: {
server-1  |     status: 429,
server-1  |     statusText: 'Too Many Requests',
server-1  |     headers: Object [AxiosHeaders] {
server-1  |       'cache-control': 'private, max-age=0',
server-1  |       'retry-after': '79892',
server-1  |       'access-control-allow-origin': '*',
server-1  |       'access-control-allow-headers': 'Accept, App-Platform, Authorization, Content-Type, Origin, Retry-After, Spotify-App-Version, X-Cloud-Trace-Context, client-token, content-access-token',
server-1  |       'access-control-allow-methods': 'GET, POST, OPTIONS, PUT, DELETE, PATCH',
server-1  |       'access-control-allow-credentials': 'true',
server-1  |       'access-control-max-age': '604800',
server-1  |       'strict-transport-security': 'max-age=31536000',
server-1  |       'x-content-type-options': 'nosniff',
server-1  |       date: 'Fri, 23 Aug 2024 21:44:07 GMT',
server-1  |       server: 'envoy',
server-1  |       via: 'HTTP/2 edgeproxy, 1.1 google',
server-1  |       'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000',
server-1  |       'transfer-encoding': 'chunked'
server-1  |     },
server-1  |     config: {
server-1  |       transitional: [Object],
server-1  |       adapter: [Array],
server-1  |       transformRequest: [Array],
server-1  |       transformResponse: [Array],
server-1  |       timeout: 0,
server-1  |       xsrfCookieName: 'XSRF-TOKEN',
server-1  |       xsrfHeaderName: 'X-XSRF-TOKEN',
server-1  |       maxContentLength: -1,
server-1  |       maxBodyLength: -1,
server-1  |       env: [Object],
server-1  |       validateStatus: [Function: validateStatus],
server-1  |       headers: [Object [AxiosHeaders]],
server-1  |       baseURL: 'https://api.spotify.com/v1',
server-1  |       method: 'get',
server-1  |       url: 'https://api.spotify.com/v1/tracks?ids=1OJM0F5Qm4HsB4sM2V44Ql,0EIb7XkputeTjwQ8I8nwHk,6ErFraw24w4dsws22tmhQp,3HlK8txWAdtKMrbsqX40pl,0usxQA2FN1YMjDU3VLBkdJ,1b5H3zxRuFcalPIhl5di62,5cuwIuGhMvdxuGgakdV2l1,64LidMt0VfCyt0Op180BEq,5Cra5G974PUHma5AJJrtNK,25t9f3jxPiblWalMFJ8yU1,6PvrRK8gYyIZNirN9hVdh6,77QqkYm8HV0ttONtStqMZc,110rywo98Evull4beppQ7S,74wCdvPRGpzMCuwxMxr2k4,74wCdvPRGpzMCuwxMxr2k4,0Ndvi7Y4BZps8SqtZmpISB,5dVv5kCMDdBwgNIuWiV8jq,70ZmxdXo2JxT3K4MYBb4N9,5jzeuuTaGqe55AM2jCByZW,1L9Oiyyx3BjlYagjhMEez1',
server-1  |       data: undefined
server-1  |     },
server-1  |     request:  ClientRequest {
server-1  |       _events: [Object: null prototype],
server-1  |       _eventsCount: 7,
server-1  |       _maxListeners: undefined,
server-1  |       outputData: [],
server-1  |       outputSize: 0,
server-1  |       writable: true,
server-1  |       destroyed: true,
server-1  |       _last: true,
server-1  |       chunkedEncoding: false,
server-1  |       shouldKeepAlive: true,
server-1  |       maxRequestsOnConnectionReached: false,
server-1  |       _defaultKeepAlive: true,
server-1  |       useChunkedEncodingByDefault: false,
server-1  |       sendDate: false,
server-1  |       _removedConnection: false,
server-1  |       _removedContLen: false,
server-1  |       _removedTE: false,
server-1  |       strictContentLength: false,
server-1  |       _contentLength: 0,
server-1  |       _hasBody: true,
server-1  |       _trailer: '',
server-1  |       finished: true,
server-1  |       _headerSent: true,
server-1  |       _closed: true,
server-1  |       socket: [TLSSocket],
server-1  |       _header: 'GET /v1/tracks?ids=1OJM0F5Qm4HsB4sM2V44Ql,0EIb7XkputeTjwQ8I8nwHk,6ErFraw24w4dsws22tmhQp,3HlK8txWAdtKMrbsqX40pl,0usxQA2FN1YMjDU3VLBkdJ,1b5H3zxRuFcalPIhl5di62,5cuwIuGhMvdxuGgakdV2l1,64LidMt0VfCyt0Op180BEq,5Cra5G974PUHma5AJJrtNK,25t9f3jxPiblWalMFJ8yU1,6PvrRK8gYyIZNirN9hVdh6,77QqkYm8HV0ttONtStqMZc,110rywo98Evull4beppQ7S,74wCdvPRGpzMCuwxMxr2k4,74wCdvPRGpzMCuwxMxr2k4,0Ndvi7Y4BZps8SqtZmpISB,5dVv5kCMDdBwgNIuWiV8jq,70ZmxdXo2JxT3K4MYBb4N9,5jzeuuTaGqe55AM2jCByZW,1L9Oiyyx3BjlYagjhMEez1 HTTP/1.1\r\n' +
server-1  |         'Accept: application/json, text/plain, */*\r\n' +
server-1  |         'Content-Type: application/json\r\n' +
server-1  |         'Authorization: Bearer ' +
server-1  |         'User-Agent: axios/1.6.7\r\n' +
server-1  |         'Accept-Encoding: gzip, compress, deflate, br\r\n' +
server-1  |         'Host: api.spotify.com\r\n' +
server-1  |         'Connection: keep-alive\r\n' +
server-1  |         '\r\n',
server-1  |       _keepAliveTimeout: 0,
server-1  |       _onPendingData: [Function: nop],
server-1  |       agent: [Agent],
server-1  |       socketPath: undefined,
server-1  |       method: 'GET',
server-1  |       maxHeaderSize: undefined,
server-1  |       insecureHTTPParser: undefined,
server-1  |       joinDuplicateHeaders: undefined,
server-1  |       path: '/v1/tracks?ids=1OJM0F5Qm4HsB4sM2V44Ql,0EIb7XkputeTjwQ8I8nwHk,6ErFraw24w4dsws22tmhQp,3HlK8txWAdtKMrbsqX40pl,0usxQA2FN1YMjDU3VLBkdJ,1b5H3zxRuFcalPIhl5di62,5cuwIuGhMvdxuGgakdV2l1,64LidMt0VfCyt0Op180BEq,5Cra5G974PUHma5AJJrtNK,25t9f3jxPiblWalMFJ8yU1,6PvrRK8gYyIZNirN9hVdh6,77QqkYm8HV0ttONtStqMZc,110rywo98Evull4beppQ7S,74wCdvPRGpzMCuwxMxr2k4,74wCdvPRGpzMCuwxMxr2k4,0Ndvi7Y4BZps8SqtZmpISB,5dVv5kCMDdBwgNIuWiV8jq,70ZmxdXo2JxT3K4MYBb4N9,5jzeuuTaGqe55AM2jCByZW,1L9Oiyyx3BjlYagjhMEez1',
server-1  |       _ended: true,
server-1  |       res: [IncomingMessage],
server-1  |       aborted: false,
server-1  |       timeoutCb: null,
server-1  |       upgradeOrConnect: false,
server-1  |       parser: null,
server-1  |       maxHeadersCount: null,
server-1  |       reusedSocket: true,
server-1  |       host: 'api.spotify.com',
server-1  |       protocol: 'https:',
server-1  |       _redirectable: [Writable],
server-1  |       [Symbol(shapeMode)]: false,
server-1  |       [Symbol(kCapture)]: false,
server-1  |       [Symbol(kBytesWritten)]: 0,
server-1  |       [Symbol(kNeedDrain)]: false,
server-1  |       [Symbol(corked)]: 0,
server-1  |       [Symbol(kOutHeaders)]: [Object: null prototype],
server-1  |       [Symbol(errored)]: null,
server-1  |       [Symbol(kHighWaterMark)]: 16384,
server-1  |       [Symbol(kRejectNonStandardBodyWrites)]: false,
server-1  |       [Symbol(kUniqueHeaders)]: null
server-1  |     },
server-1  |     data: 'Too many requests'
server-1  |   }
server-1  | }
server-1  | [error]  This import failed, but metadata is kept so that you can retry it later in the settings
server-1  | [info]  [DbLoop] starting for 1 users
server-1  | [info]  []: refreshing...
server-1  | [info]  No missing tracks, passing...
server-1  | [info]  []: 0 tracks, 0 albums, 0 artists
server-1  | [info]  [DbLoop] starting for 1 users
server-1  | [info]  []: refreshing...
server-1  | [error]  Retrying crashed promise, 1/10, retrying in 30 seconds...
server-1  | [info]  No missing tracks, passing...
server-1  | [info]  []: 0 tracks, 0 albums, 0 artists
server-1  | [info]  [DbLoop] starting for 1 users
server-1  | [info]  []: refreshing...
server-1  | [info]  Refreshed token for 
server-1  | [info]  No missing tracks, passing...
server-1  | [info]  []: 0 tracks, 0 albums, 0 artists
server-1  | [info]  [DbLoop] starting for 1 users

Yooooomi commented 3 months ago

Hello, I think I found the issue, it is related to how I queue the calls, if the queue is empty I call instantly. Thus when you import your songs it does the calls one by one so the queue is always empty, so the calls are never delayed how they should be.

Yooooomi commented 3 months ago

Started looking on what could be the issue. Turns out my implementation of the queue is not guilty of what's happening. I feel like Spotify changed their rate limits. I cannot see any numbers on their documentation that I have to respect so I'm a bit blind on how to fix. Also, it seems that they send no headers concerning the current state of the rate limites. They only send the retry-after header when it's already too late. I'm looking deeper to see if there is any written rule I can respect. EDIT: Also it would be insane to wait for 429's then having to wait 24 hours before doing the next request 💀

samip5 commented 3 months ago

The wait 24h might be a okay ish workaround for the moment though..

EDIT: Is it doing the tracks etc, one by one when importing? I think instead it would be beneficial to do bulk requests if possible? Eg include multiple tracks in one request? EDIT2: Related: https://community.spotify.com/t5/Spotify-for-Developers/Constantly-getting-24-hour-Rate-Limited-during-development/td-p/5814282

Yooooomi commented 3 months ago

The 24 hours wait would stop the whole app from functioning for other users. I request all the infos I need every 20 track processed. Maybe I could increase this number but I don't think this would change radically. Maybe the cleanest way would be to import only 150k entries a day.

TaylorBurnham commented 3 months ago

Yeah, if you enforce the 24 hour wait it will break ongoing listens from being tracked, and then you'll have gaps in your history.

I think you can work around that 24 hour wait with a rolling counter. Increment every call, and decrement +24hr from the time of that call. You can have a high water mark for when the imports halt, but keep the polling task for active listening still running.

It's not glamorous, but when dealing with undocumented boundaries/limits: you gotta do what you gotta do.

samip5 commented 3 months ago

I meant to enforce it for the import only. :)

samip5 commented 3 months ago

So now that I tried to clean it up, my instance is in somewhat corrupt state (it auto-updated so the import file(s) stored in /tmp no longer exists), POST /import/full-privacy 400 132.870 ms - 28 and says An import is already running on this account. How do I recover it?

Yooooomi commented 3 months ago

I think it cleans corrupted imports at start so I guess you could try restarting the container if you did not already try that. Otherwise I ça have a look at that.

samip5 commented 3 months ago

I think it cleans corrupted imports at start so I guess you could try restarting the container if you did not already try that. Otherwise I ça have a look at that.

The auto-update had the implication that it did already restart (automatically), which is why it failed but it's still somehow in a state where the import is stuck, and I cannot start a new one.

samip5 commented 3 months ago

It seems that it required the restart of both mongo and server, instead of just server for some reason..

samip5 commented 3 months ago

Oooof

Yooooomi / your_spotify

Spotify Rate Limit #411