Gotham token issues

[adamef93]

I'm so sorry for making another post, but something appears to be up with how Gotham is handling tokens.

I've gotten into the habit recently of making sure the MSG+/now Gotham integration works on game days and have often needed to delete the config volume and re-authenticate fresh in order for things to work. It was working fine last night, but I needed to delete the config volume and re-authenticate fresh this morning. Toggling the integration in the web UI and re-authenticating didn't work. The issue seems to be with the TVE account I'm using and not with the Gotham login specifically because events are still listed in this state, they just won't play. It also looks like toggling the integration/using the re-authenticate button only affects the Gotham login and doesn't clear any associated TVE credentials.

I probably should've let you know this was a problem sooner since it was also happening under MSG, my bad. The fix was quick enough that I just dealt with it. Things are working for the game tonight and I can keep re-authenticating the way I have been for future games, so there's no rush to take a look at this.

Thanks in advance!

AxiosError: Request failed with status code 403
    at settle (/app/node_modules/axios/lib/core/settle.js:19:12)
    at Unzip.handleStreamEnd (/app/node_modules/axios/lib/adapters/http.js:512:11)
    at Unzip.emit (node:events:529:35)
    at Unzip.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ERR_BAD_REQUEST',
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [ 'xhr', 'http' ],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: { FormData: [Function], Blob: [class Blob] },
    validateStatus: [Function: validateStatus],
    headers: AxiosHeaders {
      Accept: 'application/json, text/plain, */*',
      Authorization: 'Bearer eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiI1ODZiMWFlYS1hZDlmLTRkZmQtODZiMC0xYmZhMzRmZmFjMjgiLCJuYmYiOjE3MzA0NzU4MjUsImlzcyI6ImF1dGguYWRvYmUuY29tIiwic2NvcGVzIjoiY29uZmlnOmNsaWVudCxkZWNpc2lvbnM6Y2xpZW50LHBsYXRmb3JtOnNzbyxwcm9maWxlOmNsaWVudCx0b2tlbnM6Y2xpZW50IiwiZXhwIjoxNzMwNDk3NDI1LCJpYXQiOjE3MzA0NzU4MjV9.SKaL9Ho4FbH7TFw4pS3ofkpUprgO7LE7B4f-OTHv-HleqYUKgETZWSqkLLidg6-nt1vOJwFwuMHAB_9Slp7ZAV8f2KMS4y2sxr8GtXH9ZJDRl6U7rVPIPvw9hPudJ9YLlw8V4lHon92T9PuKg6sBfdjl3uUrH9_hifdpnd_biB2E1Gl6DMp4J92ppi2PTMr_CYirT-ciTOdP8MXWlJMQtTrQhFzpxeTF2P2zHj2rmzgcq82IIud577ETTLg0PKSe5acGeZUCsRUJKMZd4aSsS32wY6-7aE0JeKwH0cHOOYOv8T9npiZP9VuIKOyHE-BMYb9lp2vciGyqD-_lFzPK2A',
      'user-agent': 'okhttp/4.11.0',
      'Accept-Encoding': 'gzip, compress, deflate, br'
    },
    method: 'get',
    url: 'https://api.auth.adobe.com/api/v1/authorize?deviceId=f15a5c38-1571-4e42-b840-ae4eb6e6431f&requestor=Gotham&resource=MSGGO',
    data: undefined
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      finish: [Function: requestOnFinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: true,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    maxRequestsOnConnectionReached: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: false,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    strictContentLength: false,
    _contentLength: 0,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: true,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'api.auth.adobe.com',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 9,
      connecting: false,
Could not ping Adobe
      _hadError: false,
      _parent: null,
      _host: 'api.auth.adobe.com',
      _closeAfterHandlingError: false,
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: null,
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [Circular *1],
      write: [Function: writeAfterFIN],
      [Symbol(alpncallback)]: null,
      [Symbol(res)]: null,
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 2327,
      [Symbol(kHandle)]: null,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: false,
      [Symbol(kSetKeepAlive)]: true,
      [Symbol(kSetKeepAliveInitialDelay)]: 60,
      [Symbol(kBytesRead)]: 690,
      [Symbol(kBytesWritten)]: 933,
      [Symbol(connect-options)]: [Object]
    },
    _header: 'GET /api/v1/authorize?deviceId=f15a5c38-1571-4e42-b840-ae4eb6e6431f&requestor=Gotham&resource=MSGGO HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Authorization: Bearer eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiI1ODZiMWFlYS1hZDlmLTRkZmQtODZiMC0xYmZhMzRmZmFjMjgiLCJuYmYiOjE3MzA0NzU4MjUsImlzcyI6ImF1dGguYWRvYmUuY29tIiwic2NvcGVzIjoiY29uZmlnOmNsaWVudCxkZWNpc2lvbnM6Y2xpZW50LHBsYXRmb3JtOnNzbyxwcm9maWxlOmNsaWVudCx0b2tlbnM6Y2xpZW50IiwiZXhwIjoxNzMwNDk3NDI1LCJpYXQiOjE3MzA0NzU4MjV9.SKaL9Ho4FbH7TFw4pS3ofkpUprgO7LE7B4f-OTHv-HleqYUKgETZWSqkLLidg6-nt1vOJwFwuMHAB_9Slp7ZAV8f2KMS4y2sxr8GtXH9ZJDRl6U7rVPIPvw9hPudJ9YLlw8V4lHon92T9PuKg6sBfdjl3uUrH9_hifdpnd_biB2E1Gl6DMp4J92ppi2PTMr_CYirT-ciTOdP8MXWlJMQtTrQhFzpxeTF2P2zHj2rmzgcq82IIud577ETTLg0PKSe5acGeZUCsRUJKMZd4aSsS32wY6-7aE0JeKwH0cHOOYOv8T9npiZP9VuIKOyHE-BMYb9lp2vciGyqD-_lFzPK2A\r\n' +
      'user-agent: okhttp/4.11.0\r\n' +
      'Accept-Encoding: gzip, compress, deflate, br\r\n' +
      'Host: api.auth.adobe.com\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 443,
      protocol: 'https:',
      options: [Object: null prototype],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype] {},
      freeSockets: [Object: null prototype] {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 0,
      maxCachedSessions: 100,
      _sessionCache: [Object],
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'GET',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    joinDuplicateHeaders: undefined,
    path: '/api/v1/authorize?deviceId=f15a5c38-1571-4e42-b840-ae4eb6e6431f&requestor=Gotham&resource=MSGGO',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      joinDuplicateHeaders: undefined,
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 403,
      statusMessage: '',
      client: [TLSSocket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'https://api.auth.adobe.com/api/v1/authorize?deviceId=f15a5c38-1571-4e42-b840-ae4eb6e6431f&requestor=Gotham&resource=MSGGO',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 26,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'api.auth.adobe.com',
    protocol: 'https:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'https://api.auth.adobe.com/api/v1/authorize?deviceId=f15a5c38-1571-4e42-b840-ae4eb6e6431f&requestor=Gotham&resource=MSGGO',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kBytesWritten)]: 0,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      authorization: [Array],
      'user-agent': [Array],
      'accept-encoding': [Array],
      host: [Array]
    },
    [Symbol(errored)]: null,
    [Symbol(kHighWaterMark)]: 16384,
    [Symbol(kRejectNonStandardBodyWrites)]: false,
    [Symbol(kUniqueHeaders)]: null
  },
  response: {
    status: 403,
    statusText: '',
    headers: AxiosHeaders {
      date: 'Fri, 01 Nov 2024 15:44:17 GMT',
      'content-type': 'application/json',
      'transfer-encoding': 'chunked',
      connection: 'close',
      'access-control-expose-headers': 'pass_sfp',
      'cache-control': 'No-Cache',
      'access-control-allow-methods': 'POST,GET,OPTIONS,DELETE',
      'access-control-allow-credentials': 'true',
      p3p: 'CP="NOI DSP COR CURa ADMa DEVa OUR BUS IND UNI COM NAV STA"',
      'access-control-allow-headers': 'ap_11,ap_42,ap_z,ap_19,ap_23,authorization,pass_sfp,AP-Session-Identifier,AP-Device-Identifier,AP-SDK-Identifier,X-Device-Info',
      server: 'api-gateway/1.9.3.1',
      'strict-transport-security': 'max-age=60'
    },
    config: {
      transitional: [Object],
      adapter: [Array],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      env: [Object],
      validateStatus: [Function: validateStatus],
      headers: [AxiosHeaders],
      method: 'get',
      url: 'https://api.auth.adobe.com/api/v1/authorize?deviceId=f15a5c38-1571-4e42-b840-ae4eb6e6431f&requestor=Gotham&resource=MSGGO',
      data: undefined
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: true,
      socket: [TLSSocket],
      _header: 'GET /api/v1/authorize?deviceId=f15a5c38-1571-4e42-b840-ae4eb6e6431f&requestor=Gotham&resource=MSGGO HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Authorization: Bearer eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiI1ODZiMWFlYS1hZDlmLTRkZmQtODZiMC0xYmZhMzRmZmFjMjgiLCJuYmYiOjE3MzA0NzU4MjUsImlzcyI6ImF1dGguYWRvYmUuY29tIiwic2NvcGVzIjoiY29uZmlnOmNsaWVudCxkZWNpc2lvbnM6Y2xpZW50LHBsYXRmb3JtOnNzbyxwcm9maWxlOmNsaWVudCx0b2tlbnM6Y2xpZW50IiwiZXhwIjoxNzMwNDk3NDI1LCJpYXQiOjE3MzA0NzU4MjV9.SKaL9Ho4FbH7TFw4pS3ofkpUprgO7LE7B4f-OTHv-HleqYUKgETZWSqkLLidg6-nt1vOJwFwuMHAB_9Slp7ZAV8f2KMS4y2sxr8GtXH9ZJDRl6U7rVPIPvw9hPudJ9YLlw8V4lHon92T9PuKg6sBfdjl3uUrH9_hifdpnd_biB2E1Gl6DMp4J92ppi2PTMr_CYirT-ciTOdP8MXWlJMQtTrQhFzpxeTF2P2zHj2rmzgcq82IIud577ETTLg0PKSe5acGeZUCsRUJKMZd4aSsS32wY6-7aE0JeKwH0cHOOYOv8T9npiZP9VuIKOyHE-BMYb9lp2vciGyqD-_lFzPK2A\r\n' +
        'user-agent: okhttp/4.11.0\r\n' +
        'Accept-Encoding: gzip, compress, deflate, br\r\n' +
        'Host: api.auth.adobe.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/api/v1/authorize?deviceId=f15a5c38-1571-4e42-b840-ae4eb6e6431f&requestor=Gotham&resource=MSGGO',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'api.auth.adobe.com',
      protocol: 'https:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    data: { status: 403, message: 'User not Authenticated' }
  }
}

[adamef93]

Just wanted to add that I noticed a new error message while checking the container logs

TypeError: Cannot read properties of undefined (reading 'zip')
    at GothamHandler.<anonymous> (/app/services/gotham-handler.ts:957:41)
    at Generator.next (<anonymous>)
    at /app/services/gotham-handler.ts:8:71
Could not add TVE subscription for Gotham!
    at new Promise (<anonymous>)
    at __awaiter (/app/services/gotham-handler.ts:4:12)
    at GothamHandler.addTVESubscription (/app/services/gotham-handler.ts:946:108)
    at GothamHandler.<anonymous> (/app/services/gotham-handler.ts:869:18)
    at Generator.next (<anonymous>)
    at fulfilled (/app/services/gotham-handler.ts:5:58)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

[SoSJames]

I'm not sure this is related, but I'm unable to even get the popup from the GUI to add the TVE provider. In the browser, the toggle generates a 500 error, and in the EPlusTV container log, I get:

Error [InvalidTokenError]

    at Object.<anonymous> (/app/node_modules/jwt-decode/lib/index.js:9:31)

    at Module._compile (node:internal/modules/cjs/loader:1364:14)

    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1422:10)

    at Module.load (node:internal/modules/cjs/loader:1203:32)

    at Function.Module._load (node:internal/modules/cjs/loader:1019:12)

    at Module.require (node:internal/modules/cjs/loader:1231:19)

    at require (node:internal/modules/helpers:177:18)

    at Object.<anonymous> (/app/services/espn-handler.ts:7:1)

    at Module._compile (node:internal/modules/cjs/loader:1364:14)

    at Module.m._compile (/app/node_modules/ts-node/src/index.ts:1310:23) {

  message: 'Invalid token specified'

}

Could not start the authentication process for Gotham!

TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be of type string or an instance of Buffer, ArrayBuffer, or Array or an Array-like Object. Received undefined

    at new NodeError (node:internal/errors:405:5)

    at Function.from (node:buffer:325:9)

    at /app/services/providers/gotham/views/TveLogin.tsx:15:25

    at Generator.next (<anonymous>)

    at fulfilled (/app/services/providers/gotham/views/TveLogin.tsx:5:58)

    at processTicksAndRejections (node:internal/process/task_queues:95:5) {

  code: 'ERR_INVALID_ARG_TYPE'

}

[adamef93]

Try deleting the container and associated data volume if you haven't already. I haven't seen this particular error, but clearing everything and starting from scratch has fixed any auth/refresh problem I've had

I just tried it after seeing your comment to confirm if Gotham changed how authentication was being done and broke the app, but it still worked for me

[SoSJames]

I've deleted everything more than once, and even changed permissions. It looks like the app is failing to serve the auth asset to create the popup. I've tried everything else I can think of, unfortunately. Unlike in your case, mine never authenticated to TVE in the first place. The good news is the new Gotham service seems to have eliminated the stream breaks and audio sync issues going to and from commercial.

[adamef93]

What you're describing sounds similar to how adblockers work. Do you have a pihole or adguard server or something filtering outbound communication? There's a lot of auth services being used that can easily get blocked. I speak from experience and had to whitelist my docker host from using my pihole when I first deployed to my network

[SoSJames]

I have both. However, I took pihole out of the loop for my docker host and the PC I'm using to access the GUI. Also disabled local adblockers and even tried it in multiple browsers, including a few with no adblocker plugin even installed. As far as I'm aware, the 500 error is an internal server error inside the EplusTV container. It doesn't give any further detail, though, so I don't know where to look next. Plus, the EplusTV log is throwing an invalid token error, which I assume means it's not constructing the TVE request correctly.

[adamef93]

Do you have any forced redirection to the pihole/adguard on your router? I used a guide like this to make sure all devices would be forced to the pihole even if they have baked in DNS configs (like smart TVs having 8.8.8.8 hardcoded in)

https://coygeek.com/docs/pihole-pfsense-redirect-netgate/

Something is definitely blocking outbound to Gotham or whatever auth service it's using. It isn't anything with the browser since the server is the one making the outbound request

[SoSJames]

Nope, no forced redirection and no indication in firewall logs of any rewrites.

[adamef93]

Just also wanted to mention that I noticed simply deleting and recreating the container without deleting the config volume and fully re-authenticating might fix the token issues I'm experiencing.