libremdb doesn't work anymore

[mariopaolo]

since yesterday, libremdb stopped working on my instance. I thought this was due to some problem on my part, but after some thorough troubleshooting, I realized this isn't related to my specific install. so I tested every libremdb public instance I could find (including those in the README.md file) and, to my amazement, no one worked.

not sure what is happening, but anytime I use the title/ path I get

something went wrong (500)

you can easily reproduce it with the links below (and probably with your own instance as well) https://libremdb.herokuapp.com/title/tt23556804 https://libremdb.pussthecat.org/title/tt23556804 https://libremdbeu.herokuapp.com/title/tt23556804 https://lmdb.tokhmi.xyz/title/tt23556804 https://libremdb.esmailelbob.xyz/title/tt23556804 https://libremdb.pussthecat.org/title/tt23556804

P.S.: I looked for logs inside the pod, but couldn't find any.

[zyachel]

my instance is running fine, so I suspect it might have something to do with docker. I'll have to look into it.

As for the logs, there is zero logging, both in development and production.

[mohammadrafigh]

Same as here with docker

[zyachel]

Can you check if the error is still present?

[mohammadrafigh]

Yes, fetched changes, built docker image again and still 500

[zyachel]

can you add a little console.log in fetchers/title.ts to see the cause maybe? I'm not able to use docker at the moment. Else I'll look into this on weekend.

[mohammadrafigh]

Sorry for delay, the docker image doesn't rebuild the project after a restart so it takes some time to rebuild the whole image. We need to fix that too. For now here are the logs:

libremdb        | warn  - You have enabled experimental features (images, isrMemoryCacheSize) in next.config.mjs.
libremdb        | warn  - Experimental features are not covered by semver, and may cause unexpected or broken application behavior. Use at your own risk.
libremdb        | 
libremdb        | info  - SWC minify release candidate enabled. https://nextjs.link/swcmin
libremdb        | [AxiosError: Request failed with status code 403] {
libremdb        |   code: 'ERR_BAD_REQUEST',
libremdb        |   config: {
libremdb        |     transitional: {
libremdb        |       silentJSONParsing: true,
libremdb        |       forcedJSONParsing: true,
libremdb        |       clarifyTimeoutError: false
libremdb        |     },
libremdb        |     adapter: [Function: httpAdapter],
libremdb        |     transformRequest: [ [Function: transformRequest] ],
libremdb        |     transformResponse: [ [Function: transformResponse] ],
libremdb        |     timeout: 50000,
libremdb        |     xsrfCookieName: 'XSRF-TOKEN',
libremdb        |     xsrfHeaderName: 'X-XSRF-TOKEN',
libremdb        |     maxContentLength: -1,
libremdb        |     maxBodyLength: -1,
libremdb        |     env: { FormData: [Function] },
libremdb        |     validateStatus: [Function: validateStatus],
libremdb        |     headers: {
libremdb        |       Accept: 'application/json, text/plain, */*',
libremdb        |       'User-Agent': 'axios/0.27.2'
libremdb        |     },
libremdb        |     baseURL: 'https://www.imdb.com/',
libremdb        |     url: '/title/tt1630029',
libremdb        |     method: 'get',
libremdb        |     data: undefined
libremdb        |   },
libremdb        |   request: <ref *1> ClientRequest {
libremdb        |     _events: [Object: null prototype] {
libremdb        |       abort: [Function (anonymous)],
libremdb        |       aborted: [Function (anonymous)],
libremdb        |       connect: [Function (anonymous)],
libremdb        |       error: [Function (anonymous)],
libremdb        |       socket: [Function (anonymous)],
libremdb        |       timeout: [Function (anonymous)],
libremdb        |       finish: [Function: requestOnFinish]
libremdb        |     },
libremdb        |     _eventsCount: 7,
libremdb        |     _maxListeners: undefined,
libremdb        |     outputData: [],
libremdb        |     outputSize: 0,
libremdb        |     writable: true,
libremdb        |     destroyed: false,
libremdb        |     _last: true,
libremdb        |     chunkedEncoding: false,
libremdb        |     shouldKeepAlive: false,
libremdb        |     maxRequestsOnConnectionReached: false,
libremdb        |     _defaultKeepAlive: true,
libremdb        |     useChunkedEncodingByDefault: false,
libremdb        |     sendDate: false,
libremdb        |     _removedConnection: false,
libremdb        |     _removedContLen: false,
libremdb        |     _removedTE: false,
libremdb        |     strictContentLength: false,
libremdb        |     _contentLength: 0,
libremdb        |     _hasBody: true,
libremdb        |     _trailer: '',
libremdb        |     finished: true,
libremdb        |     _headerSent: true,
libremdb        |     _closed: false,
libremdb        |     socket: TLSSocket {
libremdb        |       _tlsOptions: [Object],
libremdb        |       _secureEstablished: true,
libremdb        |       _securePending: false,
libremdb        |       _newSessionPending: false,
libremdb        |       _controlReleased: true,
libremdb        |       secureConnecting: false,
libremdb        |       _SNICallback: null,
libremdb        |       servername: 'www.imdb.com',
libremdb        |       alpnProtocol: false,
libremdb        |       authorized: true,
libremdb        |       authorizationError: null,
libremdb        |       encrypted: true,
libremdb        |       _events: [Object: null prototype],
libremdb        |       _eventsCount: 10,
libremdb        |       connecting: false,
libremdb        |       _hadError: false,
libremdb        |       _parent: null,
libremdb        |       _host: 'www.imdb.com',
libremdb        |       _closeAfterHandlingError: false,
libremdb        |       _readableState: [ReadableState],
libremdb        |       _maxListeners: undefined,
libremdb        |       _writableState: [WritableState],
libremdb        |       allowHalfOpen: false,
libremdb        |       _sockname: null,
libremdb        |       _pendingData: null,
libremdb        |       _pendingEncoding: '',
libremdb        |       server: undefined,
libremdb        |       _server: null,
libremdb        |       ssl: [TLSWrap],
libremdb        |       _requestCert: true,
libremdb        |       _rejectUnauthorized: true,
libremdb        |       parser: null,
libremdb        |       _httpMessage: [Circular *1],
libremdb        |       timeout: 50000,
libremdb        |       [Symbol(res)]: [TLSWrap],
libremdb        |       [Symbol(verified)]: true,
libremdb        |       [Symbol(pendingSession)]: null,
libremdb        |       [Symbol(async_id_symbol)]: 374,
libremdb        |       [Symbol(kHandle)]: [TLSWrap],
libremdb        |       [Symbol(lastWriteQueueSize)]: 0,
libremdb        |       [Symbol(timeout)]: Timeout {
libremdb        |         _idleTimeout: 50000,
libremdb        |         _idlePrev: [TimersList],
libremdb        |         _idleNext: [TimersList],
libremdb        |         _idleStart: 68144,
libremdb        |         _onTimeout: [Function: bound ],
libremdb        |         _timerArgs: undefined,
libremdb        |         _repeat: null,
libremdb        |         _destroyed: false,
libremdb        |         [Symbol(refed)]: false,
libremdb        |         [Symbol(kHasPrimitive)]: false,
libremdb        |         [Symbol(asyncId)]: 382,
libremdb        |         [Symbol(triggerId)]: 377
libremdb        |       },
libremdb        |       [Symbol(kBuffer)]: null,
libremdb        |       [Symbol(kBufferCb)]: null,
libremdb        |       [Symbol(kBufferGen)]: null,
libremdb        |       [Symbol(kCapture)]: false,
libremdb        |       [Symbol(kSetNoDelay)]: false,
libremdb        |       [Symbol(kSetKeepAlive)]: true,
libremdb        |       [Symbol(kSetKeepAliveInitialDelay)]: 60,
libremdb        |       [Symbol(kBytesRead)]: 0,
libremdb        |       [Symbol(kBytesWritten)]: 0,
libremdb        |       [Symbol(connect-options)]: [Object]
libremdb        |     },
libremdb        |     _header: 'GET /title/tt1630029 HTTP/1.1\r\n' +
libremdb        |       'Accept: application/json, text/plain, */*\r\n' +
libremdb        |       'User-Agent: axios/0.27.2\r\n' +
libremdb        |       'Host: www.imdb.com\r\n' +
libremdb        |       'Connection: close\r\n' +
libremdb        |       '\r\n',
libremdb        |     _keepAliveTimeout: 0,
libremdb        |     _onPendingData: [Function: nop],
libremdb        |     agent: Agent {
libremdb        |       _events: [Object: null prototype],
libremdb        |       _eventsCount: 2,
libremdb        |       _maxListeners: undefined,
libremdb        |       defaultPort: 443,
libremdb        |       protocol: 'https:',
libremdb        |       options: [Object: null prototype],
libremdb        |       requests: [Object: null prototype] {},
libremdb        |       sockets: [Object: null prototype],
libremdb        |       freeSockets: [Object: null prototype] {},
libremdb        |       keepAliveMsecs: 1000,
libremdb        |       keepAlive: false,
libremdb        |       maxSockets: Infinity,
libremdb        |       maxFreeSockets: 256,
libremdb        |       scheduling: 'lifo',
libremdb        |       maxTotalSockets: Infinity,
libremdb        |       totalSocketCount: 1,
libremdb        |       maxCachedSessions: 100,
libremdb        |       _sessionCache: [Object],
libremdb        |       [Symbol(kCapture)]: false
libremdb        |     },
libremdb        |     socketPath: undefined,
libremdb        |     method: 'GET',
libremdb        |     maxHeaderSize: undefined,
libremdb        |     insecureHTTPParser: undefined,
libremdb        |     path: '/title/tt1630029',
libremdb        |     _ended: true,
libremdb        |     res: IncomingMessage {
libremdb        |       _readableState: [ReadableState],
libremdb        |       _events: [Object: null prototype],
libremdb        |       _eventsCount: 4,
libremdb        |       _maxListeners: undefined,
libremdb        |       socket: [TLSSocket],
libremdb        |       httpVersionMajor: 1,
libremdb        |       httpVersionMinor: 1,
libremdb        |       httpVersion: '1.1',
libremdb        |       complete: true,
libremdb        |       rawHeaders: [Array],
libremdb        |       rawTrailers: [],
libremdb        |       aborted: false,
libremdb        |       upgrade: false,
libremdb        |       url: '',
libremdb        |       method: null,
libremdb        |       statusCode: 403,
libremdb        |       statusMessage: 'Forbidden',
libremdb        |       client: [TLSSocket],
libremdb        |       _consuming: false,
libremdb        |       _dumped: false,
libremdb        |       req: [Circular *1],
libremdb        |       responseUrl: 'https://www.imdb.com/title/tt1630029',
libremdb        |       redirects: [],
libremdb        |       [Symbol(kCapture)]: false,
libremdb        |       [Symbol(kHeaders)]: [Object],
libremdb        |       [Symbol(kHeadersCount)]: 26,
libremdb        |       [Symbol(kTrailers)]: null,
libremdb        |       [Symbol(kTrailersCount)]: 0
libremdb        |     },
libremdb        |     aborted: false,
libremdb        |     timeoutCb: null,
libremdb        |     upgradeOrConnect: false,
libremdb        |     parser: null,
libremdb        |     maxHeadersCount: null,
libremdb        |     reusedSocket: false,
libremdb        |     host: 'www.imdb.com',
libremdb        |     protocol: 'https:',
libremdb        |     _redirectable: Writable {
libremdb        |       _writableState: [WritableState],
libremdb        |       _events: [Object: null prototype],
libremdb        |       _eventsCount: 3,
libremdb        |       _maxListeners: undefined,
libremdb        |       _options: [Object],
libremdb        |       _ended: true,
libremdb        |       _ending: true,
libremdb        |       _redirectCount: 0,
libremdb        |       _redirects: [],
libremdb        |       _requestBodyLength: 0,
libremdb        |       _requestBodyBuffers: [],
libremdb        |       _onNativeResponse: [Function (anonymous)],
libremdb        |       _currentRequest: [Circular *1],
libremdb        |       _currentUrl: 'https://www.imdb.com/title/tt1630029',
libremdb        |       _timeout: null,
libremdb        |       [Symbol(kCapture)]: false
libremdb        |     },
libremdb        |     [Symbol(kCapture)]: false,
libremdb        |     [Symbol(kBytesWritten)]: 0,
libremdb        |     [Symbol(kEndCalled)]: true,
libremdb        |     [Symbol(kNeedDrain)]: false,
libremdb        |     [Symbol(corked)]: 0,
libremdb        |     [Symbol(kOutHeaders)]: [Object: null prototype] {
libremdb        |       accept: [Array],
libremdb        |       'user-agent': [Array],
libremdb        |       host: [Array]
libremdb        |     },
libremdb        |     [Symbol(kUniqueHeaders)]: null
libremdb        |   },
libremdb        |   response: {
libremdb        |     status: 403,
libremdb        |     statusText: 'Forbidden',
libremdb        |     headers: {
libremdb        |       'content-type': 'text/html',
libremdb        |       'content-length': '118',
libremdb        |       connection: 'close',
libremdb        |       server: 'Server',
libremdb        |       date: 'Thu, 01 Dec 2022 17:34:01 GMT',
libremdb        |       'x-amz-rid': 'PTQW2H6BAT19XA7BSTJM',
libremdb        |       'strict-transport-security': 'max-age=47474747; includeSubDomains; preload',
libremdb        |       vary: 'Content-Type,Accept-Encoding,User-Agent',
libremdb        |       'x-cache': 'Error from cloudfront',
libremdb        |       via: '1.1 e0a5445a9b6b20c3399e57d2c05d4520.cloudfront.net (CloudFront)',
libremdb        |       'x-amz-cf-pop': 'ARN1-C1',
libremdb        |       'alt-svc': 'h3=":443"; ma=86400',
libremdb        |       'x-amz-cf-id': '1zEu1yU9TOve8707Ssb5XxsBBQYgCuyYnEvj1GxqBWZYnPybfNeRUQ=='
libremdb        |     },
libremdb        |     config: {
libremdb        |       transitional: [Object],
libremdb        |       adapter: [Function: httpAdapter],
libremdb        |       transformRequest: [Array],
libremdb        |       transformResponse: [Array],
libremdb        |       timeout: 50000,
libremdb        |       xsrfCookieName: 'XSRF-TOKEN',
libremdb        |       xsrfHeaderName: 'X-XSRF-TOKEN',
libremdb        |       maxContentLength: -1,
libremdb        |       maxBodyLength: -1,
libremdb        |       env: [Object],
libremdb        |       validateStatus: [Function: validateStatus],
libremdb        |       headers: [Object],
libremdb        |       baseURL: 'https://www.imdb.com/',
libremdb        |       url: '/title/tt1630029',
libremdb        |       method: 'get',
libremdb        |       data: undefined
libremdb        |     },
libremdb        |     request: <ref *1> ClientRequest {
libremdb        |       _events: [Object: null prototype],
libremdb        |       _eventsCount: 7,
libremdb        |       _maxListeners: undefined,
libremdb        |       outputData: [],
libremdb        |       outputSize: 0,
libremdb        |       writable: true,
libremdb        |       destroyed: false,
libremdb        |       _last: true,
libremdb        |       chunkedEncoding: false,
libremdb        |       shouldKeepAlive: false,
libremdb        |       maxRequestsOnConnectionReached: false,
libremdb        |       _defaultKeepAlive: true,
libremdb        |       useChunkedEncodingByDefault: false,
libremdb        |       sendDate: false,
libremdb        |       _removedConnection: false,
libremdb        |       _removedContLen: false,
libremdb        |       _removedTE: false,
libremdb        |       strictContentLength: false,
libremdb        |       _contentLength: 0,
libremdb        |       _hasBody: true,
libremdb        |       _trailer: '',
libremdb        |       finished: true,
libremdb        |       _headerSent: true,
libremdb        |       _closed: false,
libremdb        |       socket: [TLSSocket],
libremdb        |       _header: 'GET /title/tt1630029 HTTP/1.1\r\n' +
libremdb        |         'Accept: application/json, text/plain, */*\r\n' +
libremdb        |         'User-Agent: axios/0.27.2\r\n' +
libremdb        |         'Host: www.imdb.com\r\n' +
libremdb        |         'Connection: close\r\n' +
libremdb        |         '\r\n',
libremdb        |       _keepAliveTimeout: 0,
libremdb        |       _onPendingData: [Function: nop],
libremdb        |       agent: [Agent],
libremdb        |       socketPath: undefined,
libremdb        |       method: 'GET',
libremdb        |       maxHeaderSize: undefined,
libremdb        |       insecureHTTPParser: undefined,
libremdb        |       path: '/title/tt1630029',
libremdb        |       _ended: true,
libremdb        |       res: [IncomingMessage],
libremdb        |       aborted: false,
libremdb        |       timeoutCb: null,
libremdb        |       upgradeOrConnect: false,
libremdb        |       parser: null,
libremdb        |       maxHeadersCount: null,
libremdb        |       reusedSocket: false,
libremdb        |       host: 'www.imdb.com',
libremdb        |       protocol: 'https:',
libremdb        |       _redirectable: [Writable],
libremdb        |       [Symbol(kCapture)]: false,
libremdb        |       [Symbol(kBytesWritten)]: 0,
libremdb        |       [Symbol(kEndCalled)]: true,
libremdb        |       [Symbol(kNeedDrain)]: false,
libremdb        |       [Symbol(corked)]: 0,
libremdb        |       [Symbol(kOutHeaders)]: [Object: null prototype],
libremdb        |       [Symbol(kUniqueHeaders)]: null
libremdb        |     },
libremdb        |     data: '<html>\r\n' +
libremdb        |       '<head><title>403 Forbidden</title></head>\r\n' +
libremdb        |       '<body>\r\n' +
libremdb        |       '<center><h1>403 Forbidden</h1></center>\r\n' +
libremdb        |       '</body>\r\n' +
libremdb        |       '</html>\r\n'
libremdb        |   }
libremdb        | }

[mohammadrafigh]

Well we get 403 from imdb, some header is missing?

[mohammadrafigh]

Found the problem. It seems IMDB is blocking requests from default Axios User Agent. Changing user agent to something different like Firefox fixed the problem.

[zyachel]

seems like they don't like axios' default headers. Can you try changing them to generic browser ones?

AXIOS_USERAGENT='Mozilla/5.0 (X11; Linux x86_64; rv:104.0) Gecko/20100101 Firefox/104.0'
AXIOS_ACCEPT='text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8'

try adding these to your .env.local, or docker-compose.yml

[mohammadrafigh]

We got it at the same time ;)

[zyachel]

Saw you comment after sending mine :p

I'll make these headers the default.

[mariopaolo]

@zyachel tried the suggested values and can confirm it now works. thanks

[danielwerg]

I'm getting this error right now.

[zyachel]

@danielwerg just fixed it in 8ce02d02364c8e1f03a8b16594bc20ee6766a8c6.

I don't get an email unless someone mentions, so please do mention me if you get error next time, or even better, open a new issue :)