Kalhama / finder-scraper

Työkalu yritystietojen hakuun Finderistä ja YTJ:n avoimesta rajapinnasta
3 stars 0 forks source link

504 error - timeout encountered after 142 pages scraped #11

Closed JJooaa closed 6 months ago

JJooaa commented 7 months ago
response: {
    status: 504,
    statusText: 'Gateway Time-out',
    headers: Object [AxiosHeaders] {
      server: 'awselb/2.0',
      date: 'Tue, 09 Apr 2024 20:26:49 GMT',
      'content-type': 'text/html',
      'content-length': '132',
      connection: 'keep-alive'
    },
    config: {
      transitional: {
        silentJSONParsing: true,
        forcedJSONParsing: true,
        clarifyTimeoutError: false
      },
      adapter: [ 'xhr', 'http' ],
      transformRequest: [ [Function: transformRequest] ],
      transformResponse: [ [Function: transformResponse] ],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      env: {
        FormData: [Function: FormData] {
          LINE_BREAK: '\r\n',
          DEFAULT_CONTENT_TYPE: 'application/octet-stream'
        },
        Blob: [class Blob]
      },
      validateStatus: [Function: validateStatus],
      headers: Object [AxiosHeaders] {
        Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8',
        'Content-Type': undefined,
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/115.0',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        DNT: '1',
        Connection: 'keep-alive',
        Cookie: 'session=s%3APPC-IsHQADUivTp9UkEg9-Y1LH6wkqw7.Q%2FRstJwXD3ezFHTCI1m%2F9wDZgvEr8hVRDfFIW9mmWBo',
        'Upgrade-Insecure-Requests': '1',
        'Sec-Fetch-Dest': 'document',
        'Sec-Fetch-Mode': 'navigate',
        'Sec-Fetch-Site': 'none',
        'Sec-Fetch-User': '?1'
      },
      method: 'get',
      url: 'https://www.finder.fi/Paperit+paperituotteet+kartongit/Cardetta+Oy/Humppila/yhteystiedot/228348',
      data: undefined
    },
Kalhama commented 6 months ago

fixed in version 2.3.0