prashantghimire / sofifa-web-scraper

It has over 18k detailed players info and stats from EA FC 24 scrapped from SoFIFA.com.
24 stars 2 forks source link

Humanoid stopped working #15

Closed Geffikk closed 3 weeks ago

Geffikk commented 3 months ago

the humanoid stopped working. First when you implemented this update, everything worked fine, but yesterday it stopped working due to the following error. Is this problem only on my site ?

npm run download-urls

starting to download latest player urls...

C:\Projects\sofifa-web-scraper\node_modules\request-promise-core\lib\errors.js:14
        Error.captureStackTrace(this);
              ^
RequestError: TypeError [ERR_INVALID_CHAR]: Invalid character in header content ["User-Agent"]
    at new RequestError (C:\Projects\sofifa-web-scraper\node_modules\request-promise-core\lib\errors.js:14:15)
    at plumbing.callback (C:\Projects\sofifa-web-scraper\node_modules\request-promise-core\lib\plumbing.js:87:29)
    at Request.RP$callback [as _callback] (C:\Projects\sofifa-web-scraper\node_modules\request-promise-core\lib\plumbing.js:46:31)
    at self.callback (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:185:22)
    at Request.emit (node:events:514:28)
    at Request.start (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:753:10)
    at Request.end (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:1505:10)
    ... 2 lines matching cause stack trace ...
    at process.processImmediate (node:internal/timers:478:21) {
  cause: TypeError [ERR_INVALID_CHAR]: Invalid character in header content ["User-Agent"]
      at ClientRequest.setHeader (node:_http_outgoing:655:3)
      at new ClientRequest (node:_http_client:286:14)
      at Object.request (node:https:378:10)
      at Request.start (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:751:32)
      at Request.end (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:1505:10)
      at end (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:564:14)
      at Immediate.<anonymous> (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:578:7)
      at process.processImmediate (node:internal/timers:478:21) {
    code: 'ERR_INVALID_CHAR'
  },
  error: TypeError [ERR_INVALID_CHAR]: Invalid character in header content ["User-Agent"]
      at ClientRequest.setHeader (node:_http_outgoing:655:3)
      at new ClientRequest (node:_http_client:286:14)
      at Object.request (node:https:378:10)
      at Request.start (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:751:32)
      at Request.end (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:1505:10)
      at end (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:564:14)
      at Immediate.<anonymous> (C:\Projects\sofifa-web-scraper\node_modules\request\request.js:578:7)
      at process.processImmediate (node:internal/timers:478:21) {
    code: 'ERR_INVALID_CHAR'
  },
  options: {
    resolveWithFullResponse: true,
    jar: RequestJar {
      _jar: CookieJar {
        enableLooseMode: true,
        store: MemoryCookieStore { idx: {} }
      }
    },
    simple: false,
    gzip: true,
    encoding: null,
    headers: {
      Accept: 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
      'Accept-Encoding': 'gzip, deflate, br',
      Connection: 'keep-alive',
      Host: 'sofifa.com',
      'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36\r'
    },
    method: 'GET',
    uri: 'https://sofifa.com/players?col=oa&sort=desc&offset=0',
    callback: [Function: RP$callback],
    transform: undefined,
    transform2xxOnly: false
  },
  response: undefined
}

Node.js v20.9.0

Process finished with exit code 1
prashantghimire commented 3 months ago

Not sure about this one, it could be humanoid issue specific to the system you're in. This has not been a problem for me, you could try using linux based vms or docker image as workaround.