search

facundoolano / google-play-scraper

Node.js scraper to get data from Google Play
MIT License
2.27k stars 623 forks source link

IP problem #665

Open Zexiong-Xiang opened 8 months ago

Zexiong-Xiang commented 8 months ago

Description:

Now, I installed dependencies, Run the demo, Try switching IP proxies, But it always times out or the read fails, and I think there is a problem with the request of the source code itself.

Example code:

import gplay from "google-play-scraper";

gplay.list({
    category: gplay.category.GAME_ACTION,
    collection: gplay.collection.TOP_FREE,
    num: 2
  })
  .then(console.log, console.log);

Error message:

[Running] node "c:\Users\axiong\PycharmProjects\Privacy_Policy\10.12\index.js"
Error: Error requesting Google Play:connect ETIMEDOUT 142.251.42.238:443
    at request (file:///c:/Users/axiong/node_modules/google-play-scraper/lib/utils/request.js:40:17)
    at processTicksAndRejections (internal/process/task_queues.js:93:5) {
  status: undefined
}
[Done] exited with code=0 in 21.704 seconds

Or

[Running] node "c:\Users\axiong\PycharmProjects\Privacy_Policy\10.12\index.js"
Error: Error requesting Google Play:read ECONNRESET
    at request (file:///c:/Users/axiong/node_modules/google-play-scraper/lib/utils/request.js:40:17)
    at processTicksAndRejections (internal/process/task_queues.js:93:5) {
  status: undefined
}
[Done] exited with code=0 in 15.788 seconds
walrus543 commented 2 months ago

I had this issue some times ago: https://github.com/facundoolano/google-play-scraper/issues/400#issuecomment-625848506 And it's back...

{
  title: 'USAF Delayed Entry Program',
  description: 'The USAF Delayed Entry Program ( DEP ) app provides Air Force recruits mobile access to essential information. The app allows for self-improvement and preparation prior to entering the strict environments of Basic Military Training (BMT) and Officer Training School (OTS). Users are able to access reporting statements, drill movements, training schedules and fitness and nutrition information. The USAF DEP app also includes a fitness calculator, interactive packing and financial readiness checklists, quizzes, and flash cards to help prep DEP members for Basic Military and Officer Training.',
  descriptionHTML: 'The USAF Delayed Entry Program ( DEP ) app provides Air Force recruits mobile access to essential information. The app allows for self-improvement and preparation prior to entering the strict environments of Basic Military Training (BMT) and Officer Training School (OTS). Users are able to access reporting statements, drill movements, training schedules and fitness and nutrition information. The USAF DEP app also includes a fitness calculator, interactive packing and financial readiness checklists, quizzes, and flash cards to help prep DEP members for Basic Military and Officer Training.',
  summary: 'The USAF Delayed Entry Program ( DEP ) app is an aid for Air Force recruits.',
  installs: '100,000+',
  minInstalls: 100000,
  maxInstalls: 101525,
  score: 4.21,
  scoreText: '4.2',
  ratings: 183,
  reviews: 55,
  histogram: { '1': 17, '2': 6, '3': 16, '4': 16, '5': 124 },
  price: 0,
  originalPrice: undefined,
  discountEndDate: undefined,
  free: true,
  currency: 'USD',
  priceText: 'Free',
  available: true,
  offersIAP: false,
  IAPRange: undefined,
  androidVersion: '5.0',
  androidVersionText: '5.0',
  androidMaxVersion: 'VARY',
  developer: 'US Air Force Recruiting Service',
  developerId: 'US+Air+Force+Recruiting+Service',
  developerEmail: 'AFRS.RSIXS.MobileApp@us.af.mil',
  developerWebsite: 'http://www.recruiting.af.mil/',
  developerAddress: undefined,
  privacyPolicy: 'https://airforce.com/privacy-policy',
  developerInternalID: 'US+Air+Force+Recruiting+Service',
  genre: 'Books & Reference',
  genreId: 'BOOKS_AND_REFERENCE',
  categories: [ { name: 'Books & Reference', id: 'BOOKS_AND_REFERENCE' } ],
  icon: 'https://play-lh.googleusercontent.com/letedTyP5xNkOUNYuXhWrMElx6TizDEwOHnuygwV6ZrL1bdX6yo1zDRc86Vif9983Eg',
  headerImage: 'https://play-lh.googleusercontent.com/LyVuNOv-MxkuCWnMynAB8zfWk-fu7XK_cnPb1wDUT2Sa5NVTzSEW-qEF1TMCN4CAzCA',
  screenshots: [
    'https://play-lh.googleusercontent.com/J-JTgKP5QYKXHk_U4viijpvY5hLL4EkVD0oh1EOzZRU5nlnMbXp_QWw9qCA3IWCzz1A',
    'https://play-lh.googleusercontent.com/pfTXXRFjzIL9z8XE9n6eDYkrSMqFkZKYCLRZvIrRke3H0R6iVN1-mEpRzyrLt_lP5Og',
    'https://play-lh.googleusercontent.com/uxsQbop5Q8W-NW8JiCfBPx1g6OW5agRi2sS-db_8Xr7GWW7e3JC3dce4rmvk_mza7qLc',
    'https://play-lh.googleusercontent.com/X09WFZnYF_fPaE6Q3MT7qD6GYdf8bmMeOIj4zhPE4Eqy-KIvjNmXloSHEr6Rb3NVwA',
    'https://play-lh.googleusercontent.com/FJ5HWxQyzcjKYTYSJwu-EBp7syGvgdAqJzz8bMNrtYjDobBnD0L34rJ4l49OUfAW0J8w',
    'https://play-lh.googleusercontent.com/fOWCAxuTWc4xfH7HUCqgKYHuW1ttaSctEW5BUOOGy9RCntVE0CM1tPVCREMBwj3bAQ',
    'https://play-lh.googleusercontent.com/sdFKhH_ftBpv1eplrQuXjTWryOH-K7u4L-McCu-YeZyWYm3Hy_xx_9LQwo8yynNEvw',
    'https://play-lh.googleusercontent.com/p9VRxYrKib2GH3hjp6mb6FyWDoNITcwaPwca0CnBpdgw48k9pSRlOqWtYIMYnNpNRg',
    'https://play-lh.googleusercontent.com/VDbdLTHneo1_kiok_Zi9Jm6K7S89BcNx_oOt4rGHBcHk8R11iLuHrj997KG7UXBiPg',
    'https://play-lh.googleusercontent.com/T8Xz6Ab-tYsZ-EiIb4uAgRopchnlb3KCCHMcd-5oIHsxkN1nFRwLsf1e9Ds-pZlD06c',
    'https://play-lh.googleusercontent.com/zMRa0Y8au5ao5TxJ7gZoue6a8dKFiO_O5r--h89P7dUMS3IQHpd_2wnsZvZNUB4wwaw',
    'https://play-lh.googleusercontent.com/OHz5eD7PUe3U9cO4iQqi0wul8riBsFhFZO-4u3WqXl_lxAqJZqrrVN5XapBnhZt3d6s',
    'https://play-lh.googleusercontent.com/wYLIRv5YEcH7xQiSS4Sk8CTeAD98Pnh__OB9iy1_KE8PANi14Vt5efcPaLoQ4I01CPo',
    'https://play-lh.googleusercontent.com/4hlaobbXLjkJKU5wpWsRp2rfA3zF0og1O-lkwCv2ux5wSW9SlGttm1LIeEK-qJatfGc',
    'https://play-lh.googleusercontent.com/PwLBrOKu4uA9o48OcCR8LlZAM7_KlVBmemrdxYbb6tNOLJfDH53FMYF9xoXMtXssR_U',
    'https://play-lh.googleusercontent.com/87ZE4gazp5pm_GJUHkFu-HX0m9iSGPla_61RSE3cRT75Hbec-U1AxPMLWnpXRFLSmfE',
    'https://play-lh.googleusercontent.com/YNZTxu1RtgNvpZYy1Hr7Ijt1y2V9Y5S1q7Rn0fbWpzwu4UPvv3h1aHTKqvjFhiqfBts',
    'https://play-lh.googleusercontent.com/Vnv8dNODucGi_lw3q3LFkTm8b0nVk5x_6TpglMp49tJbh22PWrgsxF7mAlURYg8hHe0',
    'https://play-lh.googleusercontent.com/vyLNzVgfEvM6KpO3HWN8DNw6pTwcZbGg-XZgTDk2rXUcJxGpXLkRctuAGYw432LeFg',
    'https://play-lh.googleusercontent.com/1E_MIDSKuHxMdoTswdKXU8xApwTC2XNl1Q--X9TJ9VsF7cSmVP6lCXk2OUgYv8eIkg',
    'https://play-lh.googleusercontent.com/s2CKzp68wwJyF-R1s0fbBMg2hE2IVMOtKK_I6Qz8te6mmYQaIQoUvVEto7-jaVXgY3w',
    'https://play-lh.googleusercontent.com/z5AwOcbWv3kpxB9W0GCY_r0KPsrhxnLlpX-BKODRbG0IUREgIlQ7nlfCAvNeh6ypHNc',
    'https://play-lh.googleusercontent.com/_K4f8cXKFmChkoyNicFZzW5ecbFSxsfOBv8DuGV_aqdnj8k3ZqxrZ2eZuhXb_IAlKKqg',
    'https://play-lh.googleusercontent.com/KfApSWQASMhjTLIXHWCd2CZYWizH0bnFU82EnJtqWOG_dzNr4qEx9lik2LXmR9L1vRk'
  ],
  video: undefined,
  videoImage: undefined,
  previewVideo: undefined,
  contentRating: 'Everyone 10+',
  contentRatingDescription: undefined,
  adSupported: false,
  released: 'Feb 1, 2018',
  updated: 1680294257000,
  version: '2.3.5',
  recentChanges: '- Updated OTS Arriving &gt; Reporting directions<br>- Updated BMT Packing List<br>- Updated BMT Chain of Command',
  comments: [ undefined ],
  preregister: false,
  earlyAccessEnabled: false,
  isAvailableInPlayPass: false,
  appId: 'com.afrs.delayedentryprogram',
  url: 'https://play.google.com/store/apps/details?id=com.afrs.delayedentryprogram&hl=en&gl=us'
}
node:events:496
      throw er; // Unhandled 'error' event
      ^

AggregateError [ETIMEDOUT]: 
    at internalConnectMultiple (node:net:1116:18)
    at afterConnectMultiple (node:net:1683:7)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (node:_http_client:500:9)
    at TLSSocket.emit (node:events:518:28)
    at emitErrorNT (node:internal/streams/destroy:169:8)
    at emitErrorCloseNT (node:internal/streams/destroy:128:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ETIMEDOUT',
  [errors]: [
    Error: connect ETIMEDOUT 142.250.187.246:443
        at createConnectionError (node:net:1646:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1705:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '142.250.187.246',
      port: 443
    },
    Error: connect ETIMEDOUT 2a00:1450:4009:81f::2016:443
        at createConnectionError (node:net:1646:14)
        at afterConnectMultiple (node:net:1676:16) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '2a00:1450:4009:81f::2016',
      port: 443
    }
  ]

I switched to different IPs/countries via ProtonVPN. I can download a few icons then the downloads are hanging. Without ProtonVPN, I can't download anything.