search

Dam998 / shopify-monitor

Support proxies, keywords, multiple webhook url, avoid ban and more - current version: 1.5
MIT License
202 stars 38 forks source link

issues due to sites more than 10. #10

Open ankurghai opened 4 years ago

ankurghai commented 4 years ago

Hey. my server is crashing every now and then either says Try running less websites or contact Damm998. and the second error at times i get is too many tasks for node.

i had 30 websites in the monitor. then tried creating three versions of the monitor in different directories to run 10 sites each. still got the errors after some time. what am I doing wrong or what would be the best way to setup multiple monitors on my site without them clashing with each other? i have around 30 to 40 sites to monitor.

Thanks.

Dam998 commented 4 years ago

If you run multiple instances you need to change the db name in the config.json because if you didn't all instance work over the same db, so the monitors crash

in config.json you have: "mongodb://127.0.0.1:27017/shopify" in this string change the word 'shopify' for the different instances, no need what do you write, you can write what you want

if you run 3 instances, for example, you can write: "mongodb://127.0.0.1:27017/shopify1" "mongodb://127.0.0.1:27017/shopify2" "mongodb://127.0.0.1:27017/shopify3" in the different config.json files

ankurghai commented 4 years ago

so i did the setup as you said. now i am getting this error today. And i am getting it on a single instance as well

C:\shopify-monitor\src\classes\Discord.js:16 global.config.webhook_url.forEach(x => { ^

TypeError: global.config.webhook_url.forEach is not a function at Object. (C:\shopify-monitor\src\classes\Discord.js:16:27) at Module._compile (internal/modules/cjs/loader.js:1158:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10) at Module.load (internal/modules/cjs/loader.js:1002:32) at Function.Module._load (internal/modules/cjs/loader.js:901:14) at Module.require (internal/modules/cjs/loader.js:1044:19) at require (internal/modules/cjs/helpers.js:77:18) at Object. (C:\shopify-monitor\src\classes\Task.js:4:17) at Module._compile (internal/modules/cjs/loader.js:1158:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:1178:10) at Module.load (internal/modules/cjs/loader.js:1002:32) at Function.Module._load (internal/modules/cjs/loader.js:901:14) at Module.require (internal/modules/cjs/loader.js:1044:19) at require (internal/modules/cjs/helpers.js:77:18) at Object. (C:\shopify-monitor\index.js:6:14) at Module._compile (internal/modules/cjs/loader.js:1158:30)

Dam998 commented 4 years ago

maybe because in your config.json file you didn't insert the webhook_url in an array, like I did in the example

ankurghai commented 4 years ago

Done. So they are up and running now. I'll update here in a few hours. Although i hope they should be fine now. :) Thanks btw

Dam998 commented 4 years ago

you're welcome

ankurghai commented 4 years ago

just bought 5 dc proxies and started the instances on them. working like a charm. gonna go lower than 30 sec now. btw, what's this error linked to?

[ERROR ] cncpts.com has an high level of protection from monitors, please notify Dam998 by opening an issue on github

ankurghai commented 4 years ago

why is this happening a lot in between buddy? i am not sure I understand why this error occurs.

code: 'ERR_SOCKET_CLOSED', May 11 09:57:32 streetbud-monitor node[16748]: config: { May 11 09:57:32 streetbud-monitor node[16748]: url: 'https://wishatl.com/products.json', May 11 09:57:32 streetbud-monitor node[16748]: method: 'get', May 11 09:57:32 streetbud-monitor node[16748]: headers: { May 11 09:57:32 streetbud-monitor node[16748]: Accept: 'application/json, text/plain, /', May 11 09:57:32 streetbud-monitor node[16748]: 'User-Agent': 'axios/0.19.2' May 11 09:57:32 streetbud-monitor node[16748]: }, May 11 09:57:32 streetbud-monitor node[16748]: transformRequest: [ [Function: transformRequest] ], May 11 09:57:32 streetbud-monitor node[16748]: transformResponse: [ [Function: transformResponse] ], May 11 09:57:32 streetbud-monitor node[16748]: timeout: 0, May 11 09:57:32 streetbud-monitor node[16748]: adapter: [Function: httpAdapter], May 11 09:57:32 streetbud-monitor node[16748]: xsrfCookieName: 'XSRF-TOKEN', May 11 09:57:32 streetbud-monitor node[16748]: xsrfHeaderName: 'X-XSRF-TOKEN', May 11 09:57:32 streetbud-monitor node[16748]: maxContentLength: -1, May 11 09:57:32 streetbud-monitor node[16748]: validateStatus: [Function: validateStatus], May 11 09:57:32 streetbud-monitor node[16748]: httpsAgent: HttpsProxyAgent { May 11 09:57:32 streetbud-monitor node[16748]: _events: [Object: null prototype] {}, May 11 09:57:32 streetbud-monitor node[16748]: _eventsCount: 0, May 11 09:57:32 streetbud-monitor node[16748]: _maxListeners: undefined, May 11 09:57:32 streetbud-monitor node[16748]: timeout: null, May 11 09:57:32 streetbud-monitor node[16748]: maxFreeSockets: 1, May 11 09:57:32 streetbud-monitor node[16748]: maxSockets: 1, May 11 09:57:32 streetbud-monitor node[16748]: sockets: {}, May 11 09:57:32 streetbud-monitor node[16748]: requests: {}, May 11 09:57:32 streetbud-monitor node[16748]: secureProxy: false, May 11 09:57:32 streetbud-monitor node[16748]: proxy: [Object], May 11 09:57:32 streetbud-monitor node[16748]: promisifiedCallback: [Function: callback], May 11 09:57:32 streetbud-monitor node[16748]: [Symbol(kCapture)]: false

Dam998 commented 4 years ago

I confirm you that cncpts.com have a protection from monitor, I'll check in the next days how can I pass it but maybe I need to create another monitor specific for that site, I don't know, I'll check it

Dam998 commented 4 years ago

for the error: do you use proxies?

ankurghai commented 4 years ago

for the error: do you use proxies?

yes i do. have 5 private proxies filled in them currently.

Dam998 commented 4 years ago

but the error happen all time you start the monitor? after, more or less, 1 minute?

ankurghai commented 4 years ago

but the error happen all time you start the monitor? after, more or less, 1 minute?

i get this error most of the times throughout my instances, but it's not specific which site it's happening on. it's almost on everyone at some point or another.. doesnt happen straightaway but starts coming up in atleast one of the instance after the first 2-3 minutes. currently i have 4 instances running, all filled with these errors. i am still getting some restock updates but not sure if they are all the stores or just a few which are not crashing.

Dam998 commented 4 years ago

it's seems to be an error of the request with a proxy did you write the proxies well? example http://asdf:qwerty@192.168.254.1:8000

where: asdf is the username qwerty is the password

ankurghai commented 4 years ago

ohh i think i know what the problem is. so, my proxies have 3 concurrent connection support. so maybe they are crashing coz of that. it just has to reset when a 4th connection happens.

just to be clear. if i have 4 instances running up, and i am using 8 proxies which support upto 3 connections at a time. how should i place them between my instances to make sure they dont clash.

also do i need to change the timeout in the index.js to make tasks reload a second late?

ankurghai commented 4 years ago

or should i get more proxies and fill them up? how many would you reckon should not cause any issue and how low can i go with the requestTiming then?

Dam998 commented 4 years ago

Hi, about your last error seems that proxy socket close too early, I reproduce it on my computer this is not a problem of the monitor because with only pc (without proxy) it's working well

I reproduce it with 4 istance of monitor with the same proxies running

I would like to inform you that I'm working on find a solution on this problem, hope to find it fast I'll update you here when I have news

Dam998 commented 4 years ago

do you still have this problem?

ankurghai commented 4 years ago

yepp. like it's not affecting the monitor in the long run as eventually they all connect back. i created a discord bot for monitoring the status of each site inside my monitor instances. so most of the times all sites are connected. only get to see this when I open the JournalCTL of my server.

asdat3 commented 4 years ago

hey, I am sorry I can´t help you, but I am trying since 3 days now to run the monitor 2 times on my windows laptop and I am not able to... I tried nodeenv and got nowhere. Please help

ankurghai commented 4 years ago

hey, I am sorry I can´t help you, but I am trying since 3 days now to run the monitor 2 times on my windows laptop and I am not able to... I tried nodeenv and got nowhere.

Please help

Make sure you change mongodb name in the config.json file like “ipaddress:port/shopify” and “ipaddress:port/shopify2” keeping it different for both of the instances of monitor.. other than that just keep them in two separate folders. Npm install in both of them.. then npm start in both of the directories.

Dam998 commented 4 years ago

yes, you need to change the mongo db name

however @ankurghai do you have the same problem? the ERR_SOCKET_CLOSED? are you using proxy?

asdat3 commented 4 years ago

Thank you so much <3

ankurghai commented 4 years ago

yes, you need to change the mongo db name

however @ankurghai do you have the same problem? the ERR_SOCKET_CLOSED?

are you using proxy?

Yes brother. Still the same. Please read my message two posts above. Tomorrow i’ll get sme more proxies... and i’ll try to put unique proxies in each monitor... lets see if they work without the error then.. and also, does the monitor switch proxy when there is a ban or it circulates proxy with each time a site is accessed?

Dam998 commented 4 years ago

try to remove ?limit=250 in task.js, let me know if works