search

jef / streetmerchant

🤖 The world's easiest, most powerful stock checker
https://jef.buzz/streetmerchant
MIT License
4.97k stars 1.3k forks source link

Heavy Disk Usage Spikes #2376

Closed mattrunyon closed 3 years ago

mattrunyon commented 3 years ago

Expected Behavior

Disk read/write should be minimal.

Current Behavior

Disk usage has frequent bursts of 100MB/s+ writes when street merchant is running. With street merchant not running, never goes above 100 kB/s writes

Steps to Reproduce

I am running street merchant with npm start

Environment

Ubuntu 18.04 VM on a Hyper-V VM in Windows 10. VM has 8GB RAM allocated and 2 vCPUs. Hyper-V smart paging file is set to a different drive than the drive containing the virtual hard disk. Virtual hard disk is 20GB.

dotenv 
ASCII_BANNER=
ASCII_COLOR=
APNS_AUTHKEY=
APNS_BUNDLEID=
APNS_DEVICETOKEN=
APNS_KEYID=
APNS_PRODUCTION=
APNS_TEAMID=
AUTO_ADD_TO_CART=
BROWSER_TRUSTED=
DESKTOP_NOTIFICATIONS=
DISCORD_NOTIFY_GROUP=
DISCORD_NOTIFY_GROUP_3060=
DISCORD_NOTIFY_GROUP_3060TI=
DISCORD_NOTIFY_GROUP_3070=
DISCORD_NOTIFY_GROUP_3080=
DISCORD_NOTIFY_GROUP_3090=
DISCORD_NOTIFY_GROUP_CORSAIR_SF=
DISCORD_NOTIFY_GROUP_RX6800=
DISCORD_NOTIFY_GROUP_RX6800XT=
DISCORD_NOTIFY_GROUP_RX6900XT=
DISCORD_NOTIFY_GROUP_RYZEN5600=
DISCORD_NOTIFY_GROUP_RYZEN5800=
DISCORD_NOTIFY_GROUP_RYZEN5900=
DISCORD_NOTIFY_GROUP_RYZEN5950=
DISCORD_NOTIFY_GROUP_SONYPS5C=
DISCORD_NOTIFY_GROUP_SONYPS5DE=
DISCORD_NOTIFY_GROUP_XBOXSX=
DISCORD_NOTIFY_GROUP_XBOXSS=
DISCORD_NOTIFY_GROUP_TEST=
DISCORD_WEB_HOOK=
EMAIL_PASSWORD=<removed>
EMAIL_TO=
EMAIL_USERNAME=<removed>
HEADLESS=
IN_STOCK_WAIT_TIME=
LOG_LEVEL=-1
LOW_BANDWIDTH=
MAX_PRICE_SERIES_3060="350"
MAX_PRICE_SERIES_3060TI=
MAX_PRICE_SERIES_3070="650"
MAX_PRICE_SERIES_3080="850"
MAX_PRICE_SERIES_3090=
MAX_PRICE_SERIES_CORSAIR_SF=
MAX_PRICE_SERIES_RX6800=
MAX_PRICE_SERIES_RX6800XT=
MAX_PRICE_SERIES_RX6900XT=
MAX_PRICE_SERIES_RYZEN5600=
MAX_PRICE_SERIES_RYZEN5800=
MAX_PRICE_SERIES_RYZEN5900=
MAX_PRICE_SERIES_RYZEN5950=
MAX_PRICE_SERIES_SONYPS5C=
MAX_PRICE_SERIES_SONYPS5DE=
MAX_PRICE_SERIES_XBOXSS=
MAX_PRICE_SERIES_XBOXSX=
MAX_PRICE_SERIES_TEST=
MICROCENTER_LOCATION=
MQTT_BROKER_ADDRESS=
MQTT_BROKER_PORT=
MQTT_CLIENT_ID=
MQTT_PASSWORD=
MQTT_QOS=
MQTT_TOPIC=
MQTT_USERNAME=
OPEN_BROWSER=
PAGE_BACKOFF_MIN=
PAGE_BACKOFF_MAX=
PAGE_SLEEP_MIN=
PAGE_SLEEP_MAX=
PAGE_TIMEOUT=
PAGERDUTY_INTEGRATION_KEY=
PAGERDUTY_SEVERITY=
PHILIPS_HUE_API_KEY=
PHILIPS_HUE_CLOUD_ACCESS_TOKEN=
PHILIPS_HUE_CLOUD_CLIENT_ID=
PHILIPS_HUE_CLOUD_CLIENT_SECRET=
PHILIPS_HUE_CLOUD_REFRESH_TOKEN=
PHILIPS_HUE_LAN_BRIDGE_IP=
PHILIPS_HUE_LIGHT_COLOR=
PHILIPS_HUE_LIGHT_IDS=
PHILIPS_HUE_LIGHT_PATTERN=
PHONE_CARRIER=tmobile
PHONE_NUMBER=<removed>
PLAY_SOUND=
PROXY_ADDRESS=
PROXY_PROTOCOL=
PROXY_PORT=
PUSHBULLET=
PUSHOVER_EXPIRE=
PUSHOVER_DEVICE=
PUSHOVER_PRIORITY=
PUSHOVER_RETRY=
PUSHOVER_SOUND=
PUSHOVER_TOKEN=
PUSHOVER_USER=
RESTART_TIME=
SCREENSHOT=
SHOW_ONLY_BRANDS=
SHOW_ONLY_MODELS=
SHOW_ONLY_SERIES=3060,3070,3080
SLACK_CHANNEL=
SLACK_TOKEN=
SMARTTHINGS_TOKEN=
SMARTTHINGS_SWITCH_LABEL=
SMTP_ADDRESS=
SMTP_PORT=
STORES=amazon,antonline,bestbuy,newegg,asus,bandh,corsair,evga,gamestop,microcenter,officedepot,pny,target,walmart,zotac
TELEGRAM_ACCESS_TOKEN=
TELEGRAM_CHAT_ID=
TWILIO_ACCOUNT_SID=
TWILIO_AUTH_TOKEN=
TWILIO_FROM_NUMBER=
TWILIO_TO_NUMBER=
TWITCH_ACCESS_TOKEN=
TWITCH_CHANNEL=
TWITCH_CLIENT_ID=
TWITCH_CLIENT_SECRET=
TWITCH_REFRESH_TOKEN=
TWITTER_ACCESS_TOKEN_KEY=
TWITTER_ACCESS_TOKEN_SECRET=
TWITTER_CONSUMER_KEY=
TWITTER_CONSUMER_SECRET=
TWITTER_TWEET_TAGS=
STREAMLABS_ACCESS_TOKEN=
STREAMLABS_TYPE=
STREAMLABS_IMAGE=
STREAMLABS_SOUND=
STREAMLABS_DURATION=
FREEMOBILE_ID=
FREEMOBILE_API_KEY=
WEB_PORT=

Logs

Can't find any info on where logs are stored, but using iotop, jdb2 and puppeteer are the offenders.

AlexP11223 commented 3 years ago

Why is this a problem?

I guess Chrome writes some kind of caches or other data. You would probably have the same behavior if using Chrome manually, especially opening/closing many tabs all the time like SM does.

mattrunyon commented 3 years ago

I haven't monitored it longer than a few minutes yet, but yesterday after running it for 2 days it was maxing my SSD I/O to the point I had to forcibly shut down the VM. The SSD is just a data drive on the host OS and wasn't being used.

It's a problem because it will cause people's drives to die early. I think it wrote a couple TBs of data in just 2 days from what crystal disk info shows. It also made my SSD hit 50C sustained temp when it normally hovers at 28-30C.

In addition to early drive failure, it may cause slowdowns to people's systems. When I tried to read a file from the disk that the VM uses for it's virtual disk, it took 10s+ to load

It might be more of a problem with puppeteer. Normally using chrome doesn't cause any sorts of writes like this. I've seen peaks of 450MB/s for a few seconds indicating it's writing 1GB of data in that time.

AlexP11223 commented 3 years ago

Normally using chrome doesn't cause any sorts of writes like this.

But SM is not normal usage. It is opening hundreds of pages per minute.

It's a problem because it will cause people's drives to die early.

Still cheaper than buying GPUs from scalpers :D

For me it is like this, 50 MB/s max, 8 stores. Maybe +0.1 TB after 2-3 hours.

https://user-images.githubusercontent.com/5680466/115074488-6d73f900-9f02-11eb-8811-b71d7e34c72b.mp4

One way to minimize disk writes is to stop SM when it is not likely to be useful, like at night or maybe even outside of working hours when most stores probably will not update.

Also maybe check VM/OS config, add more RAM if needed, disable swap if enabled, etc.

AlexP11223 commented 3 years ago

fatrace says that indeed lots of some chrome cache-related files are written.

https://user-images.githubusercontent.com/5680466/115076586-70241d80-9f05-11eb-8b09-a14c2047d8b6.mp4

mattrunyon commented 3 years ago

Weird when I run it on my main Windows install it peaks at 10MB/s or so with the same settings. Gonna try in a Windows VM next to see if it's related to Hyper-V or if it might be some weird Ubuntu setting/setup.

mattrunyon commented 3 years ago

As a temporary fix I've mounted /tmp as a ramdisk using tmpfs which reduces the writes to 0 since it seems to be writing only to /tmp.

Still not sure why it's writing so much on Ubuntu in Hyper-V and significantly less in Windows 10.

AlexP11223 commented 3 years ago

Still not sure why it's writing so much on Ubuntu in Hyper-V and significantly less in Windows 10.

I think Linux disk i/o is supposed to be more efficient than on Windows, so maybe that's why Chrome on Linux caches to disk more aggressively.

Also maybe it depends on the amount of RAM available.

As a temporary fix I've mounted /tmp as a ramdisk using tmpfs

Why temporary? Sounds like a good idea unless you do not have enough RAM, but for me it does not seem to exceed 400-500MB.

mattrunyon commented 3 years ago

Why temporary? Sounds like a good idea unless you do not have enough RAM, but for me it does not seem to exceed 400-500MB.

Guess I should say it's a permanent fix for me lol. Probably not a better way around it as it's most likely due to puppeteer if I had to guess.

For anybody here in the future, I added this to my /etc/fstab to mount /tmp as tmpfs (RAM disk) on boot

tmpfs /tmp tmpfs rw,nosuid,nodev

Seems to be running fine w/ 4GB RAM on Ubuntu desktop 18.04. Uses around 2-3GB. Could probably drop to 2GB if using a lighter desktop than Ubuntu 18's default or no desktop environment at all.