scraperwiki / code-scraper-in-browser-tool

Just like on ScraperWiki Classic; now a part of QuickCode.
https://quickcode.io
Other
38 stars 8 forks source link

98GB log file #122

Open zarino opened 10 years ago

zarino commented 10 years ago

A box on our free server was taking up 98GB of space. Turns out it was a code-in-your-browser tool, with a 98GB logs/out file.

The box was created at 22:35 on 25th October 2013, and its status was last updated (to "ok") 6 minutes later, at 22:41 on the 25th. It has no crontab.

The box ID is am72wxy. When I SSHed in this morning, the code was still running.

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND                                                                                                              
18152 am72wxy   20   0  175m  14m 6704 R  69.5  0.1   8446:21 php                                                                                                                  
16550 am72wxy   20   0 17636 1292  976 R   0.0  0.0   0:00.01 top                                                                                                                  
18147 am72wxy   20   0  4336  368  288 S   0.0  0.0   0:00.00 flock                                                                                                                
18149 am72wxy   20   0  9460 1284 1092 S   0.0  0.0   0:00.00 actually-run                                                                                                         
22868 am72wxy   20   0 18108 2032 1556 S   0.0  0.0   0:00.04 bash
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
am72wxy  18147  0.0  0.0   4336   368 ?        S    Oct25   0:00 flock -n ./running.flock bash -c tool/actually-run
am72wxy  18149  0.0  0.0   9460  1284 ?        S    Oct25   0:00 /bin/bash tool/actually-run
am72wxy  18152 61.8  0.0 179628 14348 ?        R    Oct25 8446:34 php ./code/scraper
am72wxy  19872  0.0  0.0  15572  1148 pts/1    R+   09:23   0:00 ps waxwux
am72wxy  22868  0.0  0.0  18108  2032 pts/1    S    09:11   0:00 -su
am72wxy@free-x-live-0:~$ ls -hitlar logs/out 
22814771 -rw-r--r-- 1 am72wxy databox 98G Nov  4 09:25 logs/out
am72wxy@free-x-live-0:~$ tail logs/out
                                                                                                                                                                                                                                (
                                                                                                                                                                                                                                    [nodetype] => 1
                                                                                                                                                                                                                                    [tag] => span
                                                                                                                                                                                                                                    [attr] => Array
                                                                                                                                                                                                                                        (
                                                                                                                                                                                                                                        )

                                                                                                                                                                                                                                    [children] => Array
                                                                                                                                                                                                                                        (

I've killed the script and deleted the log file.

drj11 commented 10 years ago

@morty and I just cleaned up one of these.

We'd like code-scraper-in-browser-tool to police and truncate big log files.

Or we won't approve the next update for release to our platform. :)