search

rajatomar788 / pywebcopy

Locally saves webpages to your hard disk with images, css, js & links as is.
https://rajatomar788.github.io/pywebcopy/
Other
527 stars 106 forks source link

BUG REPORT: Log file never flushes causing drive to run out of space #36

Closed kaavik closed 4 years ago

kaavik commented 4 years ago

I have a tool that uses pywebcopy to save_webpage for a site with specific IP data. The problem is that I am using a loop in my tool to make repeated calls to pywebcopy with a different project_name each time BUT pywebcopy module continues to add to a persistent pywebcopy_log.log file with each call AND writes a copy of this increasing log file to the project_folder after each call to save_webpage.

EXPECTED BEHAVIOR: Log file flushed with each call to pywebcopy.save_webpage with a new project_name (and associated project_folder).

So if each call to pywebcopy.save_webpage() generated a 1MB log, the first call for the first project_name would write a 1MB file to project_folder_1, the second call to the second project_name would write a 2MB file to project_folder_2, the third call to the third project_name would write a 3MB file to project_folder_3, etc.

This becomes a major problem when you might loop through hundreds or thousands of project_names (my tool currently processes 1600 web pages - which is 1600 separate project_names - per batch). As you can probably imagine, this becomes very problematic when the log size of the first few projects is only a few kilobytes but the logs for later projects has grown to +25MB each.

This is definitely a bug and needs to be fixed in order to support pywebcopy logging with anything more than a single project_name.

SAMPLE OUTPUT SHOWING ERRORS AND DISK UTILIZATION: Preserving 'DNS' from intodns.com for beetfarmprepared.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for cranberrypowers.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for lifesavertip.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for handraisedvote.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for glasscleardeals.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for tallfootball.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for notordered.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for tellingbanks.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for gigglesshared.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for maybeinevitable.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for gettingreadytoleave.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for gettingouthere.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for actuallytoogood.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for laughterjump.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for movinghopfence.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Preserving 'DNS' from intodns.com for stayrealpeace.com... ExceptionType: <class 'OSError'>, Exception: OSError(28, 'No space left on device') Traceback (most recent call last): File "chirp.py", line 494, in <module> main() File "chirp.py", line 487, in main Evidence(cidrfile, sleep) File "chirp.py", line 96, in __init__ self.write_json_to_file(self.dir, self.basename, self.records) File "chirp.py", line 474, in write_json_to_file with open(jpath, 'w') as f: OSError: [Errno 28] No space left on device: 'master.batches.20200307.2145/batch001.20200307.2151/batch001.json' Sun Mar 8 06:38:39 EDT 2020 (venv36) [root@oszlmu01 ~/cpt/chirp]# pwd /root/cpt/chirp You have new mail in /var/spool/mail/root (venv36) [root@oszlmu01 ~/cpt/chirp]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 909M 0 909M 0% /dev tmpfs 920M 4.0K 920M 1% /dev/shm tmpfs 920M 97M 823M 11% /run tmpfs 920M 0 920M 0% /sys/fs/cgroup /dev/sda5 9.8G 9.1G 782M 93% / /dev/sda3 2.0G 33M 2.0G 2% /tmp /dev/sda1 197M 177M 21M 90% /boot tmpfs 184M 0 184M 0% /run/user/0 (venv36) [root@oszlmu01 ~/cpt/chirp]#

EXAMPLE OF PROBLEM: (Note that size of pywebcopy_log.log continues to increase) (venv36) [root@oszlmu01 ~/cpt/chirp]# find . -name "pywebcopy_log.log" -exec ls -lh {} \; -rw-r--r--. 1 root root 29M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/quipito.com/pywebcopy_log.log -rw-r--r--. 1 root root 29M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydecorism.com/pywebcopy_log.log -rw-r--r--. 1 root root 28M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydesignish.com/pywebcopy_log.log -rw-r--r--. 1 root root 28M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydesignize.com/pywebcopy_log.log -rw-r--r--. 1 root root 28M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydesignery.com/pywebcopy_log.log -rw-r--r--. 1 root root 28M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydesignism.com/pywebcopy_log.log -rw-r--r--. 1 root root 28M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydetailish.com/pywebcopy_log.log -rw-r--r--. 1 root root 28M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydetailize.com/pywebcopy_log.log -rw-r--r--. 1 root root 28M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydetailery.com/pywebcopy_log.log -rw-r--r--. 1 root root 28M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nancydetailism.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadecorish.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadecorize.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadecorery.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadecorism.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadesignish.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadesignize.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadesignery.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadesignism.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadetailish.com/pywebcopy_log.log -rw-r--r--. 1 root root 27M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadetailize.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadetailery.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/normadetailism.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nicoledecor.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nicoledetail.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nicoledecorish.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nicoledecorize.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nicoledecorery.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nicoledecorism.com/pywebcopy_log.log -rw-r--r--. 1 root root 26M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nicoledesignish.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nicoledesignize.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/uncoveredtaste.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/amusingflower.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/pourcontinue.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/agreementthumb.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/supportlumber.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/divisionbustling.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/marvelousbelief.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/stewjewel.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/willinginsect.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/memorisewrap.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/hystericalfuturistic.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/emptyannounce.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/expertelated.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/supremethoughtful.com/pywebcopy_log.log -rw-r--r--. 1 root root 25M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/workableruddy.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/bootdistance.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/plausiblenose.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/chillyvanish.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/finehesitant.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/broadhumorous.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/concentrateviolet.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/equablecagey.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/attractpossess.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/closeddescribe.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/gratistooth.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/squealingritzy.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lazyillustrious.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/answercoat.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/purringtoes.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverdecor.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/washerze.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/washerism.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/washerish.com/pywebcopy_log.log -rw-r--r--. 1 root root 24M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverdecorish.com/pywebcopy_log.log -rw-r--r--. 1 root root 23M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverdecorize.com/pywebcopy_log.log -rw-r--r--. 1 root root 23M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverdecorery.com/pywebcopy_log.log -rw-r--r--. 1 root root 23M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverdecorism.com/pywebcopy_log.log -rw-r--r--. 1 root root 23M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverdesignish.com/pywebcopy_log.log -rw-r--r--. 1 root root 23M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverdesignize.com/pywebcopy_log.log -rw-r--r--. 1 root root 23M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverdesignism.com/pywebcopy_log.log -rw-r--r--. 1 root root 23M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverlandscapeish.com/pywebcopy_log.log -rw-r--r--. 1 root root 23M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/riverlandscapeism.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakedecorish.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakedecorize.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakedecorism.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakedesignish.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakedesignize.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakedesignery.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakedesignism.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakelandscapeish.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakelandscapeize.com/pywebcopy_log.log -rw-r--r--. 1 root root 22M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lakelandscapeism.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/poddecorish.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/poddecorize.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/poddecorery.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/hutandhouse.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/ingloohomes.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/brinkhomesinwind.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/skilledruth.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/clumsylier.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/hslipper.com/pywebcopy_log.log -rw-r--r--. 1 root root 21M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/romanboat.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/uoutput.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/coldclipper.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/fuzzypruner.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/ibegonia.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/realferry.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/fbengal.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/urgentquartz.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/silkycard.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/modernmice.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/neatradar.com/pywebcopy_log.log -rw-r--r--. 1 root root 20M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/stormystarter.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nuttyseat.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/ldogsled.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/unawaregold.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/marriedcall.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/funnyelement.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/lbuffet.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/amateurlaundry.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/visualoval.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/oddchive.com/pywebcopy_log.log -rw-r--r--. 1 root root 19M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/puremodem.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/subtlepalm.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/purplegeorge.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/crudelute.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nmitten.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/honorabletelevision.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/fearlesscommercial.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/obviousregulation.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/polishedbank.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/acclaimedunion.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/remarkabletime.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/fluidaccounting.com/pywebcopy_log.log -rw-r--r--. 1 root root 18M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/nutritiouscosts.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/classicworkforce.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/chiefroom.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/experiencedplace.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/formalworkplace.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/substantialtelevision.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/coordinatedbreaks.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/calculatingage.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/judiciouscost.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/largemission.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/beneficialplace.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/astonishingmission.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/caringscheme.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/awarepackage.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/possiblepractices.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/imaginativehealth.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/attentiveemployee.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/profitablebroker.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/delectablemarket.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/valuablebalance.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/anothertax.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/delectabletime.com/pywebcopy_log.log -rw-r--r--. 1 root root 17M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/algorithmscoal.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/algorithmsingredient.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/algorithmsnod.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/archdistributor.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/archoverlay.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/clayboot.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/attorneycharacters.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/bailcharacters.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/bootpossibilities.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/chapteringredient.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/chapterpainter.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/chaptervictim.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/characterspossibilities.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/claydebris.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/clayseas.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/computationscoal.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/courtsseas.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/crossbutt.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/debriscomputations.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/debrispossibilities.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/debristears.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/debristrousers.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/debrisvolt.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/directionssecond.com/pywebcopy_log.log -rw-r--r--. 1 root root 16M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/directionsvictim.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/dolliesvolt.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/dwellingredient.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/dwellseas.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/elbowsarch.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/experiencedperformance.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/familiarweb.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/formalopportunity.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/forthrightadvertising.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/forthrightsector.com/pywebcopy_log.log -rw-r--r--. 1 root root 15M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/forthrightwebsite.com/pywebcopy_log.log -rw-r--r--. 1 root root 14M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/fortunatestock.com/pywebcopy_log.log -rw-r--r--. 1 root root 14M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/glitteringissue.com/pywebcopy_log.log -rw-r--r--. 1 root root 14M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/gracefulexchange.com/pywebcopy_log.log -rw-r--r--. 1 root root 14M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/graciouspayments.com/pywebcopy_log.log -rw-r--r--. 1 root root 14M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/gratefulcosts.com/pywebcopy_log.log -rw-r--r--. 1 root root 14M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/groundedbroker.com/pywebcopy_log.log -rw-r--r--. 1 root root 14M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/groundedhuman.com/pywebcopy_log.log -rw-r--r--. 1 root root 14M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/harmlesstrademark.com/pywebcopy_log.log -rw-r--r--. 1 root root 13M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/helpfulemployee.com/pywebcopy_log.log -rw-r--r--. 1 root root 13M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/helpfulplanning.com/pywebcopy_log.log -rw-r--r--. 1 root root 13M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/imaginativecosts.com/pywebcopy_log.log -rw-r--r--. 1 root root 13M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/immaculateaccounting.com/pywebcopy_log.log -rw-r--r--. 1 root root 13M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/immaculatetax.com/pywebcopy_log.log -rw-r--r--. 1 root root 13M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/immediateagency.com/pywebcopy_log.log -rw-r--r--. 1 root root 13M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/infatuatedsales.com/pywebcopy_log.log -rw-r--r--. 1 root root 13M Mar 8 06:38 ./master.batches.20200307.2145/batch001.20200307.2151/judiciousloan.com/pywebcopy_log.log

ADDITIONAL LOG DETAILS: --- Logging error --- Traceback (most recent call last): File "/usr/lib64/python3.6/logging/__init__.py", line 998, in emit self.flush() File "/usr/lib64/python3.6/logging/__init__.py", line 978, in flush self.stream.flush() OSError: [Errno 28] No space left on device Call stack: File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/root/cpt/chirp/venv36/lib64/python3.6/site-packages/pywebcopy/elements.py", line 334, in run % (len(self._stack), self.file_path)) Message: '[2] CSS linked files are found in file [/root/cpt/chirp/master.batches.20200307.2145/batch001.20200307.2151/veteransfinancesolutions.com/intodns.com/static/style/9f1e1094__thickbox.css]' Arguments: () --- Logging error --- Traceback (most recent call last): File "/usr/lib64/python3.6/logging/__init__.py", line 998, in emit self.flush() File "/usr/lib64/python3.6/logging/__init__.py", line 978, in flush self.stream.flush() OSError: [Errno 28] No space left on device Call stack: File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/root/cpt/chirp/venv36/lib64/python3.6/site-packages/pywebcopy/elements.py", line 334, in run % (len(self._stack), self.file_path)) Message: '[2] CSS linked files are found in file [/root/cpt/chirp/master.batches.20200307.2145/batch001.20200307.2151/veteransfinancesolutions.com/intodns.com/static/style/9f1e1094__thickbox.css]' Arguments: () --- Logging error --- Traceback (most recent call last): File "/usr/lib64/python3.6/logging/__init__.py", line 998, in emit self.flush() File "/usr/lib64/python3.6/logging/__init__.py", line 978, in flush self.stream.flush() OSError: [Errno 28] No space left on device Call stack: File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/root/cpt/chirp/venv36/lib64/python3.6/site-packages/pywebcopy/elements.py", line 334, in run % (len(self._stack), self.file_path)) Message: '[2] CSS linked files are found in file [/root/cpt/chirp/master.batches.20200307.2145/batch001.20200307.2151/veteransfinancesolutions.com/intodns.com/static/style/9f1e1094__thickbox.css]' Arguments: () --- Logging error --- Traceback (most recent call last): File "/usr/lib64/python3.6/logging/__init__.py", line 998, in emit self.flush() File "/usr/lib64/python3.6/logging/__init__.py", line 978, in flush self.stream.flush() OSError: [Errno 28] No space left on device Call stack: File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/root/cpt/chirp/venv36/lib64/python3.6/site-packages/pywebcopy/elements.py", line 334, in run % (len(self._stack), self.file_path)) Message: '[2] CSS linked files are found in file [/root/cpt/chirp/master.batches.20200307.2145/batch001.20200307.2151/veteransfinancesolutions.com/intodns.com/static/style/9f1e1094__thickbox.css]' Arguments: () --- Logging error --- Traceback (most recent call last): File "/usr/lib64/python3.6/logging/__init__.py", line 998, in emit self.flush() File "/usr/lib64/python3.6/logging/__init__.py", line 978, in flush self.stream.flush() OSError: [Errno 28] No space left on device Call stack: File "/usr/lib64/python3.6/threading.py", line 884, in _bootstrap self._bootstrap_inner() File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib64/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/root/cpt/chirp/venv36/lib64/python3.6/site-packages/pywebcopy/elements.py", line 334, in run % (len(self._stack), self.file_path)) Message: '[2] CSS linked files are found in file [/root/cpt/chirp/master.batches.20200307.2145/batch001.20200307.2151/veteransfinancesolutions.com/intodns.com/static/style/9f1e1094__thickbox.css]' Arguments: ()

rajatomar788 commented 4 years ago

Fix should remove this issue otherwise reopen it again.

kaavik commented 4 years ago

The fix worked well enough. It still seemed "noisy" in my usage - displaying a bunch of logs to console which seemed to indicate there may be a problem but, in the end, it worked without writing any logfile and kept my system from running out of disk space. :-)

@rajatomar788 - thank you for addressing the issue so quickly!