Open CharlesBelov opened 8 years ago
CharlesBelov
commented
8 years ago If I edit the config file by deleting all but two pages and all but two screen sizes, wraith runs through to completion and I get a gallery, so I've verified that the installation is correct.
Stanton
commented
8 years ago I have the same issue (I believe), I have 417 paths being captured. restarting my local machine allows my task to complete, but that's far from ideal.
DEBUG: #################################################
DEBUG: Command run: latest wraith.yml
DEBUG: Wraith version: 3.2.1
DEBUG: Ruby version: ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin16]
DEBUG: ImageMagick: Version: ImageMagick 6.9.5-9 Q16 x86_64 2016-09-10 http://www.imagemagick.org
DEBUG: PhantomJS version: 2.1.1
DEBUG: CasperJS version: 1.1.2
DEBUG: #################################################
DEBUG:
Config validated. No serious issues found.
JimBledsoe
commented
7 years ago I ran into the same problem processing a site with 2000+ pages and while I don't have a fix, I have a hackey work-around. After the spidering is complete and it begins processing images, I run a bash script in another session that takes a snapshot of the phantomjs process list and compares it to a list 60 seconds later and kills and session that is still running. Wraith will spin up a new attempt up to 5 times for a single page. The script keeps looping until all phantomjs processes have completed.
A configuration file and CasperJS files that work fine for me on Ubuntu result in the capture process stalling and failing to resume when run in a new install of wraith on Mac OS X.
Repeated attempts to run results in the stall happening at different points in the run.
Details:
I have just installed wraith on a Mac OS X 10.11.6 with CasperJS and am attempting my first successful run, with the following command:
wraith capture /Users/[userid]/wraith/configs/sfmta_full.yaml 2> errors.txt
It starts well, with the following messages:
WARN: You have specified a
history_dirin your config, but this is used inhistorymode, NOTcapturemode. See the docs at http://bbc-news.github.io/wraith/ Config validated. No serious issues found. Creating Folders SAVING IMAGESFollowed by 2095 (first try) or 205 (second try) or 505 (try below) occurrences of:
Snapping [page] at: [screen size]
(I'm expecting Snapping to occur 139 pages x 10 screen sizes x 2 websites = 2780 times)
At some point, it stalls. The ps command shows it is still running, but it seems to be just sitting there 20 minutes later :
20283 ttys001 0:00.41 /System/Library/Frameworks/Ruby.framework/Versions/2.0 20325 ttys001 0:16.05 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20328 ttys001 0:16.17 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20329 ttys001 0:04.62 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20330 ttys001 0:05.00 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20331 ttys001 0:15.14 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20332 ttys001 0:14.96 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20335 ttys001 0:02.53 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20336 ttys001 0:04.23 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin
and a several minutes later
20283 ttys001 0:00.41 /System/Library/Frameworks/Ruby.framework/Versions/2.0 20325 ttys001 0:16.89 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20328 ttys001 0:17.02 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20329 ttys001 0:05.47 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20330 ttys001 0:05.84 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20331 ttys001 0:16.09 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20332 ttys001 0:15.90 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20335 ttys001 0:03.48 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20336 ttys001 0:05.18 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin
Casper is still using time, but the Ruby framework isn't.
Unlike on Ubuntu, I never see a process called wraith.
If I look at the files that have been created on the 205 "Snapping" run:
9 folders have the expected 20 snapshots each 2 folders each have 10 snapshots: a snapshot from each site for five of the screen sizes 1 folder has 5 snapshots: a snapshot from one site for the same five screen sizes
Several minutes later, Casper is still using time, but no new images:
20283 ttys001 0:00.41 /System/Library/Frameworks/Ruby.framework/Versions/2.0 20325 ttys001 0:17.73 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20328 ttys001 0:17.86 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20329 ttys001 0:06.30 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20330 ttys001 0:06.68 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20331 ttys001 0:17.02 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20332 ttys001 0:16.84 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20335 ttys001 0:04.42 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin 20336 ttys001 0:06.10 phantomjs /usr/local/Cellar/casperjs/1.1.3/libexec/bin
On the 505 try, I have 26 folders with 20 snapshots each and one folder, chineseProjRightColumn, with 5 snapshots:
979x1500_casperjs_staging.png 980x1500_casperjs_staging.png 1199x1500_casperjs_staging.png 1200x1500_casperjs_staging.png 1600x1500_casperjs_staging.png
On this run, the snapshots are from the staging site, but on the 205 run they were from the webmaster site.
The config file is the exact same one that is working fine on Ubuntu.
My OS X installation was mostly but not completely routine: I installed imagemagick and casperJS using homebrew in the default location (/usr/local/) I installed wraith using gem install wraith Wraith version is 3.2.1
Only thing out of the ordinary was that my initial gem install wraith failed because nokogiri complained that a library it needed wasn't present, and to reinstall nokogiri using
gem install nokogiri -- --use-system-libraries
I did so, and then "gem install wraith" worked.
However, if I do a "gem list" it shows two versions of nokogiri installed:
nokogiri (1.6.8, 1.5.6)
The only CasperJS command in the config file is:
before_capture: 'javascript/wait--casper.js'
which executes:
module.exports = function (casper, ready) { // make Wraith wait a bit longer before taking the screenshot casper.wait(10000, ready); // you MUST call the ready() callback for Wraith to continue }
Again, this Casper file was copied from Ubuntu and worked there.
Any ideas on how to unstick wraith?
Reporting a problem? Please describe the issue above, and complete the following checklist so that we can help you more quickly.
Issue checklist:
verbose: trueto my config) and pasted the output below: (IDs and password redacted)