Wraith does not work for HTTPS urls

[renettarenula]

Hi there,

I am trying out wraith for a site and unfortunately, it only generates empty images for a HTTPS site. Now, I have been looking at issues on Github and it seems that I have to put this setting in my config/component.yaml:

phantomjs_options: --ssl-protocol=tlsv1

However, it is still generating empty images.

When I try it with a non-HTTPS page, the screenshot is created as expected. So, I am guessing that this is related to HTTPS issue.

This is my config/component.yaml file for the HTTPS (non-working) version.

# Add as many domains as necessary. Key will act as a label
phantomjs_options: --ssl-protocol=tlsv1
domains:
  english: "https://www.viki.com"

# (optional) JavaScript file to execute before taking screenshot of every path.
# Can be used to reload current page with headers specified, or to dynamically
# AJAX in content, or all manner of other uses!
#before_capture: 'javascript/global.js'

# Type page URL paths below, here are a couple of examples
paths:
  home: /

# amount of fuzz ImageMagick will use when comparing images. A higher fuzz makes the comparison less strict.
fuzz: '20%'

# the maximum acceptable level of difference (in %) between two images.
# Wraith considers it a failure if an image diff goes above this threshold.
threshold: 5

# screen widths (and optional height) to resize the browser to before taking the screenshot
screen_widths:
  - 320
  - 600x768
  - 768
  - 1024
  - 1280

# the engine to run Wraith with.
browser:
  phantomjs: "casperjs"   # variant of PhantomJS that allows screenshots by element selector
  #phantomjs: "phantomjs" # PhantomJS, built on top of Webkit and JavascriptCore (like Safari)
  #slimerjs: "slimerjs"   # SlimerJS, built on top of Gecko and SpiderMonkey (like Firefox)

# the file in charge of taking the screenshot
snap_file: "javascript/casper.js"

# the directory that your base screenshots will be stored in
history_dir: 'shots_base'

# the directory that your latest screenshots will be stored in
directory: 'shots'

# choose how results are displayed in the gallery (default is `alphanumeric` if omitted)
# Different screen widths are always grouped together.
# Options:
#   alphanumeric - all paths (with or without a difference) are shown, sorted by path
#   diffs_first - all paths (with or without a difference) are shown, sorted by difference size (largest first)
#   diffs_only - only paths with a difference are shown, sorted by difference size (largest first)
mode: alphanumeric

Are there any sort of settings that I may have missed pertaining to HTTPS settings?

In case if it helps, my system informations are:

PhantomJS version 1.9.7 ImageMagick 6.9.2-5 CasperJS version 1.0.0 Mac OS X El Capitan Version 10.11.1

Thanks in advance for your help!

[fredrivett]

@renettarenula I'm getting the same issue, http works fine, but https not so, just generates a blank png.

My component.yaml file is like so:

# Add as many domains as necessary. Key will act as a label
domains:
  english: "https://www.google.com"

# (optional) JavaScript file to execute before taking screenshot of every path.
# Can be used to reload current page with headers specified, or to dynamically
# AJAX in content, or all manner of other uses!
#before_capture: 'javascript/global.js'

# Type page URL paths below, here are a couple of examples
paths:
  home: /

  # clickable_guide:
  #   path: /news/entertainment-arts-27221191
  #   selector: '.idt__news' # selector to take a screenshot of
  # clickable_guide__after_click:
  #   path: /news/entertainment-arts-27221191
  #   selector: '.idt__news'
  #   # (optional) JavaScript file to execute before taking the screenshot of this path.
  #   # This is always executed AFTER the 'global' before_capture hook toward the top of this file.
  #   before_capture: 'javascript/beforeCapture--casper_example.js'

# amount of fuzz ImageMagick will use when comparing images. A higher fuzz makes the comparison less strict.
fuzz: '20%'

# the maximum acceptable level of difference (in %) between two images.
# Wraith considers it a failure if an image diff goes above this threshold.
threshold: 5

# screen widths (and optional height) to resize the browser to before taking the screenshot
screen_widths:
  - 320
  - 600x768
  - 768
  - 1024
  - 1280

# the engine to run Wraith with.
browser:
  phantomjs: "casperjs"   # variant of PhantomJS that allows screenshots by element selector
  #phantomjs: "phantomjs" # PhantomJS, built on top of Webkit and JavascriptCore (like Safari)
  #slimerjs: "slimerjs"   # SlimerJS, built on top of Gecko and SpiderMonkey (like Firefox)

# the file in charge of taking the screenshot
snap_file: "config/wraith/javascript/casper.js"

# the directory that your base screenshots will be stored in
history_dir: 'shots_base'

# the directory that your latest screenshots will be stored in
directory: 'shots'

# choose how results are displayed in the gallery (default is `alphanumeric` if omitted)
# Different screen widths are always grouped together.
# Options:
#   alphanumeric - all paths (with or without a difference) are shown, sorted by path
#   diffs_first - all paths (with or without a difference) are shown, sorted by difference size (largest first)
#   diffs_only - only paths with a difference are shown, sorted by difference size (largest first)
mode: alphanumeric

With terminal output of:

$ wraith capture config/wraith/configs/component.yaml
Creating Folders
2015-11-05 11:29:52.190 phantomjs[41532:23337360] *** WARNING: Method userSpaceScaleFactor in class NSView is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.
2015-11-05 11:29:52.196 phantomjs[41533:23337361] *** WARNING: Method userSpaceScaleFactor in class NSView is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.
2015-11-05 11:29:52.209 phantomjs[41534:23337362] *** WARNING: Method userSpaceScaleFactor in class NSView is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.
2015-11-05 11:29:52.222 phantomjs[41535:23337363] *** WARNING: Method userSpaceScaleFactor in class NSView is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.
2015-11-05 11:29:52.313 phantomjs[41536:23337364] *** WARNING: Method userSpaceScaleFactor in class NSView is deprecated on 10.7 and later. It should not be used in new applications. Use convertRectToBacking: instead.
Snapping https://www.google.com/ at: 600x768
Snapping https://www.google.com/ at: 1024x1500
Snapping https://www.google.com/ at: 768x1500
Snapping https://www.google.com/ at: 320x1500
Snapping https://www.google.com/ at: 1280x1500
cropping images
cropping images
cropping images
/Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/wraith-2.7.0/lib/wraith/crop.rb:40:in `initialize': no implicit conversion of nil into String (TypeError)
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/wraith-2.7.0/lib/wraith/crop.rb:40:in `open'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/wraith-2.7.0/lib/wraith/crop.rb:40:in `image_dimensions'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/wraith-2.7.0/lib/wraith/crop.rb:21:in `block in crop_images'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:410:in `call'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:410:in `call_with_index'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:379:in `process_incoming_jobs'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:361:in `block in worker'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:354:in `fork'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:354:in `worker'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:345:in `block in create_workers'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:344:in `each'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:344:in `create_workers'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:301:in `work_in_processes'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:232:in `map'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/parallel-1.6.1/lib/parallel.rb:194:in `each'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/wraith-2.7.0/lib/wraith/crop.rb:16:in `crop_images'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/wraith-2.7.0/lib/wraith/cli.rb:64:in `crop_images'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/wraith-2.7.0/lib/wraith/cli.rb:91:in `capture'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/gems/wraith-2.7.0/bin/wraith:5:in `<top (required)>'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/bin/wraith:23:in `load'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/bin/wraith:23:in `<main>'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/bin/ruby_executable_hooks:15:in `eval'
    from /Users/fredrivett/.rvm/gems/ruby-2.2.2@tpo-social/bin/ruby_executable_hooks:15:in `<main>'

[ChrisBAshton]

Hi @renettarenula - there is no phantomjs_options property in Wraith, so whatever you put there in your config file is not going to be passed to the PhantomJS command line.

It would have to be something we add to a future version of Wraith. Just to confirm (after a bit of looking around), it looks like the following extra command line options would be useful:

--ignore-ssl-errors=true
--ssl-protocol=tlsv1

We could perhaps add this to a future version of Wraith as follows:

ignore_ssl_errors: true # false by default
ssl_protocol: 'tlsv1' # 'SSLv3' by default. Change as per http://phantomjs.org/api/command-line.html

These options would apply to both PhantomJS and CasperJS. More investigation required to see if this would also be of benefit to SlimerJS.

[peruukki]

@ChrisBAshton, actually there is phantomjs_options in Wraith, see e.g. https://github.com/BBC-News/wraith/pull/251. I'm not sure if it's documented, though.

[ChrisBAshton]

@peruukki it appears you're right! Thanks for saying. I'll make a note to add this to the documentation.

[jmdodge]

I'm having a similar issue, but using casperjs - is there any way to pass command line options to casperjs?

[jmdodge]

In my case, I resolved this issue another way - I had installed phantomjs via Ubuntu's apt package manager where it was apparently a bit out of date. When I removed and reinstalled via npm, a more recent version was downloaded and it wasn't necessary to do anything beyond the default to get a successful capture on my https site.

It still sounds like a nice feature to have extra command line options available to you though.

[aaronbauman]

Thanks for the guidance, throwing the ignore errors solved it for me, since i'm using a self-signed cert:

phantomjs_options: --ignore-ssl-errors=true

[renettarenula]

Hi all, I just wanna let your guys know that as of Wraith version 3.1.4, this issue no longer happens for me. So feel free to close the task (unless if some of you are still having issues). Thank you for the fix!

[CharlesBelov]

Still happening for me in Wraith 3.1.4 with apt-get version of phantomjs (1.9.0-1). Installed phantomjs version 2.1.1 and it works fine! Be sure to uninstall the old version of phantomjs.

[whyAto8]

Adding phantomjs_options: --ignore-ssl-errors=true worked for me, but --ssl-protocol=tlsv1 didnt.

[kentr]

Wraith 4.0.0 and 4.0.1 have been working for me consistently with HTTPS urls.

I'm using these options:

phantomjs_options: '--ignore-ssl-errors=true --ssl-protocol=tlsv1'

[swati-qa]

For phantomJS it worked, can someone tell what should be done when using casperJS browser. Adding this line does not help here.

[kentr]

@swati-qa, I am using the config in my previous comment with CasperJS.

Looks like one person in this thread noted that upgrading PhantomJS solved the problem for them.

On Apr 17, 2017, at 5:33 AM, swati-qa notifications@github.com wrote:

For phantomJS it worked, can someone tell what should be done when using casperJS browser. Adding this line does not help here.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[chandragupthan]

Hello, I am still having this problem. I am using Wraith 4.2.1 and I have tried phantomjs 2.1.1 (current latest) and 1.9.7 and 1.9.8 and all having the same problem. Empty images are captured when an "https" site is used. The same works fine when I use an "http" site. Any pointers please?

[DobriGenchev]

Hello, as @chandragupthan pointed out the problem still exists. I am running Wraith 4.2.3 (on Windows machine), PhantomJS 1.9.7 (due to Windows issue) and I am not able to get proper image over https. Any help is appreciated!

[kentr]

I suspect it's related to the system OpenSSL library.

Try --ssl-protocol=any

From the PhantomJS docs:

--ssl-protocol=[sslv3|sslv2|tlsv1|tlsv1.1|tlsv1.2|any] sets the SSL protocol for secure connections (default is SSLv3). Not all values may be supported, depending on the system OpenSSL library.

For me, https URLs are still working fine with this setup:

• wraith 4.2.1
• phantoms 2.1.1
• casperjs 1.1.3
• macOS High Sierra
• OpenSSL 1.0.2q (MacPorts)
• --ssl-protocol=tlsv1 or --ssl-protocol=any