search

maaaaz / webscreenshot

A simple script to screenshot a list of websites
GNU Lesser General Public License v3.0
653 stars 162 forks source link

Screenshot not being saved #55

Closed MarcoLeder-zz closed 4 years ago

MarcoLeder-zz commented 4 years ago

Dear Maaaaz

First of all thank you so much for such a great tool (eventhough I am not able to use it correctly). I am running latest raspian os (32bit) on my Raspberry Pi 4 and everything with installation worked out pretty darn well. However, if I run the script as intended, it does not save the screenshot nor does it return any errors.

All the folders in which I am working have permission 777, including the screenshots folder which was sucessfully created by your program. I have tried to force ssl and tried different websites but none of them works. I used to use the renderer binary from chromium but as I saw no results switched to PhantomJS, but still nothing.

Here is the debug info (though I don't know if this is worth anything for you):

root@**hostname**:/somepath/# python webscreenshot -vv --renderer-binary /opt/phantomjs-2.1.1-linux-i686/bin/phantomjs 192.168.0.220
webscreenshot.py version 2.91

[DEBUG][General] Options: Namespace(URL='192.168.0.220', ajax_max_timeouts='1400,1800', cookie=None, crop=None, format='png', header=None, http_password=None, http_username=None, imagemagick_binary=None, input_file=None, label=False, label_bg_color='NavajoWhite', label_size=60, log_level='DEBUG', multiprotocol=False, no_xserver=False, output_directory='/somepath/screenshots', port=None, proxy=None, proxy_auth=None, proxy_type=None, quality=75, renderer='phantomjs', renderer_binary='/opt/phantomjs-2.1.1-linux-i686/bin/phantomjs', ssl=False, timeout=30, verbosity=2, window_size='1200,800', workers=4)

[INFO][General] '192.168.0.220' has been formatted as 'http://192.168.0.220:80' with supplied overriding options
[+] 1 URLs to be screenshot
[DEBUG][http://192.168.0.220:80] Shell command to be executed
'/opt/phantomjs-2.1.1-linux-i686/bin/phantomjs --ignore-ssl-errors=true --ssl-protocol=any --ssl-ciphers=ALL "/usr/local/lib/python3.7/dist-packages/webscreenshot/webscreenshot.js" url_capture=http://192.168.0.220:80 output_file="/somepath/screenshots/http_192.168.0.220_80.png" width=1200 height=800 format=png quality=75 ajaxtimeout=1400 maxtimeout=1800'

[+] 1 actual URLs screenshot
[+] 0 error(s)

As you can see I am running latest version of everything so and system is up to date :) Any ideas?

Best regards Marco Leder

maaaaz commented 4 years ago

Hello @MarcoLeder, Thanks for the support !

As debugging steps could you try :

  1. to directly execute this command and look if you have any output (stdout and/or screenshot file)

    /opt/phantomjs-2.1.1-linux-i686/bin/phantomjs --ignore-ssl-errors=true --ssl-protocol=any --ssl-ciphers=ALL "/usr/local/lib/python3.7/dist-packages/webscreenshot/webscreenshot.js" url_capture=http://192.168.0.220:80 output_file="/somepath/screenshots/http_192.168.0.220_80.png" width=1200 height=800 format=png quality=75 ajaxtimeout=1400 maxtimeout=1800

  2. to directly execute this command to look for phantomjs error (see the debug flag) :

    /opt/phantomjs-2.1.1-linux-i686/bin/phantomjs --debug=true --ignore-ssl-errors=true --ssl-protocol=any --ssl-ciphers=ALL "/usr/local/lib/python3.7/dist-packages/webscreenshot/webscreenshot.js" url_capture=http://192.168.0.220:80 output_file="/somepath/screenshots/http_192.168.0.220_80.png" width=1200 height=800 format=png quality=75 ajaxtimeout=1400 maxtimeout=1800

  3. do you have an X server on your raspberry ? because if not, you should pass --no-xserver to webscreenshot (before that, apt-get install xvfb)

  4. Can you try another renderer ? Cheers.

MarcoLeder-zz commented 4 years ago

Ok after quite some tinkering I think we are very close to getting it to work. I do not want to leave any steps out which I did or tried, in case they work for someone else :)

First I ran your 1st suggested command and got the following back:

Auto configuration failed
3069648912:error:25066067:DSO support routines:DLFCN_LOAD:could not load the shared library:dso_dlfcn.c:185:filename(libssl_conf.so): libssl_conf.so: Cannot open the shared object file: File or directory not found
3069648912:error:25070067:DSO support routines:DSO_load:could not load the shared library:dso_lib.c:244:
3069648912:error:0E07506E:configuration file routines:MODULE_LOAD_DSO:error loading dso:conf_mod.c:285:module=ssl_conf, path=ssl_conf
3069648912:error:0E076071:configuration file routines:MODULE_RUN:unknown module name:conf_mod.c:222:module=ssl_conf

Therefore I went to google a bit and found out that phantomJS is not supported for arm architectures.. (at least from what I found) and that this might have been the problem

I then switched to the chromium renderer binary since phantomJS does not seem to be an option anymore:

root@hostname:/somepath# python webscreenshot -vv --renderer-binary /usr/lib/chromium-browser/chromium-browser 192.168.0.220
webscreenshot.py version 2.91

[DEBUG][General] Options: Namespace(URL='192.168.0.220', ajax_max_timeouts='1400,1800', cookie=None, crop=None, format='png', header=None, http_password=None, http_username=None, imagemagick_binary=None, input_file=None, label=False, label_bg_color='NavajoWhite', label_size=60, log_level='DEBUG', multiprotocol=False, no_xserver=False, output_directory='/somepath/screenshots', port=None, proxy=None, proxy_auth=None, proxy_type=None, quality=75, renderer='phantomjs', renderer_binary='/usr/lib/chromium-browser/chromium-browser', ssl=False, timeout=30, verbosity=2, window_size='1200,800', workers=4)

[INFO][General] '192.168.0.220' has been formatted as 'http://192.168.0.220:80' with supplied overriding options
[+] 1 URLs to be screenshot
[DEBUG][http://192.168.0.220:80] Shell command to be executed
'/usr/lib/chromium-browser/chromium-browser --ignore-ssl-errors=true --ssl-protocol=any --ssl-ciphers=ALL "/usr/local/lib/python3.7/dist-packages/webscreenshot/webscreenshot.js" url_capture=http://192.168.0.220:80 output_file="/somepath/screenshots/http_192.168.0.220_80.png" width=1200 height=800 format=png quality=75 ajaxtimeout=1400 maxtimeout=1800'

[ERROR][http://192.168.0.220:80] Shell command PID 28298 returned an abnormal error code: '1'
[ERROR][http://192.168.0.220:80] Screenshot somehow failed

[+] 0 actual URLs screenshot
[+] 1 error(s)
    http://192.168.0.220:80

Then I tried your first command again but with the chromium renderer binary now so I entered the following:

/usr/lib/chromium-browser/chromium-browser --ignore-ssl-errors=true --ssl-protocol=any --ssl-ciphers=ALL "/usr/local/lib/python3.7/dist-packages/webscreenshot/webscreenshot.js" url_capture=http://192.168.0.220:80 output_file="/somepath/screenshots/http_192.168.0.220_80.png" width=1200 height=800 format=png quality=75 ajaxtimeout=1400 maxtimeout=1800

This was the result:

[28363:28363:0616/104200.582523:ERROR:zygote_host_impl_linux.cc(89)] Running as root without --no-sandbox is not supported. See https://crbug.com/638180.

I therefore first assumed that I had picked the wrong filetype for rendering and went to usr/lib/chromium-browser/ where there are quite some files:

browser_tests_apprtc.runtime_deps  
flash_manifest.json            
master_preferences  
top_domain_generator
bytecode_builtins_list_generator   
headless_lib.pak               
MEIPreload         
torque
cddl                   
icudtl.dat                 
natives_blob.bin    
transport_security_state_generator
chrome_100_percent.pak         
libffmpeg.so                   
neon               
v8_context_snapshot.bin
chrome_200_percent.pak         
libpepflashplayer.so               
plugins        
v8_context_snapshot_v7.bin
chrome-sandbox             
libs                       
protozero_plugin    
xdg-mime
chromium-browser           
libVkICD_mock_icd.so               
resources.pak       
xdg-settings
chromium-browser-v7        
locales                
snapshot_blob.bin
Flash.htm              
make_top_domain_list_for_edit_distance  
swiftshader

I then wanted to ask you if I should use something else than chromium-browser, given the list of files above.

I then took another look at the --no-sandbox error (which was caused because I used root I assumed) and tried to solve it. I went ahead and executed the script WITHOUT being root and this is what happened:

Chromium opens quite a lot of tabs (9 in total) with the first file showing your entire code. The other files all look rather the same, they all say not able to reach xxx (see screenshot).

I must say that is the first time the browser window actually opened! Has never happened before so we are definitely closer to the finish line. Thank you ever so much for your time :) Any ideas on what to do next?

Oh yeah and btw I have tried with other websites than just 192.168.0.220 but still no luck :( Xserver should be in use since I use Raspberry Pi with desktop interface: 

whereis Xserver
Xserver: /usr/share/man/man1/Xserver.1.gz

Furthermore, when using the --no-xserver but leaving everything the same this happens: renderer binary could not have been found in your current PATH environment variable, exiting

Oh yeah totally forgot - here the output with chromium and --debug flag:

/usr/lib/chromium-browser/chromium-browser --debug=true --ignore-ssl-errors=true --ssl-protocol=any --ssl-ciphers=ALL "/usr/local/lib/python3.7/dist-packages/webscreenshot/webscreenshot.js" url_capture=http://192.168.0.220:80 output_file="/somepath/
![IMG_0496](https://user-images.githubusercontent.com/55808530/84758094-4c323b80-afc5-11ea-97ac-9e8c146f38d2.jpeg)
screenshots/http_192.168.0.220_80.png" width=1200 height=800 format=png quality=75 ajaxtimeout=1400 maxtimeout=1800
[3907:4215:0616/112315.977653:ERROR:object_proxy.cc(632)] Failed to call method: org.freedesktop.DBus.Properties.Get: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
[3907:4215:0616/112315.978325:ERROR:object_proxy.cc(632)] Failed to call method: org.freedesktop.UPower.GetDisplayDevice: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
[3907:4215:0616/112315.979235:ERROR:object_proxy.cc(632)] Failed to call method: org.freedesktop.UPower.EnumerateDevices: object_path= /org/freedesktop/UPower: org.freedesktop.DBus.Error.ServiceUnknown: The name org.freedesktop.UPower was not provided by any .service files
Fontconfig error: Cannot load default config file
[3933:3933:0616/112316.741959:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 1 times!
[3933:3933:0616/112316.816580:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 2 times!
[3933:3933:0616/112316.893609:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 3 times!
[3933:3933:0616/112316.932119:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 4 times!
[3933:3933:0616/112316.981122:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 5 times!
[3933:3933:0616/112317.010311:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 6 times!
[3933:3933:0616/112317.039930:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 7 times!
[3933:3933:0616/112317.054077:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 8 times!
[3933:3933:0616/112317.089111:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 9 times!
[3933:3933:0616/112317.105537:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 10 times!
[3933:3933:0616/112317.169144:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 11 times!
[3933:3933:0616/112317.193274:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 12 times!
[3933:3933:0616/112317.222567:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 13 times!
[3933:3933:0616/112317.236980:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 14 times!
[3933:3933:0616/112317.246041:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 15 times!
[3933:3933:0616/112317.259452:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 16 times!
[3933:3933:0616/112317.265308:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 17 times!
[3933:3933:0616/112317.280241:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 18 times!
[3933:3933:0616/112317.285174:ERROR:gl_surface_presentation_helper.cc(259)] GetVSyncParametersIfAvailable() failed for 19 times!
[3907:4252:0616/112324.704247:ERROR:udev_watcher.cc(94)] Failed to begin udev enumeration.

Best regards Marco Leder

maaaaz commented 4 years ago

Therefore I went to google a bit and found out that phantomJS is not supported for arm

Indeed, but it seems that there are some unofficial builds here and here and here that you can try.

I then switched to the chromium renderer binary since phantomJS does not seem to be an option anymore

You forgot to specify the renderer -r option, your tests show that you use the phantomjs renderer (default) while specifying a chrome binary (and of course, it cannot work). Can you try this command:

$ python webscreenshot -vv -r chromium --renderer-binary /usr/lib/chromium-browser/chromium-browser 192.168.0.220
MarcoLeder-zz commented 4 years ago

Oh my god thank you ever so much! Totally my fault, I was not aware that I have to specify both of them :) Thank you for taking the time to solve my issue 👍 Indeed it works now, however with using the chromium render binary I ran into this problem as well.. hope that you are able to add this feature soon !

Best regards Marco Leder