Parsing issues with 0.2.1

[yoshimo]

so i still have trouble with the page mentioned in #10 but this time i will try to supply more detailed information.

System is a Kubuntu 14.04 64bit computer, running ruby 1.9.1 with german locale activated.

New testfile is available at: http://fbe.am/tOF

Again i can't extract scripts from the page, but the more intresting part is that jsdetox fails as soon as i try to execute the file, no matter if trace eval and/or "do not trace variable values" is checked .

We get " Unexpected token ILLEGAL (Line 3)" JSdetox trace is:

at :3:25,/home/me/Downloads/jsdetox/jsdetox/lib/framework/jsengine_v8.rb:211:in execute',/home/me/Downloads/jsdetox/jsdetox/app/controllers/backend.rb:32:inblock (2 levels) in <top (required)>',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:569:in call',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:569:inblock in route',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:51:in []',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:51:inblock (3 levels) in process_destination_path',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:876:in route_eval',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:51:inblock (2 levels) in process_destination_path',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:51:in catch',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:51:inblock in process_destination_path',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:25:in instance_eval',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:25:inprocess_destination_path',(eval):134:in block (2 levels) in inject_root_methods',(eval):124:incatch',(eval):124:in block in inject_root_methods',/var/lib/gems/1.9.1/gems/http_router-0.10.2/lib/http_router/node/root.rb:92:in[]',/var/lib/gems/1.9.1/gems/http_router-0.10.2/lib/http_router.rb:119:in block in call',/var/lib/gems/1.9.1/gems/http_router-0.10.2/lib/http_router.rb:119:incatch',/var/lib/gems/1.9.1/gems/http_router-0.10.2/lib/http_router.rb:119:in call',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:919:inroute!',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/application/routing.rb:909:in dispatch!',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:794:inblock in call!',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:946:in block in invoke',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:946:incatch',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:946:in invoke',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:794:incall!',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:780:in call',/var/lib/gems/1.9.1/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:225:incontext',/var/lib/gems/1.9.1/gems/rack-1.5.2/lib/rack/session/abstract/id.rb:220:in call',/var/lib/gems/1.9.1/gems/sass-3.3.13/lib/sass/plugin/rack.rb:54:incall',/var/lib/gems/1.9.1/gems/rack-1.5.2/lib/rack/head.rb:11:in call',/var/lib/gems/1.9.1/gems/rack-1.5.2/lib/rack/methodoverride.rb:21:incall',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/reloader.rb:250:in call',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/logger.rb:388:incall',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/showexceptions.rb:21:in call',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:1417:inblock in call',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:1499:in synchronize',/var/lib/gems/1.9.1/gems/sinatra-1.3.6/lib/sinatra/base.rb:1417:incall',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/router.rb:83:in block in call',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/router.rb:76:ineach',/var/lib/gems/1.9.1/gems/padrino-core-0.10.7/lib/padrino-core/router.rb:76:in call',/var/lib/gems/1.9.1/gems/rack-1.5.2/lib/rack/handler/webrick.rb:60:inservice',/usr/lib/ruby/1.9.1/webrick/httpserver.rb:138:in service',/usr/lib/ruby/1.9.1/webrick/httpserver.rb:94:inrun',/usr/lib/ruby/1.9.1/webrick/server.rb:191:in `block in start_thread'

Once i received: [2014-08-05 19:43:46] ERROR bad Request-Line `\x16\x03\x01\x00�\x01\x00\x00�\x03\x03E\x1EN8%Z��:W\a;/~'.

and there are some minor warnings WARN Could not determine content-length of response body. Set content-length of the response or set Response#chunked = true

How would i a) extract scripts? b) get past the "illegal" warning?

If there is anything else than ruby version or OS that you need to help me solve it, just shout

[svent]

Thanks for your bug report - I tried to reproduce it, but it works on my two test systems (one system is a fresh Linux Mint 17 (based on Ubuntu 14.04.) with a default english locale).

I also managed to execute the script on my system. It does contain some HTML comment lines (starting with "<--") that were included in the script tags (browsers do ignore these). These have to be removed to execute the script in JSDetox (script tag 53 and 58 were affected).

This bug might have something to do with the encoding (as the sample file does contain german umlauts). I will set up a Kubuntu system with a german locale tomorrow and try to reproduce this bug in that system.

[svent]

I now tested this on a fresh installation of Kubuntu 14.04 64 Bit with a german locale. Unfortunately, I was not able to reproduce the bug.

Did you upgrade that system from an earlier Kubuntu relase? I was wondering because you use Ruby 1.9.1, but Kubuntu 14.04 seems to use version 1.9.3p484 by default. Do you use the firefox browser that comes with Kubuntu?

[yoshimo]

No it was a clean install to test wether a new wifi driver works during install.

I figured out why the ruby install is "outdated", i followed the instructions at http://www.relentless-coding.com/projects/jsdetox/install but chose the Installation on Linux Mint 13 / Ubuntu 12.04 instead.

Browser this time is firefox nightly from http://ppa.launchpad.net/ubuntu-mozilla-daily/ppa/ubuntu In the other issue it was firefox stable so i doubt that is the issue here.

[yoshimo]

@svent is there anything else i could try to solve this problem? I am just trying to figure out how this website creates it's banners , can't be that difficult

[svent]

As I cannot reproduce the bug it is difficult to analyze this issue - I guess the best solution would be to install one of the latest recommended Linux distributions into a virtual machine and use that one to analyze the file.