[Neat URL] Overly-aggressive URL cleaning breaks websites; hard to diagnose due to logging ineffective

ronjouch commented 6 years ago

Open https://vividness.live/2015/10/12/developing-ethical-social-and-cognitive-competence/

Expected: site loads fully.
Actual:
- Without Neat URL, site loads fully.
- With Neat URL, site loads partially, due to several requests to css/js resources ending up failing as 400 Bad Request:

I tried to Enable Logging to try to get an idea of which URL parameter might be culprit, but nothing is logged (I think this logging feature is new, maybe it's buggy and sometimes fails to log?). Also, the requests look identical, and quickly looking at them I failed at finding a parameter that is part of NeatURL's blocked parameters set. HALP!

Second use case: https://video.vice.com/fr_ca/video/a-la-rencontre-des-jeunes-skateuses-dafghanistan/59c590a1b316d26a192b11f6 . With NeatURL, video fails to load.

Neat URL 3.0.0 with default Blocked parameters options, Firefox 58.0a1 (2017-10-08)

Aside: I had to go look at options.js on GitHub to grab the default settings; would it make sense to have a Reset default set of blocked parameters button to let users do this easily? EDIT on second thoughts, that looks like a good way for users to shoot themselves in the foot and lose their filters. Maybe a Copy to clipboard default set of blocked parameters, then? (In order to let users do the edition/merging themselves).

ronjouch commented 6 years ago

Ah, looks like a duplicate of #49 and #50, closing.

ronjouch commented 6 years ago

Re-opening, as #50 was closed by its author

gwarser commented 6 years ago

I tried to Enable Logging to try to get an idea of which URL parameter might be culprit, but nothing is logged

Look in "Browser console" (ctrl+shift+j), not in "Web console" (ctrl+shift+k)

23:32:01.005 "Neat URL: https://s0.wp.com/_static/??-eJyFkN0KwjAMhV/IWsS54YX4LP0JW7emLU1G9e3tZHqhMm9CQs7HORxZkjAxMASWOIvk594Fkt5NQHIETspM4nntDdFO/pabmEHwAAhU35gUSy5Vcyd3Y4DwxWof+zddYrbKkux91Mpv+WTwisGKFIk/rr/x1lxVgWCdAl/TBt7CMLUvalmHWsWmTUlVLbROGYhEnehmXGtZuCteDs257U5de2zGB26Gjbs= has been changed to https://s0.wp.com/_static/?%3F-eJyFkN0KwjAMhV%2FIWsS54YX4LP0JW7emLU1G9e3tZHqhMm9CQs7HORxZkjAxMASWOIvk594Fkt5NQHIETspM4nntDdFO%2FpabmEHwAAhU35gUSy5Vcyd3Y4DwxWof+zddYrbKkux91Mpv+WTwisGKFIk%2Frr%2Fx1lxVgWCdAl%2FTBt7CMLUvalmHWsWmTUlVLbROGYhEnehmXGtZuCteDs257U5de2zGB26Gjbs="  background.js:478
23:32:01.014 "Neat URL: https://s2.wp.com/_static/??/wp-content/mu-plugins/genericons/genericons/genericons.css,/wp-content/themes/pub/twentysixteen/style.css?m=1493067000j has been changed to https://s2.wp.com/_static/?%3F%2Fwp-content%2Fmu-plugins%2Fgenericons%2Fgenericons%2Fgenericons.css%2C%2Fwp-content%2Fthemes%2Fpub%2Ftwentysixteen%2Fstyle.css%3Fm=1493067000j"  background.js:478
23:32:01.019 "Neat URL: https://s0.wp.com/_static/??-eJx9kdEOwiAMRX9IrEu26IvxWxhWhgFKoAvu72VuZi4mPHEv3NMWgByEIs/oGXhAhwnC2APnsjEl82JEDyolSDxZFDkocsfiD/ADulEEO2rjE2Rz18glTspIK0yJ7M0CG/DEy+FX1KpqJGFJSTbkd0Y8rDSxhkbsLeki9ecam61BUs3Fexk3VYsHSrxMAmmQce61rjVqfZUyDu3NH7R+zNCCttRLOwdu7tp0p65rLm1zfr4Bcuqq1Q== has been changed to https://s0.wp.com/_static/?%3F-eJx9kdEOwiAMRX9IrEu26IvxWxhWhgFKoAvu72VuZi4mPHEv3NMWgByEIs%2FoGXhAhwnC2APnsjEl82JEDyolSDxZFDkocsfiD%2FADulEEO2rjE2Rz18glTspIK0yJ7M0CG%2FDEy+FX1KpqJGFJSTbkd0Y8rDSxhkbsLeki9ecam61BUs3Fexk3VYsHSrxMAmmQce61rjVqfZUyDu3NH7R+zNCCttRLOwdu7tp0p65rLm1zfr4Bcuqq1Q=%3D"  background.js:478
23:32:01.026 "Neat URL: https://s0.wp.com/_static/??-eJyFj1sKwkAMRTfkOBRaxQ9xLX3ENuO8nGSsdfWmoB/WohAISU5ubvQYFfrW5g5IG4lrhjS90tbQRv8ClMM+1Qxbh/4Nt8EzeJ5ZFxq0oDJBqnvpidA5rHAxEDsgEmhl+mkJ/Q1h/IsZ4Fi3F5WA8PGl6rKKNvfoSSew8kCnZg+LarnFA4hLHXOjeZTGRHhnAD9fH9jZShZO7liUh6qoynK/M0/gsoMY has been changed to https://s0.wp.com/_static/?%3F-eJyFj1sKwkAMRTfkOBRaxQ9xLX3ENuO8nGSsdfWmoB%2FWohAISU5ubvQYFfrW5g5IG4lrhjS90tbQRv8ClMM+1Qxbh%2F4Nt8EzeJ5ZFxq0oDJBqnvpidA5rHAxEDsgEmhl+mkJ%2FQ1h%2FIsZ4Fi3F5WA8PGl6rKKNvfoSSew8kCnZg+LarnFA4hLHXOjeZTGRHhnAD9fH9jZShZO7liUh6qoynK%2FM0%2FgsoMY="  background.js:478
23:32:01.782 "Neat URL: https://s2.wp.com/_static/??/wp-content/js/jquery/jquery.autoresize.js,/wp-content/mu-plugins/highlander-comments/script.js?m=1479964158j has been changed to https://s2.wp.com/_static/?%3F%2Fwp-content%2Fjs%2Fjquery%2Fjquery.autoresize.js%2C%2Fwp-content%2Fmu-plugins%2Fhighlander-comments%2Fscript.js%3Fm=1479964158j"  background.js:478
23:32:01.790 "Neat URL: https://s2.wp.com/_static/??-eJx9jl1uAjEMhC/U4AVE6UvVo1Qh67DeTZwQO/zcnlCpqKJSnjyyv5kxXLJxiRVZYRYY8UwO83U1yxv8OcVqcqhHYoFACwqcKlacLI8ByyusE8aG5HoAvbTFTeiqiPwokIWyCcSL8clVMZ7+dfXsvrJTSiydB+0Yic3BFohWFEtTJp2xFBpb7HPXS/jpeCQ8VY9eSCKq2a4G+CZ24FOJHT4nUeODpQIy2UJ8/J3N9BU/17th8/6x3m+G+Q6QS5ry has been changed to https://s2.wp.com/_static/?%3F-eJx9jl1uAjEMhC%2FU4AVE6UvVo1Qh67DeTZwQO%2FzcnlCpqKJSnjyyv5kxXLJxiRVZYRYY8UwO83U1yxv8OcVqcqhHYoFACwqcKlacLI8ByyusE8aG5HoAvbTFTeiqiPwokIWyCcSL8clVMZ7+dfXsvrJTSiydB+0Yic3BFohWFEtTJp2xFBpb7HPXS%2FjpeCQ8VY9eSCKq2a4G+CZ24FOJHT4nUeODpQIy2UJ8%2FJ3N9BU%2F17th8%2F6x3m+G+Q6QS5ry="  background.js:478
23:32:02.316 "Neat URL: https://s0.wp.com/_static/??-eJyFj1sKwkAMRTfkOBRaxQ9xLX3ENuO8nGSsdfWmoB/WohAISU5ubvQYFfrW5g5IG4lrhjS90tbQRv8ClMM+1Qxbh/4Nt8EzeJ5ZFxq0oDJBqnvpidA5rHAxEDsgEmhl+mkJ/Q1h/IsZ4Fi3F5WA8PGl6rKKNvfoSSew8kCnZg+LarnFA4hLHXOjeZTGRHhnAD9fH9jZShZO7liUh6qoynK/M0/gsoMY has been changed to https://s0.wp.com/_static/?%3F-eJyFj1sKwkAMRTfkOBRaxQ9xLX3ENuO8nGSsdfWmoB%2FWohAISU5ubvQYFfrW5g5IG4lrhjS90tbQRv8ClMM+1Qxbh%2F4Nt8EzeJ5ZFxq0oDJBqnvpidA5rHAxEDsgEmhl+mkJ%2FQ1h%2FIsZ4Fi3F5WA8PGl6rKKNvfoSSew8kCnZg+LarnFA4hLHXOjeZTGRHhnAD9fH9jZShZO7liUh6qoynK%2FM0%2FgsoMY="  background.js:478
23:32:03.002 "Neat URL: https://s2.wp.com/_static/??/wp-content/js/jquery/jquery.autoresize.js,/wp-content/mu-plugins/highlander-comments/script.js?m=1479964158j has been changed to https://s2.wp.com/_static/?%3F%2Fwp-content%2Fjs%2Fjquery%2Fjquery.autoresize.js%2C%2Fwp-content%2Fmu-plugins%2Fhighlander-comments%2Fscript.js%3Fm=1479964158j"  background.js:478
23:32:03.150 "Neat URL: https://s2.wp.com/_static/??-eJx9jl1uAjEMhC/U4AVE6UvVo1Qh67DeTZwQO/zcnlCpqKJSnjyyv5kxXLJxiRVZYRYY8UwO83U1yxv8OcVqcqhHYoFACwqcKlacLI8ByyusE8aG5HoAvbTFTeiqiPwokIWyCcSL8clVMZ7+dfXsvrJTSiydB+0Yic3BFohWFEtTJp2xFBpb7HPXS/jpeCQ8VY9eSCKq2a4G+CZ24FOJHT4nUeODpQIy2UJ8/J3N9BU/17th8/6x3m+G+Q6QS5ry has been changed to https://s2.wp.com/_static/?%3F-eJx9jl1uAjEMhC%2FU4AVE6UvVo1Qh67DeTZwQO%2FzcnlCpqKJSnjyyv5kxXLJxiRVZYRYY8UwO83U1yxv8OcVqcqhHYoFACwqcKlacLI8ByyusE8aG5HoAvbTFTeiqiPwokIWyCcSL8clVMZ7+dfXsvrJTSiydB+0Yic3BFohWFEtTJp2xFBpb7HPXS%2FjpeCQ8VY9eSCKq2a4G+CZ24FOJHT4nUeODpQIy2UJ8%2FJ3N9BU%2F17th8%2F6x3m+G+Q6QS5ry="  background.js:478

ronjouch commented 6 years ago

Look in "Browser console" (ctrl+shift+j), not in "Web console" (ctrl+shift+k)

@gwarser thanks! Didn't know "Browser Console" was a thing when I read it in the option sub-title 😄. That works, and I see NeatURL's logging 👍.

Smile4ever commented 6 years ago

What I did is the following:

I fixed a few issues with the usage of the URL() constructor (used since 3.0.0)
I set the default type for content filtering to main_frame. This means only visible requests will be cleaned (like clicking a link, not a page fetching a resource). The exception to this rule is the anti-tracking list which doesn't work on main_frame requests, but works on all other types of requests. Having only visible URLs cleaned makes the change less likely to break the page. If you want the old behaviour back, you can go to the options and add all of the request types listed on this page: https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest/ResourceType

These changes are available in Neat URL 3.1.0.

crssi commented 6 years ago

@Smile4ever did the old behavior covered all request types listed on that link, or just some of those?

Smile4ever commented 6 years ago

@crssi It did cover all request types.

ronjouch commented 6 years ago

@Smile4ever fix tested and confirmed, thanks for the fast response and continued work on NeatURL 🙂.

Smile4ever / firefoxaddons

[Neat URL] Overly-aggressive URL cleaning breaks websites; hard to diagnose due to logging ineffective #52