compatibility with Waterfox - strict_min_version 57.0 and 60.0

[grahamperrin]

https://github.com/claustromaniac/httpz/commit/d7d88560f8dbb707720fe296beb574989fa3e0ff#diff-051a8a7bcae8db1e4cee8eb09b52e619L5

https://github.com/claustromaniac/httpz/commit/b3c55c9a7bf4e6afce1f785c615b8dc1abae53b5#diff-051a8a7bcae8db1e4cee8eb09b52e619L5

Loosely speaking: should users of Waterfox Classic e.g. 56.2.14 expect some features of 0.6.0 and greater to be simply non-functional?

https://addons.mozilla.org/addon/httpz/versions/

tl;dr Waterfox 56.0 was based on Firefox 56.0.2 and to be clear, I'm not suggesting that (#32) Waterfox Classic should be another supported browser.

I'm just curious about functionality. Background: https://www.reddit.com/comments/dahnh0/-/f1te63v/

TIA

[claustromaniac]

A few of the times that I increased the strict_min_version was because it sometimes takes Mozilla longer to implement APIs in Firefox for Android (e.g. some API in FF 67 for desktops may be available only from 68 onwards on mobile), and AMO wouldn't have let me proceed with the release if I hadn't (because HTTPZ supports both desktop and mobile). In other words: the actual minimum Firefox version required for HTTPZ to work correctly on desktop does not match the value of strict_min_version (it is an older one).

Still, I'm not saying that HTTPZ should therefore work perfectly in Waterfox. I don't know that because I lack the time and motivation to support forks (so I haven't looked at that, and I won't).

[claustromaniac]

It should now be possible to use HTTPZ in Firefox 56.0 (desktop) and its forks without having to jump through hoops. Thanks for bringing this to my attention.

[grahamperrin]

Thanks,

not saying that HTTPZ should therefore work perfectly in Waterfox. … lack the time and motivation to support forks (so I haven't looked at that, and I won't).

That's fair.

I just now realised, a significant issue (sorry) that might cause you to be stricter, again, with the strict_min_version:

• user-set preferences at about:addons seem be be either volatile, or not truly set.

This, for example:

☑ Automatic mode

If I browse away from, then back to, the preferences page: the box is no longer checked.

---

I vaguely recall something like this (issue affecting preferences) being fixed, recently, for compatibility of a different extension. If I can find a record and if it seems relevant, I might raise a PR.

[claustromaniac]

I just now realised, a significant issue (sorry) that might cause you to be stricter, again, with the strict_min_version:

I did not lower the strict_min_version in manifest.json. I found an option on AMO that allows me to give would-be users of HTTPZ the ability to install the extension (from AMO) in Firefox 56.0+ without problems. That's all I did.

• user-set preferences at about:addons seem be be either volatile, or not truly set

HTTPZ is not meant to change anything in about:addons. Edit: I seem to have misread that as about:config. The choice of words is probably what led me to that, since I'm more used to refer to those as options, and to the ones in about:config as preferences. Anyway...

If I browse away from, then back to, the preferences page: the box is no longer checked.

Did you make sure to click the Save button at the bottom?

[grahamperrin]

Thanks,

HTTPZ is not meant to change anything in about:addons (for now).

Sorry, I didn't explain clearly. I mean, preferences for HTTPZ alone. Preferences as they appear after addition of the extension to a refreshed profile:

Did you make sure to click the Save button at the bottom?

That is my type of mistake :-) however, after checking the box in this case –

☑ Remember secure sites

– then clicking Save, if I close the tab then review preferences, the box is no longer checked.

[claustromaniac]

I know I already said I don't intend to offer support for forks, but I admit I'm a bit curious now.

In that screenshot it looks like the options page script (options.js) is crashing. If you open the browser console (Ctrl+Shift+J) and then the options page, you should see an error there (or many errors) related to options.js. Is that the case?

[grahamperrin]

Thanks!

SyntaxError: missing ( before catch

https://github.com/claustromaniac/httpz/commit/a102b0cfa879df6581378d3a78b7f89c345304b0#diff-acd9afab85ff2edbc9f5ac6500b57aabR108

[claustromaniac]

Should (hopefully) work in the version 0.10.0b2 I just uploaded.

🤞

[grahamperrin]

Thanks. Preferences GUI looks OK now, at a glance.

I found a couple of sites that don't work in automatic mode so I guess, this feature truly requires something more than Waterfox Classic.

http://www.macattorney.com/upos.html for example, an initial visit to the site presents the (browser) alert for an SSL_ERROR_BAD_CERT_DOMAIN. A second visit to the same http URL loads the http page.

[claustromaniac]

I narrowed down the cause and (hopefully) fixed it in 0.10.0b3 0.10.0b4

It was the optional loadReplace flag in tabs.update(). I made it so it is never used if not supported by the browser.

[grahamperrin]

Ha you're ahead of me with https://github.com/claustromaniac/httpz/commit/f76700e06cd4cedbb6c27299b90f86de0e453df8#diff-a01f5c2208463c5a915a29e6862dc2a6L40 :-) let me try b4 …

[grahamperrin]

Yay! Looking great. I reckon, it's properly fixed.

---

For reference only, in alphabetical order, the shortlist of URLs with which I tested:

1. http://addonconverter.fotokraina.com/
2. http://cosmicshambles.com/
3. http://meyerweb.com/eric/thoughts/2017/03/07/welcome-to-the-grid/
4. http://spaceweather.com/
5. http://thegearcalculator.appspot.com/
6. http://www.keithfullertonwhitman.com/bio
7. http://www.macattorney.com/upos.html
8. http://www.xorosho.com/xoroshaya_muzika/52710-sven-grunberg-hingus-1981-ambient.html

Here, test 5 whitelist results vary – at a glance the site appears to behave better with Waterfox Classic 2019.10 (20191103135811) than with Mozilla Firefox 70.0.1 (64-bit) (20191112090246) on FreeBSD-CURRENT. However my everyday profiles for both browsers are heavily extended, so I suspect an extensions conflict. If more careful test results suggest otherwise, in Firefox, I can raise a separate issue.

I didn't attempt to whitelist any other.

Tests 7 and 8: good for testing automated and non-automated modes. And if you like the sound of 8, you'll love https://www.youtube.com/watch?v=r_yWrMuF1RY

---

After the next release hits AMO, I'll be happy to post a link to https://www.reddit.com/r/waterfox/.

Any objection? Understood (from the pre-release notes):

Partial FF56 compatibility. Still not officially supported. Just an experiment

Thanks again!

[claustromaniac]

Thanks for reporting, and for the help!

Any objection? Understood (from the pre-release notes)

What I mean when I say that FF56 is still not officially supported is mostly that I won't test the extension on FF56 before each release to make sure it works well. Also, it is partial compatibility because some functionality is lost (for example, without the loadReplace flag in FF56, sometimes duplicate entries are left in the browser's navigation history).

In other words: I just don't want to make promises regarding this.

You're free to share your experiences as a user though. :+1:

[grahamperrin]

Done: https://redd.it/dzl75h

[grahamperrin]

Incidentally,

loadReplace

AFAICT support for this was gained with Waterfox 56.2.0, https://github.com/MrAlex94/Waterfox/commit/bad1c19a1b6b85b1129d1da65b3fa3ea20f893e2

[claustromaniac]

This is the original commit for FF57.

Before then, loadReplace did not exist. I don't know which versions of Waterfox are meant to support it, but that should not matter, because HTTPZ now uses loadReplace whenever available (it determines its availability on the fly)

[grahamperrin]

I don't know which versions of Waterfox are meant to support it, but that should not matter,

Understood, thanks.

From the range of tags, I suspect that 56.2.0 was the first release to support loadReplace … on the other hand, there's the announcement linked from (unofficial) https://github.com/grahamperrin/Waterfox/wiki/Archive-of-change-logs,-announcements-and-downloads#5620 (… Tested a new way to keep Waterfox components up to date …)

---

@MrAlex94 please, is the 56.2.0 tag on commits such as https://github.com/MrAlex94/Waterfox/commit/bad1c19a1b6b85b1129d1da65b3fa3ea20f893e2 a sure sign that the commits truly applied to release 56.2.0?

TIA

[grahamperrin]

0.11.0b test results – Waterfox Classic 2019.10 (20191103135811), FreeBSD-CURRENT

Three browser defaults:

• browser.cache.disk.enable true
• browser.cache.memory.enable true
• browser.sessionhistory.max_total_viewers -1

– and HTTPZ in its default automatic mode.

http://meyerweb.com/eric/thoughts/2017/03/07/welcome-to-the-grid

Page action behaviours all good.

http://thegearcalculator.appspot.com/

No page action.

https://thegearcalculator.appspot.com/

No page action.

For one or both of these thegearcalculator URLs, earlier I did something ticklish that caused the page action to appear (and persist, IIRC) but right now I can't reproduce the tickle.

http://vpri.org/

No page action.

https://vpri.org/

One uninterrupted presentation of the browser error page, without redirection to http. Then a few automated redirections to http (and no page action). Then again, uninterrupted presentation of the browser error page …

Switch to manual, forget ignored sites, clear whitelist, save, https://vpri.org/ presents the browser error page (not the HTTPZ error page).

[grahamperrin]

… right now I can't reproduce the tickle. …

OK, this seems to be closer to reproducible:

1. https://thegearcalculator.appspot.com/ (no page action)
2. manually change from https to http, go
3. sometimes there's reappearance of the https content, sometimes there's the (expected) http content; either way there's the page action and AFAICT after it's present, it persists (through repeated whitelisting and de-listing).

[grahamperrin]

0.11.0b3

http://www.youronlinechoices.com/uk/your-ad-choices opened in a new tab (the default) from https://adssettings.google.com/authenticated) does present a page action.

http://www.youronlinechoices.com/uk/your-ad-choices opened in a new tab from this comment does not; the result is http

about:addons

Check boxes are not apparent:

– but the layout is tidy 👍 compared to what's currently seen with e.g. Firefox 71.0.

[claustromaniac]

0.11.0b5 should fix the checkboxes' visibility, but I'm pretty sure they won't look the same as in FF71.

http://www.youronlinechoices.com/uk/your-ad-choices opened in a new tab from this comment does not; the result is http

I can't reproduce that. Do you get any errors in the console?

[claustromaniac]

Actually, the fact that you can't see the checkboxes now is rather strange, because the same checkboxes were visible in 0.9.4... There's something I'm missing.

Edit: I figured it out. I have a few options. I just have to assess their pros and cons before doing anything. I'll get back to this later.

[claustromaniac]

Please try 0.11.0b6 when you can.

[grahamperrin]

Confirmed, 0.11.0b6 fixes the about:addons UI. Thanks!

[grahamperrin]

Hi, I seem to no longer get good results for part of the test set at https://github.com/claustromaniac/httpz/issues/37#issuecomment-555361647.

Recalling your https://github.com/claustromaniac/httpz/issues/37#issuecomment-555534961 I kept quiet about most such results whilst beta testing towards 0.11.0.

https://addons.mozilla.org/addon/httpz/versions/0.11.0/updateinfo/ and https://addons.mozilla.org/addon/httpz/versions/ noted with thanks,

… Also, fixed a number of minor compatibility issues with older versions of Firefox …

– there's no longer a yellow alert re: incompatibility for 0.11.0 added to Waterfox Classic 2019.10.

I am, however, puzzled by this:

https://github.com/claustromaniac/httpz/blob/4dfe2cb6bd3780fe078fb4fa94e858fb28a9454e/src/manifest.json#L2-L7

… or maybe I should say, I'm puzzled by Waterfox Classic not showing a yellow alert for an extension with this strict_min_version

[grahamperrin]

Does it help to compare with these sections of code? From within https://robwu.nl/crxviewer/?crx=https%3A%2F%2Faddons.mozilla.org%2Ffirefox%2Fdownloads%2Ffile%2F3460706%2Fmalwarebytes_browser_guard-2.1.4-fx.xpi (for Malwarebytes Browser Guard, which "Works with firefox 57.0 and later"):

    "browser_specific_settings": {
        "gecko": {
            "strict_min_version": "57.0"
        }
    }

    "browser_specific_settings": {
        "gecko": {
            "id": "org_malwarebytes_browserguard@malwarebytes.org",
            "strict_min_version": "57.0"
        }
    }

– for this version of this extension, I do get the yellow alert at about:addons

[claustromaniac]

Hi, I seem to no longer get good results for part of the test set at https://github.com/claustromaniac/httpz/issues/37#issuecomment-555361647.

I just tested that list of sites in FF56. Here are my results:

Number	Site	Error	Redirect	STS	Page Action	Can exclude	Problem?
1	http://addonconverter.fotokraina.com/	No	No	No	Yes	Yes	No
2	http://cosmicshambles.com/	No	No	Yes	Yes	Yes*	Yes
3	http://meyerweb.com/eric/thoughts/2017/03/07/welcome-to-the-grid/	No	No	No	Yes	Yes	No
4	http://spaceweather.com/	No	Yes	No	Yes	Yes*	Yes
5	http://thegearcalculator.appspot.com/	No	No	No	Yes	Yes	No
6	http://www.keithfullertonwhitman.com/bio	No	No	No	Yes	Yes	No
7	http://www.macattorney.com/upos.html	Yes	No	No	No	No	No
8	http://www.xorosho.com/xoroshaya_muzika/52710-sven-grunberg-hingus-1981-ambient.html	Yes	No	No	No	No	No

* can exclude even after adding the site to the list of exclusions

All tests passed, but 2 and 4 made me notice a current limitation of HTTPZ. It is OK to show the page action after upgrading requests to HTTPS-only sites, but HTTPZ should not allow users to add those to the list of exclusions, because they do not support HTTP anyway. To avoid complicating things, I think I'm going to prevent HTTPZ from displaying the page action in those cases.

Edit: actually, I already had a check here for the server-initiated redirection scenario, but it does not cover STS, and it apparently does not work as intended either: https://github.com/claustromaniac/httpz/blob/f74fb71abb3d259b1d831f3e80f9f6618ff7db7e/src/bg/webRequest.js#L139-L149

Edit 2: I figured out the issue. This snippet works as intended, but not in FF56, because the page action is not automatically hidden in that version. So, I just have to make the extension hide it.

there's no longer a yellow alert re: incompatibility for 0.11.0 added to Waterfox Classic 2019.10.

I don't even know what is the yellow alert in question. Care to elaborate?

[grahamperrin]

Hi, compare redirect results for 1, 2, 3, 5 and 6 above with results in Firefox 71.

IIRC for 56-based browsers, the desired redirects were more likely to succeed with an earlier release or beta of the extension.

HTH, sorry to throw this spanner in the works

[claustromaniac]

I suppose you misunderstood what I meant by Redirect in my table. Redirect refers to server-initiated redirections from HTTPS to HTTP HTTP to HTTPS. Only the 4th site in the list does this; it does not support HTTP (only HTTPS).

With the current version of HTTPZ I get exactly the same results in FF71 as in FF56.

Your mileage may vary. Let me know if that is the case.

[claustromaniac]

For completeness' sake, I will add that Error in that table refers to the server's response when HTTPZ attempts to upgrade requests (only 7 and 8 resulted in errors), and STS stands for Strict Transport Security. Servers that implement STS do not support HTTP, only HTTPS (in this case, only the second item in the list implements STS).

[grahamperrin]

Your mileage may vary.

Ha! It did a short while ago, but not now.

Now I see page action icons for all five, even with my very heavily extended profile. Consistent, at a glance, with what's seen in Firefox 71.0.

Previously I saw none and TBH I can't recall looking to the left (to tell whether there was redirection from HTTP to HTTPS). Let's ignore this earlier result, I was being lazy/non-methodical at the time.

Thanks again!

Re: the yellow alert, I might follow up at the weekend.

Postscript

Three frames from the screen recording of the lazy test session that probably caused me to imagine a problem:

(click)

  

There was simply a long wait – twenty-eight seconds – between appearance of Done (in the status bar) and appearance of the page action. I didn't notice the page action until after a view of the recording.

Not a bug. A casualty of my choice of extensions, some of which require single-process mode.

[claustromaniac]

Now I see page action icons for all five, even with my very heavily extended profile. Consistent, at a glance, with what's seen in Firefox 71.0.

click to show


Let's ignore this earlier result, I was being lazy/non-methodical at the time.

Nah, it was very helpful. Thanks to that I ended up discovering more defects than I care to admit.

Thank you again for testing and reporting everything.

[grahamperrin]

Looking great.

From https://github.com/claustromaniac/httpz/issues/37#issuecomment-560086337 (now hidden, resolved):

http://www.youronlinechoices.com/uk/your-ad-choices opened in a new tab from this comment does not (present a page action); the result is http

That's no longer true. Thanks for the fix.

[grahamperrin]

Bonus: the screen recording at thegearcalculator.appspot.com - Firefox - Malwarebytes Forums

Interesting interactions of the site with:

• pervasive tracking protection enabled (not a default) in Waterfox Classic 2019.10
• malware protection in Malwarebytes Browser Guard 2.1.4.

Feel free to describe it as not a bonus, if my choice of test site exposed you to malware 🤯

[claustromaniac]

Feel free to describe it as not a bonus, if my choice of test site exposed you to malware 🤯

Thanks for the info. And don't worry (about me) because, whether that's malware or not, I was not exposed, because I did something not ideal from a methodology standpoint: I tested with uBlock Origin ON (and blocked JavaScript).

In the future, unless you're facing problems with a specific site and you suspect HTTPZ to be the culprit, you can use http://example.com/ or http://example.org/ for testing.

Other sites I often use for testing are: http://http.badssl.com/ (redirects from HTTPS to HTTP), and http://error.net/ (errors out over HTTPS).