Erratic whitelisting and icon behaviour

[in4u]

Version: 0.7.0b2

Settings:

{
    "ignorePeriod": 0,
    "ignored": {},
    "incognitoWhitelist": {},
    "knownSecure": {},
    "rememberSecureSites": false,
    "whitelist": {}
}

Browser: Firefox 66.0.3 (64-bit)

Sample URL: example.com

This is in continuation of the discussion which started here. I have come across the following issues while testing the latest beta version:

1. Sites whitelisted in incognito mode do not appear in the exported settings file. Also, the whitelisting doesn't last once the tab/window is closed (if it worked in the first place).

2. The HTTPZ icon for an http site redirected to https is not always enabled. Sometimes it gets enabled after either reloading the tab, restarting browser, clearing cache or refreshing after a minute, etc. The result also varies depending on whether / is present at the end or www is present at the beginning of URL.

3. Sites whitelisted in normal mode behave erratically. Sometimes the whitelist is not respected and only kicks in after trying the random tricks mentioned above. Again, the presence of / at the end or www at the beginning of URL may affect results.

4. The Import option does not work and throws up Error. Invalid file (?) message.

[claustromaniac]

1. fixed in 158a018
2. I couldn't reproduce this the previous time I tried to, but I tried a few more things and I finally could. Still, if I enter http:// before the rest of the address, the extension seems to works consistently. Also, I tried disabling autocomplete in FF64 just to see if it could be related to that, and even without entering the prefix it seemed to work consistently afterwards (the pref to disable autocomplete no longer exists since FF65, that's why I tried that in FF64). I'm thinking your tests could be tainted by some behavior of the browser itself (example.com is not an actual URL after all, just a hostname). Anyway, I'll test some more, but I suspect I may end up having to ditch the page action for a toolbar icon.
3. Have not yet tested this, but may also be related to some behavior of the browser itself.
4. I silently fixed that one yesterday in 791699b

[claustromaniac]

I tested a bit more and point number 2 happens even in version 0.6.1.

I'm pretty sure now your tests are tainted. The extension is not meant to show the icon when the browser visits a site over https on its own. When you enter example.com or example.com/ in the urlbar, Firefox has to fill the rest of the address to make the request, and if for whatever reason it fills it with https:// at the start, then the result of that test is meaningless.

[in4u]

some behavior of the browser itself

Agreed. I even noticed difference in Firefox for Android 66.0.2 where HTTPZ icon disappearing problem didn't come up that often (if at all).

I tested a bit more and point number 2 happens even in version 0.6.1

Yeah, I had mentioned earlier "...But this is a pre-existing problem which has nothing to do with your latest update. I don't use the icon so it has never bothered me."

The extension is not meant to show the icon when the browser visits a site over https on its own.

I have not experienced this behaviour. This is something new you have found. In my case, the icon often doesn't show up when it should (i.e. redirecting from http to https, sometimes even when http is typed manually) and not the other way round where it shows up when it shouldn't (i.e. visiting https).

I suspect I may end up having to ditch the page action for a toolbar icon.

If it works, this may be a better solution indeed.

[claustromaniac]

This is something new you have found.

You misunderstood, probably by my fault because I tend to be too darn concise. Heck, english is not even my native language. Anyway, what I meant to say is that as far as I can tell, the extension is not misbehaving in that regard. Those times when you entered example.com or example.com/ in the urlbar and got to https://example.com/ without HTTPZ showing the icon, it was because the one that took you to https://example.com/ was Firefox (not the extension). If you want to confirm my theory, disable the extension and repeat the exercise: if you ever land in https://example.com/, then HTTPZ has been doing what it was meant to be doing the whole time.

[in4u]

Yeah, I figured what you were trying to say and edited my comment to add that "sometimes even when http is typed manually" but you replied just before my edit!

I have one more suggestion. If you are going for the toolbar icon route then consider adding "Disable HTTPZ" option to the dropdown menu.

[claustromaniac]

sometimes even when http is typed manually

I can't reproduce that, but if you ever manage to find a way to do so reliably, please let me know. I may not be able to do anything about it, but it would be good to know even if it was a bug in the browser or something like that. Note that it could be something as simple as the browser using the in-memory cache, though (or fastback cache). When you use the back or forward navigation actions, the browser by default loads the data from memory and doesn't start a network request at all (and if there is no request, the extension doesn't have anything to do).

If you are going for the toolbar icon route then consider adding "Disable HTTPZ" option to the dropdown menu.

For now, there is no reason for me to go that route. As far as I can tell, the extension works as it was designed to work, and there is no point in showing an icon when the extension is doing nothing.

Currently, whitelisting sites means ignoring requests. HTTPS requests are ignored by design, so it doesn't make sense to show an icon to whitelist a site that the browser is connecting to securely in the first place. I could instead change the design and dictate that whitelisting means always redirecting to HTTP, but then a set of new problems will appear. For one, what should the extension do when a user whitelists a site that the servers will redirect to HTTPS anyway? (and servers that do this are more common than it sounds). Not to mention that if something ever goes wrong (Mozilla devs introduce a bug, the APIs change, whatever), I don't want the extension to erroneously downgrade a secure request.

Don't get me wrong, I really appreciate all your feedback, but the only problem we've confirmed so far (other than the bugs you reported and I already fixed) is that you're not getting from the extension the results that you expected.

[in4u]

it could be something as simple as the browser using the in-memory cache, though (or fastback cache)

Extremely possible and likely.

For now, there is no reason for me to go that route. As far as I can tell, the extension works as it was designed to work, and there is no point in showing an icon when the extension is doing nothing.

I can see how the icon may be of some use: 1) Icon/color changes to indicate if HTTPZ successfully redirected http to https, or changes to indicate if redirect resulted in an error causing a rollback to http, or stays unchanged if HTTPZ didn't need to handle the original https request. 2) The icon dropdown menu could have some options like Add to / Remove from whitelist (this is already present but hide it if not applicable, i.e. if HTTPZ didn't handle the request), Disable HTTPZ (practically equivalent to disabling the addon, i.e. letting requests pass untouched) and Open settings (shortcut to the addon's settings page).

the only problem we've confirmed so far (other than the bugs you reported and I already fixed) is that you're not getting from the extension the results that you expected.

Oh no, quite the contrary. I am more than satisfied with the extension and it works just as I expect it to! :)

I personally don't use the icon or whitelisting feature at all and was merely providing an observation. I only went along with testing and reporting since you prodded for feedback and I thought it may benefit someone for whom this might be a problem. But hey, if nobody else finds this problematic then no point in wasting our time.

[claustromaniac]

Well, I sure could add a toolbar icon for those things you mentioned, but maybe not in this release. Also, even if I add it, I will leave the page action in because it is better for the android version (more visible). Users that don't want the page action can hide it anyway, so it should be OK.

By the way, I forgot to mention I made a new beta release a while back (0.7.0b3). I hope there aren't any outstanding bugs left so I can write the documentation and do the actual release on AMO soon.

Thanks again for your interest and your help :beers:

[in4u]

I tried the new version and immediately ran into an annoying bug. There is no way to clear the incognito whitelist. The Clear whitelist button doesn't do that. And importing the modified backup file also doesn't do that since the entries get overwritten by the extension.

This version also surprised me by showing knownSecure entries when I didn't even have this option enabled. After scratching my head a bit, I recalled enabling this option temporarily just once during testing yesterday. Now I am left wondering if the settings HTTPZ actually uses are the same as shown in the settings backup file or not?!

And I have one question. What happens if knownSecure is enabled and a site decides (for testing or whatever reason) to lose its certificate and move from https to http (either temporarily or permanently)? Is this setting along the lines of HSTS where things will just get stuck in a redirection loop? Or is the user notified of error and given an option to take further action? Again, this is a theoretical question for me since I don't ever plan to enable it simply because majority of the web is now https and I don't want HTTPZ to hog resources by practically storing my history. Am just not a fan of having this setting enabled by default.

[claustromaniac]

The Clear whitelist button doesn't do that.

It does, but saving afterwards (without reloading the options page) restored it. Fixed in 0.7.0b5.

This version also surprised me by showing knownSecure entries when I didn't even have this option enabled.

That's intended. The extension never clears that list on its own. That is left entirely up to the user to do (via the button in the options page). If you disable the option and don't clear the list, the list is not used. Simple as that.

What happens if knownSecure is enabled and a site decides (for testing or whatever reason) to lose its certificate and move from https to http (either temporarily or permanently)?

HTTPZ doesn't intervene in that scenario, it respects the server's preference to downgrade to HTTP. Note that, for servers to redirect you from HTTPS to HTTP, a secure communication needs to be established first.

I don't want HTTPZ to hog resources by practically storing my history.

It wouldn't store your history, only hostnames. I just ran a test with 6 tabs open and 10,000 hostnames saved in that list, and HTTPZ takes merely 8.4 MB of RAM. Compare that to the (last time I checked, months ago) easily over 20 or 30 MB that HTTPS Everywhere uses up. Even with that option enabled, this extension will always use up less memory, because it is only saving hostnames that YOU navigate to, and you can wipe the list whenever you want. HTTPS Everywhere is instead trying to index the entire internet, which is an ever-growing environment.

I don't know, it sounded like a good compromise, but maybe that's just me.

Edit: BTW, only hostnames for 1st-party sites are ever stored in that list.

[in4u]

That's intended. The extension never clears that list on its own. That is left entirely up to the user to do (via the button in the options page). If you disable the option and don't clear the list, the list is not used. Simple as that.

Fair enough. But in the b2 version the exported file didn't show these entries and b3 suddenly showed these old entries. Which version was the actual intended behaviour - b2 or b3?

Even with that option enabled, this extension will always use up less memory ... only hostnames for 1st-party sites are ever stored in that list

As we have discussed in this thread, browser history or autofill or cache and what not already sort of "remembers" the https versions thereby bypassing the need of HTTPZ once a site is visited. Moreover, HTTPZ won't be preventing MITM attack from sites which I have never visited before. So there is very little value addition that this options brings, at least for me. All it does is store what the browser already knows plus the hostnames only opened in incognito mode. So yeah, I'll pass on that.

However, I understand that security savvy folks may appreciate this option. But if ever I am caught in the middle of MITM attack, rest assured it will not be due to HTTPZ's fault and there would be far bigger things for me to worry about than regretting not having this option enabled.

[claustromaniac]

Which version was the actual intended behaviour - b2 or b3?

The only bug I'm aware of in that feature was in b1 and I fixed it in b2 207dbff

I'm closing this because it seems I already fixed all the problems you reported here. If that is not the case, let me know and I'll reopen. Otherwise, open a new issue if you want to report other bugs or if you have more suggestions.

[claustromaniac]

Just FYI, 0.7.0b6 optimizes the option to remember secure sites. Now the extension with my test list of 10,000 hostnames uses only 500 KB of RAM instead of the previous 8.4 MB. Additionally, it introduces an alternative to that option that may be of use to you.

[in4u]

Sorry, missed b6 but just now skipped straight to 0.7.1 and it works great!

On Firefox for Android, the settings page has some minor cosmetic layout issues, e.g. text overflows in some buttons, button font sizes are inconsistent, text for some options wraps around after a word or two, etc.

[claustromaniac]

Thanks for letting me know. I still didn't get around to testing it on Android, but will soon. I didn't declare some of those things you mention so they should be at their default values, but css on android seems to be problematic in general.. sigh