Security?

[cxtal]

Hello and congratulations for an awesome project! The portal can be reached on the local network and, in case custom webpages are added that contain forms requiring credentials, then the credentials can be observed on the local network by just browsing to the device webserver.

Would there be some way to secure the entire portal? I know that ESP8266WebServercan support multiple authentication schemes such as BASIC, DIGEST, etc. but is there a way to implement that through portal.handleclient() elegantly?

Many thanks!

[Hieromon]

To secure the entire portal with the standard method of ESP8266WebServer, it is necessary to enhance the AutoConnect library.

BASIC authentication will be relatively easy to implement because AutoConnect knows its portal pages and the pages added on by AutoConnectAux. However, some users do not like this method because it is so simple. (It's better than nothing) So I have hesitated to implement.

When using MD5, we may get involved in the complex issues surrounding the captive portal. IETF95 has discussed the security issues of captive portals in the past. https://tools.ietf.org/html/draft-nottingham-capport-problem-00 After all, the behavior of the client OS determines whether it can operate as intended as a captive portal. As of now, I haven't come up with any implementable way with ESP8266 that can solve this problem.

[cxtal]

Hi @Hieromon - thanks for replying! Please let me specify: the captive portal is not a problem. In the end, it has to be open so you can connect to it the first time and perform the initial configuration! That part is already secured by WPA2 and involves just the ESP and the connecting client without any third-parties (except those that would challenge WPA2. . .).

The problem is that the ESP can be accessed via the local network by browsing to its IP address in the local network. When the connection is opened to the IP address from the local network using a browser (ie: http://192.168.1.10/_ac), all the data is transferred without any sort of security.

If you have, say, passwords saved on the portal, they are transferred back and forth in plaintext, over the local network. This allows eavesdroppers to sniff the passwords easily. On a home network, maybe, that could perhaps not be a problem but if you are setting up a bunch of ESPs for a populated network, then most likely connecting to the IP address from the local network should be secured with at least DIGEST- I have seen that the ESP8266WebServersupports DIGEST: it is lightweight, involves an exchange of nonces and tokens between the browser and the server, should provide better security in the absence of SSL and should be supported by any browser.

Many thanks and sorry for the confusion!

[Hieromon]

@evacomaroski If you are talking about the ESP8266WebServer::authenticate and requestAuthenciation methods, Due to need to call the authenticate the requestAuthentication with each page, both BASIC and DIGEST of ESP8266WebServer are still an incomplete way for certification, there is no difference. If it should affect portal pages in the /_ac range including AutoConnectAux, I need to enhance the library as I mentioned earlier. Its enhancement will invoke authenticate and requestAuthenciation corresponding to the request handler of the library. (Both AutoConnect and PageBuilder) However, this method has no effect on individual pages prepared by sketches. (Its substance is the request handler registered with the ESP8266WebServer::on function.) Conversely, if you want to pass individual credentials, it can be implemented outside of a custom web page with AutoConnectAux.

Here I want to confirm your wishes again. What are the credentials you mentioned? for WiFi access-point which handled by AutoConnect? or for something else? After all, do you want DIGEST authentication for custom web pages with AutoConnectAux?

[cxtal]

@Hieromon

Both BASIC and DIGEST authentications are cached in the browser [1] such that browsing to a different page on the same webserver will not require re-entering credentials. The difference between BASIC and DIGEST is that BASIC sends the password in plaintext whereas DIGEST encrypts the password [2] using some key-derivation scheme.

Succinctly, the problem is the following:

1. when the ESP is in AP mode, then a wireless client can connect to it and configure it
2. when the ESP is configured to connect to the local network, then a client on the local network can access the ESP webserver via the IP address and browse the pages.

In case (1), even though there is no authentication, connecting to the ESP when it is in AP mode (for the first time) and not connected to any network could be acceptable because it could be considered an initial configuration phase. Additionally, the credentials are transmitted between the ESP and the wireless client configuring the ESP such that sniffing the credentials is not possible. Sure, long-term ideas would be to generate a password that is based on some observable serial number for each ESP device [3].

In case (2), if any of the pages, both /_ac and custom pages created with AutoConnectAux contain any password forms, then any computer on the local network can see the passwords without even having to sniff the network. All that a computer on a local network would have to do, is scan the local network and check whether an "/_ac" URL is available, if so, the crawler could scan all the pages and discover pages containing sensitive information. It is true that after authentication, the information will be sent unencrypted unless SSL is added such that the credentials could be sniffed - however, not allowing access to the webserver in the first place is a step forward.

Here is an example of a custom webpage that I have created with AutoConnectAux following your documentation:

If you ever come to my house, all you have to do is find out the ESP IP, then you navigate with a browser to the AutoConnect portal and you can extract my MQTT password, username, IP address, port, etc. - none of which have a reason to be publicly accessible by anyone connected to the same network. Even so, you could do even more fun stuff by disrupting the ESP: reset it, make it connect to a different network, overwrite or vandalize the current settings, etc.

Now imagine that I am using AutoConnect for more serious stuff, let's say that I have many ESP devices spread out in a building with many employees and various network setups, each ESP containing custom pages with various sensitive information. You could get a lot of information by just accessing the AutoConnect portal and looking over the form fields.

As said earlier, this could be automated by writing a simple HTTP crawler:

1. scan the network IP block for available IPs, and for each IP,
2. check if IP/_ac is a valid URL (200),
3. crawl IP/_ac and, for all links with forms, extract all sensitive information

Again, without encryption (SSL), the information can be sniffed under certain circumstances, however finding a way to restrict access to the ESP webserver using some authentication would at least prevent access to the portal and any custom pages in the first place.

[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching [2] https://www.w3.org/Protocols/HTTP/digest_specification.html [3] https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=201720180SB327

[Hieromon]

@evacomaroski, You may have misunderstood my question, but I have not denied all authentication support. I had already considered your point. SSL is too heavy with ESP8266 and does not work properly with captive portal detection on some OSes. (I have tried it in the past and gave up, too little dynamic heap area left for user's sketch) If it is a BASIC / DIGEST certification, AutoConnect can only influence the AutoConnectAux pages. It does not affect pages that are uniquely equipped with sketches. The parameters entered on the AutoConnectAux page can be passed to the ESP8266WebServer::on handler page which missing call ESP8266WebServer::authenticate. So, it is imperfectly, and I asked you a question.

After all, do you want DIGEST authentication for custom web pages with AutoConnectAux?

Is the value still useful to users? I will always consider it positively if it is still worth it. Do you have a way to be able to authenticate the page with the "on" function? If you have it, the idea is an important help for me. I say agin, I have not denied the authentication support. Thank you for your contribution.

[cxtal]

@Hieromon Yes, I imagined that SSL might be too heavy. Since I have started using AutoConnect, I went ahead and used only AutoConnectAux for custom webpages (ie: the screenshot from the previous comment).

After all, do you want DIGEST authentication for custom web pages with AutoConnectAux?

Yes. I have no other custom web pages that do not use AutoConnectAux. It keeps things consistent and unified in one single interface. The / (home) URL is not used at all for settings. So just /_ac would have to be protected since that is where the settings are made.

I do not mind much (in the absence of SSL) whether it is DIGEST or any other authentication: perhaps a simple page with Login and Password (or perhaps just password) should be sufficient? I mentioned DIGEST because I have seen it might be easy with the ESP8266 webserver and because it provides some additional protection over BASIC but I do not think it should matter too much.

Is the value still useful to users?

Imagine we work in the same place and I find some ESP devices with AutoConnect. I can just control them by entering their IP addresses plus /_ac and then I can turn off your ESP, I can modify the settings, I can reset it, perhaps you have OTA enabled too so I can reflash it, etc...

I guess that if users only use the ESP on their own personal network with only one user then there should be no problem. Otherwise, imagine you build a firm and use ESP devices to automate the doors or window blinds. Anyone on the local network could then just disable all your IoT devices.

Imagine a pub like StarBucks that would use AutoConnect: we can just go there, buy some coffee, connect to their "free wifi", scan for ESP devices and /_ac to determine if they are using it and then, not sure, can make their coffee machines go wild. It would be a tragedy without protection. :D

Do you have a way to be able to authenticate the page with the "on" function? If you have it, the idea is an important help for me.

I am pretty new to Arduino so I am not too sure that I can provide that kind of insight. However, as per the previous paragraph, even if you just add a "login form" - like, on a router login page, it is still much much better.

Thank you for considering it! The project is awesome! :-)

[Hieromon]

@evacomaroski, Thank you. I understood your intention cleary. I start enhancement for DIGEST authentication for AutoConnect pages, including attached by AutoConnectAux as follows as a draft proposal for authentication support:

1. Add an option to AutoConnectConfig to enable authentication.
2. The authentication method is a DIGEST. BASIC authentication is also selectable to save runtime memory.
3. A user sketch can configure a scope of certification. i.e: Whether the certification range influences pages only attached by AutoConnectAux or including whole AutoConnect captive portal pages.
4. A user sketch cannot exclude for individual pages from a scope of certification. Just enable or disable authentication.

Please tell me your opinion.

[cxtal]

@Hieromon That looks very good! If it is too much of a hassle, even just a login form should at least prevent people from browsing / crawling for ESP devices on a network. :)

[Hieromon]

@evacomaroski This implementation is not too hard to work, but I am currently working for asynchronous AutoConnect. Combining it with ESPAsyncWebServer frees user sketches from the annoying headache of "When should call AutoConnect::handleClient" in a loop function. However, even though I started implementation, I get the difficulty with thread synchronization with AsyncTCP on ESP32. It's tougher than expected. So support for authentication may be a little delayed. Or maybe it's better to release the authentication feature first. Let me think about it.

[cxtal]

@Hieromon Thank you for considering it! I am not sure about others but for me the security issue is critical. I live in a densely populated area where the local network is sort-of a mutual provider for an entire set of buildings. Connecting an ESP to such a network is pure suicide because right now anyone can access it and do anything with it. It might not be a problem for people living in remote and isolated areas but where I am the cables are piling on top of each other. :D I could wake up to all my devices going crazy... :D I get like 2-5 random bluetooth pairing requests on my phone alone if I leave bluetooth as discoverable....

Just saying so you can have an image in your head on the gravity of the situation...

[Hieromon]

@evacomaroski I staged the authentication support as v1.2.0 to the development branch. You can evaluate it under the following conditions:

• AutoConenct 1.2.0 requires PageBuilder 1.4.0, which has not released yet. Please download manually and replace it with your libraries folder of Arduino/libraries/PageBuilder. Also, for AutoConnect 1.2.0 during staging with authentication support is the same.
  Or, Change a branch on your local repository after cloning to enhance/v120 for AutoConnect, to enhance/v140 for PageBuilder.

• I changed the configuration parameters a bit from the above draft. You can configure AutoConnectConfig as follows according to your requirements. It is achieved by a combination of two parameters, AutoConnectConfig::authentication and AutoConnectConfig::scope:

  ---

  1. Enable authentication AutoConnectConfig::authentication specifies enable HTTP authentication.
     • AC_AUTH_NONE : No authentication. (Default)
     • AC_AUTH_BASIC : Use HTTP authentication with BASIC method.
     • AC_AUTH_DIGEST : Use HTTP authentication with DIGEST method.
  2. Authentication scope AutoConnectConfig::scope specifies a scope of authentication influence pages.
     • AC_AUTHSCOPE_PORTAL : Whole AutoConnect portal pages including AutoConnectAux pages. (Not recommended)
     • AC_AUTHSCOPE_AUX : All AutoConnectAux pages. (Default)
     • AC_AUTHSCOPE_PARTIAL : Authenticate only specific AutoConnectAux pages which are specified by AutoConnectAux::authentication function or JSON.
  3. Authentication for the certain AutoConnectAux page with JSON You can specify the scope of authentication to certain pages as described below.
  4. Username and password You can specify a user name with AutoConnectConfig::username, password with AutoConnectConfig::password. Default values are same as APID/PSK. (Usually it is esp8266ap / 12345678 for esp8266)

• Not tested yet with ESP32.

Examples

1. Authentication for all AutoConnectAux pages.

AutoConnect portal;
AutoConnectConfig config;

void setup() {
  config.authentication = AC_AUTH_DIGEST;
  config.scope = AC_AUTHSCOPE_AUX; // Not necessary
  config.username = "admin";
  config.password = "password";
  portal.config(config);
  portal.begin();
}

2. Authentication with an only a certain page.
   • Use AUtoConnectAux::authentication function.

     Void AutoConnectAux::authentication(const AC_AUTH_t auth)

     AC_AUTH_t type is enumeration with { AC_AUTH_NONE, AC_AUTH_DIGEST, AC_AUTH_DIGEST }.

• Specify with JSON. A key is auth, a value is none (default), basic or digest. (Ignore case)

  {
  "title": "MQTT Setting",
  "uri": "/mqtt_setting",
  "menu": true,
  "auth": "digest",
  "element": [
  ]
  }

• An example sketch

  
  static const char AUX_mqtt_setting[] = R"(
  [
  {
  "title": "MQTT Setting",
  "uri": "/mqtt_setting",
  "menu": true,
  "auth": "digest",
  "element": [
    {
      "name": "header",
      "type": "ACText",
      "value": "<h2>MQTT broker settings</h2>",
      "style": "text-align:center;color:#2f4f4f;padding:10px;"
    }
  ]
  },
  {
  "title": "MQTT Setting",
  "uri": "/mqtt_save",
  "menu": false,
  "element": [
    {
      "name": "caption",
      "type": "ACText",
      "value": "<h4>Parameters saved as:</h4>",
      "style": "text-align:center;color:#2f4f4f;padding:10px;"
    }
  ]
  }
  ]
  )";

AutoConnect portal; AutoConnectConfig config;

void setup() { config.authentication = AC_AUTH_DIGEST; config.scope = AC_AUTHSCOPE_PARTIAL; // authentication for /mqtt_setting only config.username = "admin"; config.password = "password"; portal.config(config); portal.load(AUX_mqtt_setting); // Loads pages that will be authenticated according to the "auth" key specified. portal.begin(); }

#### Limitations ####
- Authentication not works under the captive portal detection state. Due to `ESP8266::requestAuthentication` does not support **511** HTTP response.

However, testing is not enough.
If you have the time, please test it and report the results. It would be helpful for me.
Thank you for your contribution.

[cxtal]

@Hieromon that looks great! Will test for you with some WeMoS Mini devices - perhaps integrate it already in my current sketch. Many thanks!

[Hieromon]

@evacomaroski I found a problem during testing with AC_AUTH_PARTIAL also AC_AUTH_AUX such as should discuss with you. If you access AutoConnectAux pages which are unauthorized yet while detecting the captive portal, then a 401 HTTP error will occur without prompting the login dialog on the web client. It is due to the condition of captive portals. When AutoConnect is under the captive portal state, it has not been authenticated to the network. That is, authentication via the upper protocol HTTP is not valid. This problem was expected as we already know, but I overlooked it for the AutoConnectAux pages behaving. If you want to enable HTTP authentication even if AutoConnect is under the captive portal state, I think unfortunately not. So, I would like you to consider the followings about the AutoConnect to better behavior:

• AutoConnectAux pages that require authentication until the ESP8266 connects to any WiFi network are always unauthorized. i.e: You cannot enter secret parameters via AutoConnectAux pages unless you connect to any WiFi network.
  Can you accept this?

• The above-mentioned inconsistency of authentication behavior can be avoided by the user sketch.

  AutoConnect portal;
  AutoConnectConfig config;
  
  void setup() {
  WiFi.begin();
  if (WiFi.waitForConnectResult() != WL_CONNECTED)
    config.authentication = AC_AUTH_NONE;
  else
    config.authentication = AC_AUTH_PARTIAL;
  portal.config(config);
  portal.begin();
  }

  But this is a strange workaround.

[cxtal]

@Hieromon Hi! I do not think that authentication when the ESP is an AP mode is needed: when the ESP is in AP mode, there are no other observers, except someone attempting to break WPA2. So, regardless what information is transferred between the wireless client and the AP, that information will always be protected by WPA2.

The only problem with unprotected pages is when the ESP is connected to a network since that network may contain adversaries that currently have unhindered access to the ESP web server without authentication. Otherwise, SSL aside and too bad for that, but the protections are fairly reasonable given the limitations.

I would not even bother with authentication in AP mode. I do not see why. There is a short window where an adversary could connect to the AP instead of you but that would be protected by the wireless password. Having said that, the only thing that I would like personally, but I know that it might not be possible, is to have a default password set for every ESP device based on its serial number, instead of the default 12345678 password for all devices. At a first glance, the serial does not seem to be printed on these boards but I have not placed the ESP under a microscope for a closer look.

Along those lines, I have made a few changes to your sketch from here: https://github.com/Hieromon/AutoConnect/blob/master/examples/mqttRSSI/mqttRSSI.ino such that:

• every device will have its own AP BSSID based on the serial number of the device,
• the MQTT client ID is configurable or defaults to some generated ESP and serial number string, instead of being randomly generated every time the device resets (some environments using MQTT sometimes make decisions based on the MQTT client ID).

If you want to take a look, this is what the sketch looks like so far: http://grimore.org/fuss/arduino/start#code - ok, the meaning of the sketch is changed a little because it is meant to toggle GPIO pins and does not broadcast RSSI but most of the connectivity is still similar between both sketches. If it serves as any inspiration, please feel free to adjust your own sketches based on it. 😄

[Hieromon]

@evacomaroski I thought that I intend to understand your circumstances, and your insight is right. But, one thing to keep in mind is how the library should be.

In my experience, some users want to use this authentication feature not as security, to use as granting the role. Earlier I was requested by another user for support to protect the configuration data for the sketch. The user intended to distribute the ESP module embedding the unique sketch to multiple users. He wanted to lock the AutoConnectAux page from other users' access to change pre-built configuration data. (eg MQTTbroker account) For them, they want to enable authenticate pages for their own sketch data even under the captive portal state. It conflicts with your requirements.

Originally, HTTP authentication is for restricting site visits, not for data protection. However, many users indeed misunderstand it. I don't want the library to capture conflicting requirements, so I will prioritize convenience and exclude certification under the captive portal state. However, I would like to know if you have a solution that can resolve these conflicting requirements.

[cxtal]

If I understand correctly:

Regardless of AutoConnectAux being used or not. Without protecting "/_ac", then anyone on the local network can change the settings, and, at the very least, either:

• make the ESP device connect to some arbitrary access point of their choosing,
• reset the device completely,
• gather some information about the local network?

etc.

For them, they want to enable authenticate pages for their own sketch data even under the captive portal state. It conflicts with your requirements.

I do not mind either way: as I have stated before, if the ESP configuration is protected in any way you like (using HTTP authentication or just a password form) then someone on the local network cannot change the settings. If you want to protect the portal in AP mode too, then that is even better!

My only concern is that on a network with multiple users, if one of those users discover the "/_ac" link then that user can abuse the device without any sort of hindrance. If one of your users is on a network that nobody connects to, then I guess this would not be an issue. Otherwise, in a more serious usage case of ESP devices, you certainly would not want anybody being able to just browse to its webpage and change the AP it connects to or poke around stuff that does not concern them.

The Tasmota firmware for Sonoff [1] devices is somewhat similar to AutoConnect in that it provides some sort of fault tolerance and an initial configuration stage while in AP mode (Tasmota in AP mode does not have any authentication at all!). However, Tasmota does have an option to enable authentication (IIRC, it's just BASIC) to protect others from accessing the device webpage and then arbitrarily change settings.

[1] https://github.com/arendst/Tasmota/wiki

[cxtal]

Attached is a screenshot of the Tasmota UI -> Configure Other -> Web Admin Password. It can e enabled and disabled using the checkbox as well as changed. Once this is enabled, then HTTP BASIC is enabled for the webserver. The developers at Tasmota protect absolutely everything through this authentication password: even the entire HTTP REST API is protected if this option is enabled and nobody on the network can access the device in any way or do anything with it.

[Hieromon]

@evacomaroski I agree with your mention of the need for confidentiality regarding /_ac access, as mentioned earlier. I know TASMOTA, but does TASMOTA use a captive portal trick to connect devices to the network? I do not know. Is it using HTTP authentication in a captive portal state? how?

I don't understand why you quoted TASMOTA. AutoConnect is a library based on esp8266 / esp32 arduino-core. It is not a framework that encompasses the entire firmware. The structure is different. I don't think suggesting a differently structured framework will help solve the problem. My difficulty that how to enable HTTP authentication properly under the captive portal state. It is an implementation methodology and extremely native content with HTTP protocol. If I can't find a way, this specification is inconsistent and will throw out someone who needs it. In this case, the specifications as a library will not be closed, as I already expected.

[cxtal]

Hi @Hieromon !

I know TASMOTA, but does TASMOTA use a captive portal trick to connect devices to the network? I do not know.

AFAIK, no, it does not.

Is it using HTTP authentication in a captive portal state? how?

No. It does not. ... probably because they used the same reasoning as I explained previously: in AP mode, there are no other adversaries and it is assumed that in AP mode the ESP will be configured on the spot.

I don't understand why you quoted TASMOTA. I don't think suggesting a differently structured framework will help solve the problem. It is an implementation methodology and extremely native content with HTTP protocol.

Either an authentication form or HTTP level implementation should suffice. Otherwise, anyone using these devices on a network with multiple network clients will be hacked (en masse).

An authentication form with a password should be sufficient and will work for either AP mode or when the ESP is connected to the network (the password will be transmitted in plaintext but if SSL is not possible, it will be just as secure as BASIC).

[Hieromon]

@evacomaroski, Preventing the attacks from the externals will reach to the minimum level with this feature, I have the same understanding as you. What I think needs to be considered is another perspective that is for the Access restriction for the secret parameters entered from AutoConnectAux pages for administrator persons. According to my specifications, the behavior is as follows:

	The behavior of AutoConnectAux page with authentication specified. Which is better for the library specification?
	#1. The previous implementation as commit db2b6eb	#2. This is the staged on the development branch
in the captive portal state (not established the network, only AP)	Always unauthorized (Without prompting the login dialog in the web browser)	Always authorized (Without prompting the login dialog in the web browser)
Established network as STA	Prompting the login in the web browser. Operators have the opportunity to be certified.	same as left

The requestAuthentication of core library function can only respond with the 401 response from the ESP8266WebServer. This is a restriction on using the ESP8266WebServer of an arduino core library. (esp32 arduino-core as well as) In this case, it is better to return a 511 response, but that does not guarantee that the client browser will work as expected with login prompting. In the captive portal state, the OS running on the client devices opens the browser in an ad hoc incognito mode window that separates from the normal browser. By being open in incognito mode, there is no using the normal browser session or cookie information. This perspective will not be a problem in personal use cases. So we can choose specification #2 as fine.

However, this specification creates situations that are undesirable for configuration administrators or owners. Once the ESP module is disconnected from the network by AutoConnect, the module will launch the captive portal on the next reboot turn. In other words, in thespecification #2, any user can access AutoConnectAux with authentication specified.

If we choose the specification #1 to circumvent it, owners or administrators will not be able to access pages specified authentication unless connected to any network. (Even if the parameters are configurable in a stand-alone state)

The difference between the constraints due to these specifications is what I say needs to be considered.

will work for either AP mode or when the ESP is connected to the network

How do I implement under the captive portal state (=AP mode) using arduino-core library or HTTP natively? That's the question I mentioned earlier. So, Have you tested v140 candidate with the development branch?

[cxtal]

@Hieromon hi!

In the captive portal state, the OS running on the client devices opens the browser in an ad hoc incognito mode window that separates from the normal browser. By being open in incognito mode, there is no using the normal browser session or cookie information.

Hmm - are you sure about this? AFAIK, "incognito mode" [1] means that the session and the cookies will be purged once the browser is closed.

Namely, citing [1]: "Cookies set in private windows are held temporarily in memory, separate from regular window cookies, and discarded at the end of your private session (after the last private window is closed)."

If the cookies are purged on browser close, then we do not care because they just need to work for the duration that the client is connected.

If "incongito mode" would not allow cookies at all, not even temporarily, then it could just as well be named "useless mode" because no websites would work. Even the spicy ones need some sort of persistent storage. 😆

Once the ESP module is disconnected from the network by AutoConnect, the module will launch the captive portal on the next reboot turn.

I know this is unrelated to this issue, but this behaviour is very annoying, albeit disruptive and leads to a tedious chore every time there is a power outage. Every time the power goes down I have to reconnect a bunch of ESP devices back to the Wifi network when the power comes back on. . .

Perhaps something could be done about this?

This is one of the largest annoyances I have with all my IoT equipment (not just AutoConnect). I am not sure why this behaviour is the default? When AutoConnect gets disconnected from Wifi, it should try to reconnect indefinitely and perhaps some hardware switch should be used to make it go back to AP mode (if need be) - or perhaps some HTTP request from a browser?

It is far more likely to have a power outage than to, say, have to make the ESP go back to AP mode. In fact, I am not sure if there is a usage case when I will want the ESP to go back to AP mode?

Hmm, perhaps when I change wifi? But that seems rare though. It is not like I'm buying a new router every day so I constantly need it to go back to AP mode. . .

In other words, in thespecification #2, any user can access AutoConnectAux with authentication specified.

Can they though? After the reboot when AutoConnect is in AP mode the password is still required to connect to the AP, no? So it will be protected via the WPA2 password challenge.

What I mean is that in variant #1 - in AP mode, there is no need for any authentication because to connect to the ESP/AutoConnect portal you need the password (default: 12345678 - of course, perhaps an easy option to change this default password conveniently and persistently).

So:

How do I implement under the captive portal state (=AP mode) using arduino-core library or HTTP natively? That's the question I mentioned earlier.

Don't? Because to connect to the AutoConnect portal in AP mode you need the wifi (default 12345678) password so it is protected anyway? Or did I get that wrong?

Perhaps, instead change the default AutoConnect interface so you can set the default Wifi password - namely, to be able to change 12345678 to something else conveniently (and persistently)?

Summarizing, the way I see it is this:

• AP/portal mode: a client needs to know 12345678 to connect and there are no other observers since there is no network (ok, a network between you and the ESP but no others and that only after authenticating with 12345678). So, secure (ok, perhaps a better default password and/or a convenient way to change it).
• client mode / connected to a wifi network: anyone on the local network can connect to the ESP, browse to "/_ac" and change settings at will without needing to know 12345678. So, insecure. Wide open, actually.

So, Have you tested v140 candidate with the development branch?

Many thanks! Will give it a spin this weekend. I just got a fresh batch of ESPs from my dealer. Will be the first thing I try. 😄

Cheers!

[1] https://support.mozilla.org/en-US/kb/private-browsing-use-firefox-without-history

[Hieromon]

@evacomaroski Thank you for your guidance.

Hmm - are you sure about this? AFAIK, "incognito mode" [1] means that the session and the cookies will be purged once the browser is closed.

Let me verify this again with various OSes. It depends on client browser implementation. It doesn't work well on iOS at least with ESP8266WebServer library.

Every time the power goes down I have to reconnect a bunch of ESP devices back to the Wifi network when the power comes back on. . .

This behavior is a bit strange. WiFi.begin without parameters will restore the connection with the SSID that was previously established. And AutoConnect::begin first tries WiFi.begin with no parameters. In case of power loss, once power is restored, the module itself will join the previously connected WiFi network and enter STA mode after reboot, even if you don't use Open SSID manually in the AutoConnect menu. The only time this feature will not work is when you explicitly disconnect from a WiFi network using the Disconnect menu.

Don't? Because to connect to the AutoConnect portal in AP mode you need the wifi (default 12345678) password so it is protected anyway? Or did I get that wrong?

You can connect to the ESP module in AP mode by the WiFi password. At that point, the AutoConnect screen will pop up automatically. That is a captive portal. On that screen, the operator can invoke the custom web pages which handle the secret parameters with authentication from the AutoConnect menu. The module remains in AP mode which equals the captive portal state in the browser. In this state, a 401 response from the server with a WWW-Authenticate header will always result in an Unauthorized error without prompting the browser login form. This is a problem in cases where the person who knows the WiFi password and the person who knows the access password for the AutoConnect custom web page are different. This is my understanding, is it wrong?, And I have received such a request to the AutoConnect previously on the issues.

I'm not so good at English, so I hope that my English gets through to you.

[cxtal]

@Hieromon ok, thanks for clearing this up and it makes more sense to me now what the problem is. Your English is great and mine is not great either!

Let me verify this again with various OSes. It depends on client browser implementation. It doesn't work well on iOS at least with ESP8266WebServer library.

Captive portals in the context of, say, 802.11X have some form of authentication (and, of course, accounting). Examples such as public space Internet (hotels, airports), EDUROAM, etc. - all these are protected by captive portals where you are supposed to enter credentials and without any sort of session tracking (ie: cookies), as you say, you could not even log in!

If you look at the referenced blog [1], all the captive portal pages (screenshots) have a username and password challenge. Without those, I am not sure how logging in would work or make sense. . . Perhaps passwords wouldn't be required if you had some client certificate or would perform MAC whitelisting but then, it would more or less defeat the notion of a public captive portal, given that the credentials would be known beforehand.

I've dug up the captive portal RFC 7710 [2] and it only really mentions that a captive portal will tell the browser that it is an insecure page. However, the RFC states it is entirely up to the browser (implementation side) to decide how to handle an insecure page.

As it turns out, indeed Apple seemingly has had some issues with captive portals involving cookies [3, 4]. Some workarounds are suggested on the latter link but they mostly have to do with client side fixes. It appears Apple bungled the job Captive Network Assistant (CNA) does not allow cookies to be stored [5]. Apparently, the workaround for Apple is to disable CNA entirely and use the regular browser [6] which will allow cookies to be stored.

Will give this a shot today and see what happens and let you know what happens if I disable CNA.

If nothing else works nor feasible, you could just pass the credentials as query strings - given that the captive portal is still protected by WPA2, it should not matter much. Guess you'll have to carry strings throughout the entire interface but that is still not that bad for a few configuration pages.

Iff. you can distinguish between a client connected to the ESP via AP or via the local network, then you could do both:

• if a client connects to the AP then create a hash of a secret and pass it around on all AutoConnect and AutoConnectAux pages as hidden fields,
• if a client connects via the network, just use cookies

Or just use a hidden field for all pages, including custom pages, since it would not be that much of a difference security-wise in the absence of SSL.

In the end, this is not really your problem, but an Apple problem (cookies work fine on Windows and Android in captive portal mode apparently). You are right, Apple is wrong. 😄

This is a problem in cases where the person who knows the WiFi password and the person who knows the access password for the AutoConnect custom web page are different.

Pretty sure this can be made a non-problem 😄 in various ways. Off the top of my head:

• the default wifi password 12345678 can become the default password for accessing the ESP via the local network - so, same password for both. It is not great, but it can be just the default and, if users want to change it, then so be it.
• add a small form to change the wifi password (and the network access password) from the default AutoConnect interface. Users can decide what to do: either set the same password for both or pick different passwords.

[1] https://gshaw0.wordpress.com/tag/aruba/ [2] https://tools.ietf.org/html/rfc7710 [3] https://discussions.apple.com/thread/5258403?tstart=0 [4] https://success.tanaza.com/s/article/Cookies-Required-issue-on-Apple-laptops [5] https://discussions.apple.com/thread/5258403 [6] https://apple.stackexchange.com/questions/45418/how-to-automatically-login-to-captive-portals-on-os-x

[Hieromon]

@evacomaroski New PR #181 from @dmaixner has occurred that allows the entrance of the captive portal to be an AutoConnectAux page. This change allows the AutoConnectAux page with HTTP authentication to pop up with the captive portal and is related to what we discussed here. I would like you to participate in this PR review with an HTTP authentication perspective. It is an important adjustment for the specification that is about to conflict. Could you help us?

[cxtal]

@Hieromon In the initial PR you solved an issue with a default behaviour which was by default insecure out of the box but is now by default secure. If users want to boot to a custom page instead of the default behaviour, then so be it but they are then responsible for their own security.

As mentioned in the previous post: a network more or less implies a multiuser environment. However, it may just be that someone is the only user on their network. In which case, authentication via the network is redundant and perhaps could just be a problem for a certain specific setup.

As long as the default case is covered and secure then I see no conflict. You cannot cover all the possible usage cases using your library but offering options is the way to go.

[Hieromon]

@cxtal Thank you for your advice. I will prepare an additional option to the v1.2.0 release. I already shifted the work for v1.2.0 implementation to documentation. It's taking time due to my poor English skills. Please wait for a while until the release.

[Hieromon]

@cxtal Hello, Much time has passed, I started preparing to release v1.2.0 with authentication support. It has your advice reflected as the candidate specification. In studying the specification, I verified the HTTP (not HTTPS, included an Authorization: request header) authentication condition on a few different OSs with the captive portal and confirmed a particular behavior on iOS and Android10 clients. When the server (ie ESP8266WebServer class) returned 401 unauthorized with WWW-Authenticate header, iOS client will not respond to Authorization: request with a credential. But, when I attempt to access the ESP module as SoftAP directly (ie 172.217.28.1 as the URL), the iOS client displays a login dialog and returns the credential to the ESP8266WebServer. Then authentication works fine. That is, as you mentioned earlier in this thread, iOS does not allow HTTP authentication under the captive portal. It will always fail, Android10 too. On the other hand, Older devices like Android7, Linux (Debian), and Windows10 return the credential entered at the login dialog as an Authorization: request to the server regardless of the captive portal. In other words, those OSs allow HTTP authentication under the captive portal. Based on this survey, I have decided the following specifications provisional:

The AutoConnectConfig-auth setting has a special bit that bypasses authentication in the captive portal state:

• AC_AUTHSCOPE_PORTAL
• AC_AUTHSCOPE_AUX
• AC_AUTHSCOPE_PARTIAL

The above is conventional enumerated values. Then I added the extended bit at MSB for the captive portal state.

• AC_AUTHSCOPE_EXCEPTCP

If this bit is set, AutoConnect skips authentication in the captive portal state. If its bit is off, it will attempt to authenticate even in the captive portal state. Therefore, the page cannot be accessed on iOS and newer Android devices with the captive portal.

AutoConnect portal;
AutoConnectConfig config;
...
config.auth = AC_AUTH_DIGEST;
config.authScope = AC_AUTHSCOPE_AUX | AC_AUTHSCOPE_EXCEPTCP;
portal.config(config);
portal.begin();
...

However, I can't have confidence in that many users can accept this specification. Anything I’m missing? I would like you to let me know the further considerations if you have. Also, you can evaluate v1.2.0-RC via enhance/v120 branch using the following Sketch:

#include <Arduino.h>
#if defined(ARDUINO_ARCH_ESP8266)
#include <ESP8266WiFi.h>
#include <ESP8266WebServer.h>
using WebServerClass = ESP8266WebServer;
#elif defined(ARDUINO_ARCH_ESP32)
#include <WiFi.h>
#include <WebServer.h>
using WebServerClass = WebServer;
#endif
#include <AutoConnect.h>

static const char PAGE_AUTH[] PROGMEM = R"(
{
  "uri": "/auth",
  "title": "Auth",
  "menu": true,
  "auth": "digest",
  "element": [
    {
      "name": "text",
      "type": "ACText",
      "value": "AutoConnect authorized",
      "style": "font-family:Arial;font-size:18px;font-weight:400;color:#191970"
    }
  ]
}
)";

WebServerClass    server;
AutoConnect       portal(server);
AutoConnectConfig config;

void setup() {
  delay(1000);
  Serial.begin(115200);
  Serial.println();

  // Comment out one to confirm the behavioral variations.
  // config.authScope = AC_AUTHSCOPE_PORTAL;
  // config.authScope = AC_AUTHSCOPE_PORTAL | AC_AUTHSCOPE_EXCEPTCP;
  config.authScope = AC_AUTHSCOPE_AUX;
  // config.authScope = AC_AUTHSCOPE_AUX | AC_AUTHSCOPE_EXCEPTCP;
  // config.authScope = AC_AUTHSCOPE_PARTIAL;
  // config.authScope = AC_AUTHSCOPE_PARTIAL | AC_AUTHSCOPE_EXCEPTCP;

  config.auth = AC_AUTH_DIGEST;
  config.ota = AC_OTA_BUILTIN;    // Built-in OTA will also be subject to authentication.

  config.immediateStart = true;
  config.portalTimeout = 1;
  config.retainPortal = true;
  portal.config(config);
  portal.load(FPSTR(PAGE_AUTH));
  portal.begin();
}

void loop() {
  portal.handleClient();
}

Thank you for your contribution.

[NOTE] For esp32, WebServer library of Arduino core 1.0.4 stable release has a bug for authentication. The fix has already merged into the master but has not released as a stable yet. It will be the next release turn of ESP32 arduino core. https://github.com/espressif/arduino-esp32/pull/3329

[cxtal]

@Hieromon Glad the project is getting along! :+1:

Would it not be possible to automatically disable the authentication when AutoConnect is not connected to a network? As per the previous: when the ESP is not connected to a network, all traffic will be secured by WAP2 (and users have to authenticate with the password); when the ESP is connected to a network that's when it is vulnerable to unauthorized access because anyone can access the pages.

Would it not be much easier to just turn off authentication when the ESP is not associated with any AP and enable authentication when it is associated? It would allow iOS to at least configure the ESP till it is associated with an AP.

In other words, just set AC_AUTHSCOPE_EXCEPTCP by default (if it disables authentication in AP mode but enables authentication when the ESP is associated?).

Otherwise, disabling security entirely is fine since it is up to the user to do as the user desires. You can only ensure that your software is secure out of the box.

[Hieromon]

@cxtal AutoConnect can automatically turn off authentication when the ESP module is standalone, that is, it is not associated with any AP. I will immediately reconsider the configuration of option settings and change the specifications. I will inform you again after the correction.

[Hieromon]

@cxtal I restored the authentication in the captive portal state to the previous specifications. The authentication behavior reverted, and the accuracy of determining whether the request is from the captive portal has improved. It will decide by the host address of the HTTP request header also WiFi status. This deprecates AC_AUTHSCOPE_EXCEPTCP and replaces it with AC_AUTHSCOPE_WITHCP, the effect of which is to enable authentication even in the captive portal state as you mentioned. That is, authentication will not allow with the captive portal by default.

// Enable authentication with the captive portal state
config.authScope = AC_AUTHSCOPE_AUX | AC_AUTHSCOPE_WITHCP;

A new RC is staged on the enhance/v120.

[JemRF]

Is it not possible just to add a password screen? Then no pages in AutoConnect or custom pages are accessible until the password is entered correctly?

[Hieromon]

@JemRF We have an option to integrate the login page into AutoConnect. It can also be customized by the user using AutoConnectAux. But it's probably plain text and you'll need to handle the request yourself to hash it using MD5. The same goes for AutoConnect, which is just a request handler for the ESP8266WebServer class. ESP8266 WebServer class requires a lot of memory to match the hashed text on the request handler side. Also related to this, I previously considered supporting HTTPS, but also gave up due to lack of memory. How we implement depends on what we want to protect with a password, but currently, v120 will add HTTP authentication based on what we have talked to in this thread.

P.S. I traced almost all the code of the ESP8266WebServer class, but couldn't find a good workaround. To be honest, I still feel there is a possibility to implement it, do you have any idea?

[JemRF]

Do you have any documentation on how to implement a login page? or is it still in the development branch as discussed in this thread?

[Hieromon]

@JemRF Development branch is enhance/v120 and still available. The current implementation is just calling ESP8266WebServer::authentication when AutoConnect dynamically generates the response page after receiving an HTTP request from ESP8266WebServer. It exists in the final step of AutoConnect::_setupPage. (Coded in AutoConnectPage.cpp. Actually, call to PageBuilder::authentication) https://github.com/Hieromon/AutoConnect/blob/a2a100bc4c747c203a5fee4ba024464ea62ef7f7/src/AutoConnectPage.cpp#L1561-L1603

There is no documentation specific to how to implement the login page. Just use AutoConnectElements (a combination of AutoConnectInput and AutoConnectSubmit) to implement AutoConnectAux and request handlers. Of course, you'll need a flag that shows the authentication status. This way can implement custom web page authentication.

[cxtal]

@cxtal AutoConnect can automatically turn off authentication when the ESP module is standalone, that is, it is not associated with any AP. I will immediately reconsider the configuration of option settings and change the specifications.

This seems the way to go. To recap, iOS considers captive portal networks to be insecure, regardless whether authentication can be performed with DIGEST instead of plaintext BASIC.

It will decide by the host address of the HTTP request header also WiFi status.

It may be perhaps better to just use the WiFi status since HTTP headers are sent by the client and cannot be trusted.

Many thanks again for a great piece of software!

[Hieromon]

Merged #280

[playground]

Hi @Hieromon, will you reconsider adding HTTPS support?

[Hieromon]

@playground I have considered its implementation in the past, but there are no plans for HTTPS support at this time.