basic example: captive portal just says "Page Not Found"

[IDC-Dragon]

Hi, I'm new, discovered this project today and can share my fresh user experience. Compiling the basic example worked. However, IDF v5.1 is a steep requirement, I had to upgrade. v5.1-rc2 is the latest I could find, IMHO 5.1 is not out yet? Starting it for the first time falls in AP mode as expected. The captive portal works, however only displays a message "Page Not Found". Besides that, it has the language setting in the upper right corner and a disclaimer footer line, therefore has delivered content. I was expecting a WiFi selection? Accessing the server via 192.168.4.1 in another browser tab shows 2 blocks of information, "System summary" and "WiFi Access Point" with current information, but also no WiFi selector. Is this perhaps misconfigured? (verbatim example code, no change)

This is my startup log, any problem here?

load:0x40080404,len:3500
entry 0x400805e4
0:00:00:00.357 I basic  starting basic 829e9f8
0:00:00:00.358 W system  (1331) SPIFFS: mount failed, -10025. formatting...
0:00:00:01.127 W config-vfs  file /root/config.json not found
0:00:00:01.127 I basic  init level 0
0:00:00:01.136 I basic  initialization complete
0:00:00:01.136 I wifi  device initialized
0:00:00:01.159 W system  (2131) phy_init: failed to load RF calibration data (0x1102), falling back to full calibration
0:00:00:01.304 I wifi  changing mode: AP -> Disabled
0:00:00:01.306 W wifi  no APs to connect to, switching to AP mode
0:00:00:01.306 I WIFI_AP_DEF  initial -> starting
0:00:00:01.307 I wifi  changing mode: Disabled -> AP
0:00:00:01.313 I WIFI_AP_DEF  setting AP config
0:00:00:01.315 I WIFI_AP_DEF  starting -> running
0:00:00:01.323 I captive-dns  captive portal starting at 192.168.4.1
0:00:01:16.067 W system  (77041) httpd_parse: cb_headers_complete: Upgrade header does not match the length of "websocket"
0:00:01:16.068 W system  (77041) httpd_parse: parse_block: parsing failed
0:00:01:16.068 W system  (77041) httpd_txrx: httpd_resp_send_err: 400 Bad Request - Bad request syntax
0:00:01:17.361 D ui-http-54  session opened

[IDC-Dragon]

This is what it looks like on my mobile.

[dyarkovoy]

Hi, thank you for the feedback. The project is under heavy development, so we use ESP-IDF master branch to get the latest features. We didn't introduce versioning yet, as there are no real-world uses as far as we know. We will of course add branches for release versions of ESP-IDF once there's a need for it. As far as captive portal, we didn't test on Apple devices, only on Android. Will have to get an iPhone to check why this happens. For now, you can either connect from Android phone, or navigate to http://192.168.4.1/cp on iPhone, it should open the page with the list of APs to connect to. Btw, are you able to check what URL it navigated to when it shows 'page not found' ?

[IDC-Dragon]

On the iPhone, I didn't manage to navigate away from the captive portal page. As soon as trying so, it disconects from the AP and returns to the previous. I did my side-peeking on a notebook. Not sure how to see the true URL. Firefox seems to wander to originaldomain/redirect by itself or originaldomain/canonical.html when clicking the button.

OT: I could help you quickly with your german translations. The are a bit ... odd ... here and there. ;-)

[IDC-Dragon]

In an attempt to help diagnose, I enabled a debug info log, first line of Httpd::incomingReq(). These are the URLs I get with my iPhone:

0:00:01:03.326 I ui-http  uri: /hotspot-detect.html
0:00:01:04.522 I ui-http  uri: /hotspot-detect.html
0:00:01:04.552 I ui-http  uri: /hotspot-detect.html
0:00:01:04.579 I ui-http  uri: /main.js
0:00:01:04.924 I ui-http  uri: /hotspot-detect.html

And these on the notebook with firefox:

0:00:04:45.174 I ui-http  uri: /connecttest.txt
0:00:04:46.092 I ui-http  uri: /canonical.html
0:00:04:46.104 I ui-http  uri: /redirect
0:00:04:46.270 I ui-http  uri: /main.js
0:00:04:46.642 I ui-http  uri: /wpad.dat
0:00:04:46.651 I ui-http  uri: /wpad.dat
0:00:04:47.635 I ui-http  uri: /wpad.dat
0:00:04:49.178 I ui-http  uri: /canonical.html
0:00:04:49.823 I ui-http  uri: /wpad.dat
0:00:04:49.832 I ui-http  uri: /wpad.dat

Does that help?

[dyarkovoy]

Okay, but does it work if you connect to the device AP and then manually navigate to http://192.168.4.1/cp in the browser on your iPhone or Mac? I noticed it shows 'captive.apple.com' on top of your screenshot, maybe that is the host portion of the URL we can use to detect captive portal request. I made some changes to the code to react to that host, please git pull, recompile and try again. If still does not work, i added more debug info about HTTP request, please uncomment the first 2 lines of Httpd::incomingReq() and send the dump of http requests. Regarding German translation, you're right, I don't speak German so I just used google translate. One other person requested this, and he is already working on providing correct translations, but if you see something really odd, you are welcome to make corrections (translations are in the web-ui/**/index.ts files) and submit a PR

[IDC-Dragon]

The iPhone won't let me navigate anywhere else while having auto-opened the captive page. When changing away from it, the AP connection gets closed and WiFi is connected back to the previous known-good station. So I can't check the /cp page with it. However I did with the notebook, that's working.

With your changes, it is fixed for the iPhone. Here is the log part:

0:00:00:00.613 I captive-dns  captive portal starting at 192.168.4.1
0:00:00:46.505 I ui-http  uri: /hotspot-detect.html
0:00:00:46.505 I system  header: Host: captive.apple.com
0:00:00:46.505 I system  header: Connection: close
0:00:00:46.505 I system  header: User-Agent: CaptiveNetworkSupport-443.120.3 wispr
0:00:00:46.510 I ui-http  redirecting to captive portal http://192.168.4.1/cp
0:00:00:46.560 I ui-http  uri: /cp
0:00:00:46.561 I system  header: Host: 192.168.4.1
0:00:00:46.561 I system  header: Connection: close
0:00:00:46.561 I system  header: User-Agent: CaptiveNetworkSupport-443.120.3 wispr
0:00:00:47.632 I ui-http  uri: /hotspot-detect.html
0:00:00:47.632 I system  header: Host: captive.apple.com
0:00:00:47.632 I system  header: Upgrade-Insecure-Requests: 1
0:00:00:47.632 I system  header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
0:00:00:47.657 I system  header: Connection: keep-alive
0:00:00:47.632 I system  header: User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148
0:00:00:47.633 I system  header: Accept-Language: de-DE,de;q=0.9
0:00:00:47.633 I system  header: Accept-Encoding: gzip, deflate
0:00:00:47.633 I system  header: Connection: keep-alive
0:00:00:47.639 I ui-http  redirecting to captive portal http://192.168.4.1/cp
0:00:00:47.656 I ui-http  uri: /cp
0:00:00:47.657 I system  header: Host: 192.168.4.1
0:00:00:47.657 I system  header: Upgrade-Insecure-Requests: 1
0:00:00:47.657 I system  header: Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
0:00:00:47.722 I system  header: Accept-Language: de-DE,de;q=0.9
0:00:00:47.657 I system  header: User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148
0:00:00:47.728 I system  header: Referer: http://192.168.4.1/cp
0:00:00:47.746 I system  header: Accept-Encoding: gzip, deflate
0:00:00:47.657 I system  header: Accept-Language: de-DE,de;q=0.9
0:00:00:47.657 I system  header: Accept-Encoding: gzip, deflate
0:00:00:47.687 I ui-http  uri: /hotspot-detect.html
0:00:00:47.687 I system  header: Host: captive.apple.com
0:00:00:47.687 I system  header: Connection: close
0:00:00:47.687 I system  header: User-Agent: CaptiveNetworkSupport-443.120.3 wispr
0:00:00:47.694 I ui-http  redirecting to captive portal http://192.168.4.1/cp
0:00:00:47.716 I ui-http  uri: /main.js
0:00:00:47.716 I system  header: Host: 192.168.4.1
0:00:00:47.716 I system  header: Connection: keep-alive
0:00:00:47.717 I system  header: Accept: */*
0:00:00:47.717 I system  header: User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 16_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148
0:00:00:48.038 I ui-http  uri: /cp
0:00:00:48.038 I system  header: Host: 192.168.4.1
0:00:00:48.038 I system  header: Connection: close
0:00:00:48.038 I system  header: User-Agent: CaptiveNetworkSupport-443.120.3 wispr
0:00:00:48.150 I ui-http  uri: /hotspot-detect.html
0:00:00:48.150 I system  header: Host: captive.apple.com
0:00:00:48.150 I system  header: Connection: close
0:00:00:48.150 I system  header: User-Agent: CaptiveNetworkSupport-443.120.3 wispr
0:00:00:48.155 I ui-http  redirecting to captive portal http://192.168.4.1/cp
0:00:00:48.174 I ui-http  uri: /cp
0:00:00:48.175 I system  header: Host: 192.168.4.1
0:00:00:48.175 I system  header: Connection: close
0:00:00:48.175 I system  header: User-Agent: CaptiveNetworkSupport-443.120.3 wispr
0:00:00:49.341 D ui-http-55  session opened
0:00:00:49.354 I wifi  changing mode: AP -> AP+STA
0:00:00:49.359 I wifi  scan started

On the notebook with Windows 10 and Firefox, it is still not being redirected to the /cp page. Will make another log for that, hang on.

[IDC-Dragon]

This is my log when connecting to the AP with Windows 10, default browser Firefox popping up to handle it: connect_attempt_firefox.log

And this happens when I click the button of Firefox, suggesting the captive portal page: connect_attempt_firefox_button.txt

In both cases, I get the "Page Not Found".

[IDC-Dragon]

About the UX with current /cp page: IMHO the main intention is to list the stations, have one picked and ask for PSK. But the page starts with verbose information about the spawned AP, what for? After having entered the PSK and successful connect, I'd expect something to happen, like thanks, stopping AP, goodby. Instead the new station client data is silently inserted at the very top, if I scroll there. The AP stays for some minutes.

[dyarkovoy]

I made changes for firefox CP, please see if it works for you now. I agree with you about the UX, moved list of APs to the top. You are right that we need to indicate successful connection to the user, will think of the best way to implement it. Maybe additional box on top saying "configuration complete, you can now access this device via http://IP_adddress/ from your LAN". I don't think stopping AP right away is a good idea. The user may have connected to the wrong AP, in which case they will not be able to access the device at all. Our current design is that the AP is taken down automatically when there are no active connections (the user switched to their router AP manually)

[IDC-Dragon]

Firefox now works, too, thank you! (I'm surprised this has to be coded for every platform/browser, is there no standard way?)

You're right about keeping the AP alive for a while. The reference-based shutdown of the AP is indeed a clever way.

[dyarkovoy]

According to my research (valid as of 2 years ago), there is no standard way. Every OS/device/browser developer seems to invent their own and unique way. There was a proposal (RFC) to make it a standard on DHCP level (an option pushed to DHCP client along with the IP address, telling it that the server is a captive portal), but I am not aware of any implementations of this proposal

[IDC-Dragon]

Forgive my naivity, can't just any access be redirected to the portal page?

[dyarkovoy]

Well it can be, but that may cause other issues. There are valid HTTP requests from the browser to the server to load parts of the UI, that must not be redirected, for example http://192.168.4.1/main.js The UI is a single-page React APP that is the same for captive portal and for the regular UI. The path part of the URL is used to show different pages of the UI, so such paths as /home, /wifi, /system, /mqtt and many others must also not be redirected. Furthermore, there's a WebSocket server at /ws endpoint that also must not be redirected. The UI is modular, so other endpoints may be added in the future, so the task to redirect everything to /cp may be less trivial compared to detecting captive portal requests from different devices. I admit your suggestion may be an option, and I did consider it in the past, but at that point decided that drawbacks outweihg the benefits. If we continue facing issues with other devices, I may re-consider and go your way.