Web UI of eBus Adaper "Header fields are too long"

[Koky05]

I try to access WebUI from internet via CloudFlare tunnel, but most of the time I get only "Header fields are too long". If I try to run in PrivateMode to start without any saved cookies etc.. it loads but only once and then show same message.

Page is shown correctly if I access it directly via IP address.

Error is presented in both latest two FW (2023-10-15 and 2023-12-17).

[john30]

then check the headers sent. should not be too many anyway

[Koky05]

I look into it and it looks like CloudFlare (I use CloudFlare tunnel to external access) add too many info into Request HTTP header. So it is irrelevant for adapter it self. It was just weird because no other service has problem with that.

[Koky05]

I try to search this problem and maybe I found solution but it need your help. Can you please compile one FW for eBus v5 with next parameter?

# HTTP Server
#
CONFIG_HTTPD_MAX_REQ_HDR_LEN=1024

[john30]

it is already at 1024. check how much headers you're actually using

[Koky05]

I try it to fount and think it was 2448. That was result of: curl ... -w '%{size_request} %{size_upload}'

If I try direct call IP address without Cloudflare tunnel with turned on login page it was 473. Can you create test FW for me with: CONFIG_HTTPD_MAX_REQ_HDR_LEN=2560

[Koky05]

I will skip login page from Cloudflare tunnel if login page will be in front of every interface, not only configuration page. Because I don't like to open REPL to the world, but I want to have option to run update on adapter also outside of my network if I am on work travel.

[john30]

why don't you just remove unnecessary headers in the request to the origin as documented in cloudflare? https://developers.cloudflare.com/rules/transform/request-header-modification/

[Koky05]

I was trying but for Cloudflare native headers (the biggest one) it is restricted to remove or change: 'remove' / 'set' is not a valid value for operation because it cannot be used on header beginning with 'cf-' (Code: 20087)

[john30]

I've created another patch for ESP-IDF to just skip too lengthy headers. will be in the next release

[john30]

done in new release, please check

[Koky05]

When I turn off authentication layer on Cloudflare tunnel and open it in Private Mode first load are successfull. But reloading of page or open it normal mode end with error Bad request syntax.

If I turn on authentication layer all call of page finish with same error Bad request syntax, no matter if I use PrivateMode.

Success call (InPrivate page without authentication):

curl "https://xxx.yyyyy.zz/" ^
  -H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ^
  -H "accept-language: sk-SK,sk;q=0.9" ^
  -H "dnt: 1" ^
  -H "priority: u=0, i" ^
  -H ^"sec-ch-ua: ^\^"Chromium^\^";v=^\^"124^\^", ^\^"Google Chrome^\^";v=^\^"124^\^", ^\^"Not-A.Brand^\^";v=^\^"99^\^"^" ^
  -H "sec-ch-ua-mobile: ?0" ^
  -H ^"sec-ch-ua-platform: ^\^"Windows^\^"^" ^
  -H "sec-fetch-dest: document" ^
  -H "sec-fetch-mode: navigate" ^
  -H "sec-fetch-site: none" ^
  -H "sec-fetch-user: ?1" ^
  -H "upgrade-insecure-requests: 1" ^
  -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

Reload of page without autentication:

curl "https://xxx.yyyyy.zz/" ^
  -H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ^
  -H "accept-language: sk-SK,sk;q=0.9" ^
  -H "cache-control: no-cache" ^
  -H "cookie: cf_clearance=***149bytes***" ^
  -H "dnt: 1" ^
  -H "pragma: no-cache" ^
  -H "priority: u=0, i" ^
  -H ^"sec-ch-ua: ^\^"Chromium^\^";v=^\^"124^\^", ^\^"Google Chrome^\^";v=^\^"124^\^", ^\^"Not-A.Brand^\^";v=^\^"99^\^"^" ^
  -H "sec-ch-ua-mobile: ?0" ^
  -H ^"sec-ch-ua-platform: ^\^"Windows^\^"^" ^
  -H "sec-fetch-dest: document" ^
  -H "sec-fetch-mode: navigate" ^
  -H "sec-fetch-site: same-origin" ^
  -H "sec-fetch-user: ?1" ^
  -H "upgrade-insecure-requests: 1" ^
  -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

New InPrivate page with authentication:

curl "https://xxx.yyyyy.zz/" ^
  -H "accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7" ^
  -H "accept-language: sk-SK,sk;q=0.9" ^
  -H "cache-control: no-cache" ^
  -H "cookie: ***1528bytes***" ^
  -H "dnt: 1" ^
  -H "pragma: no-cache" ^
  -H "priority: u=0, i" ^
  -H "referer: https://yyyyy.cloudflareaccess.com/" ^
  -H ^"sec-ch-ua: ^\^"Chromium^\^";v=^\^"124^\^", ^\^"Google Chrome^\^";v=^\^"124^\^", ^\^"Not-A.Brand^\^";v=^\^"99^\^"^" ^
  -H "sec-ch-ua-mobile: ?0" ^
  -H ^"sec-ch-ua-platform: ^\^"Windows^\^"^" ^
  -H "sec-fetch-dest: document" ^
  -H "sec-fetch-mode: navigate" ^
  -H "sec-fetch-site: cross-site" ^
  -H "sec-fetch-user: ?1" ^
  -H "upgrade-insecure-requests: 1" ^
  -H "user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

EDIT by john30: removed sensitive data

[john30]

thanks for the feedback, guess there are more edge cases to cover...

[john30]

please check with the current test firmware (update by using "?file=testing.bin" as suffix to the adapter URL and then updating to that one in the dialog)

[Koky05]

Same behavior as with 20240518 FW.

Chip ID: 58cf79f0d268, ESP32-C3, rev 3
Hostname: ebus-adapter-v5
PCB: 1.22.1
Build: 20240520
Flash: 4194304
IDF: v5.3-dev-3675-ge486f3b944-dirty

[john30]

should work now when reflashing the testing.bin . if it does, you owe me a beer ;)

[Koky05]

You made it, so I will buy you a beer and not just one. It works.

[john30]

thanks for the beer, enjoyed it at lunch today :-) closing as completed