Crash on multiple async reqs

dr3adx commented 1 year ago

Happens when I try to send around 6 requests per second. Using ASYNCHttpReqs library. I'm so frustrated I thought async requests will work. Like everything works fine until it crashes couple of minutes later.

AsyncTCP.cpp line 420 specifically


  #0  0x40083705:0x3ffb3250 in panic_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/panic.c:402
  #1  0x4008d169:0x3ffb3270 in esp_system_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp_system/esp_system.c:128
  #2  0x40092569:0x3ffb3290 in __assert_func at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/newlib/assert.c:85
  #3  0x400f68f6:0x3ffb33c0 in tcp_update_rcv_ann_wnd at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c:951
      (inlined by) tcp_update_rcv_ann_wnd at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c:931
  #4  0x400f69a4:0x3ffb33e0 in tcp_recved at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp.c:991
  #5  0x400d6f3d:0x3ffb3400 in _tcp_recved_api(tcpip_api_call_data*) at .pio/libdeps/ESP32/AsyncTCP/src/AsyncTCP.cpp:420
  #6  0x400f38e8:0x3ffb3420 in tcpip_thread_handle_msg at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:172
      (inlined by) tcpip_thread at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/api/tcpip.c:154

umerm64 commented 1 year ago

have you found any solution @dr3adx ? I am facing the same problem.

trycoon commented 1 year ago

@dr3adx @umerm64 AsyncTCP is a wonderful library riddled with bugs, there has been lots of Pull request to fix these, they are never merged, it's just too sad for something as good as this. Do this fork improve your situation? https://github.com/smarthomerocks/AsyncTCP

dr3adx commented 1 year ago

@umerm64 havent found solution, its broken on god knows what level

@trycoon i actually applied all open pull requests, to no avail. This issue happens on low level when trying to obtain IP/socket on ip_tcp level

trycoon commented 1 year ago

@dr3adx I used to have trouble with the ESP32 getting Panics and rebooting when receiving multiple requests, but after applying the pull requests in my fork it is much more stable now. Did you remeber to increase the CONFIG_ASYNC_TCP_QUEUE_SIZE to something bigger than the default 32? I'm running -DCONFIG_ASYNC_TCP_QUEUE_SIZE=512

dr3adx commented 1 year ago

@dr3adx I used to have trouble with the ESP32 getting Panics and rebooting when receiving multiple requests, but after applying the pull requests in my fork it is much more stable now. Did you remeber to increase the CONFIG_ASYNC_TCP_QUEUE_SIZE to something bigger than the default 32? I'm running -DCONFIG_ASYNC_TCP_QUEUE_SIZE=512

Hello bro, yes i've tried that, but my issue is when sending multiple requests from multiple AsyncClients. Basically web scraping. I would also get some crash along the lines: tcp_rcv_wnd < 0xFFFF

What crash specifically did you get before?

My case can be easily reproduced by making a system which should send API req to 10 different websites, each holding a unique cookie and to a different URL so its a session. So initializing 10 AsyncClient objects is where the problem happens I think, because with low amount of async reqs, e.g 3 the issue doesnt happen. But then again, even 10 is nothing, 100+ async reqs in a given time can be easily handled in nodejs for comparison

umerm64 commented 1 year ago

@trycoon I am also getting the same error on this one too. :( https://github.com/smarthomerocks/AsyncTCP

Basically the crash is happening when too many concurrent http requests are sent to controller. Is there a way to throttle/queue the requests ?

dr3adx commented 1 year ago

umerm64

have you found any solution bro?

umerm64 commented 1 year ago

@dr3adx no not yet :(

HamzaHajeir commented 1 year ago

Would this solve any issue? https://github.com/HamzaHajeir/H4AsyncTCP

dr3adx commented 1 year ago

Would this solve any issue? https://github.com/HamzaHajeir/H4AsyncTCP

gonna test it very soon bro and let u know

HamzaHajeir commented 1 year ago

I'm on it improving, last published should be ok (and far better), but now developing TLS with stabilizing the TCP management.

On Tue, Jun 6, 2023, 21:46 dr3adx @.***> wrote:

Would this solve any issue? https://github.com/HamzaHajeir/H4AsyncTCP

gonna test it very soon bro and let u know

— Reply to this email directly, view it on GitHub https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1579279384, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3C7MWH2KO4O46YFN3XJ53GHANCNFSM6AAAAAAVJWOJLI . You are receiving this because you commented.Message ID: @.***>

dr3adx commented 1 year ago

I'm on it improving, last published should be ok (and far better), but now developing TLS with stabilizing the TCP management. … On Tue, Jun 6, 2023, 21:46 dr3adx @.> wrote: Would this solve any issue? https://github.com/HamzaHajeir/H4AsyncTCP gonna test it very soon bro and let u know — Reply to this email directly, view it on GitHub <#161 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3C7MWH2KO4O46YFN3XJ53GHANCNFSM6AAAAAAVJWOJLI . You are receiving this because you commented.Message ID: @.>

you have Discord bro?

HamzaHajeir commented 1 year ago

Nope ... But why?

On Tue, Jun 6, 2023, 22:11 dr3adx @.***> wrote:

I'm on it improving, last published should be ok (and far better), but now developing TLS with stabilizing the TCP management. … <#m-2325937816652580759> On Tue, Jun 6, 2023, 21:46 dr3adx @.> wrote: Would this solve any issue? https://github.com/HamzaHajeir/H4AsyncTCP https://github.com/HamzaHajeir/H4AsyncTCP gonna test it very soon bro and let u know — Reply to this email directly, view it on GitHub <#161 (comment) https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1579279384>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3C7MWH2KO4O46YFN3XJ53GHANCNFSM6AAAAAAVJWOJLI https://github.com/notifications/unsubscribe-auth/AH3O7J3C7MWH2KO4O46YFN3XJ53GHANCNFSM6AAAAAAVJWOJLI . You are receiving this because you commented.Message ID: @.>

you have Discord bro?

— Reply to this email directly, view it on GitHub https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1579310088, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J57XS7VIOHLBLKAP4LXJ56GHANCNFSM6AAAAAAVJWOJLI . You are receiving this because you commented.Message ID: @.***>

johnnytolengo commented 1 year ago

Hi @HamzaHajeir is your library compatible with ESPAsyncWebServer?

HamzaHajeir commented 1 year ago

Hi @HamzaHajeir is your library compatible with ESPAsyncWebServer?

Hi @johnnytolengo, it's not, really.

But the stack comes with a stable H4AsyncWebServer that you can check out.

TienHuyIoT commented 1 year ago

have you found any solution @dr3adx ? I am facing the same problem.

You can use this repository with the fix-bugs relevant to memory leaks and crash tcp/ip api calls https://github.com/TienHuyIoT/AsyncTCP

dr3adx commented 1 year ago

Hi @HamzaHajeir is your library compatible with ESPAsyncWebServer?

Hi @johnnytolengo, it's not, really.

But the stack comes with a stable H4AsyncWebServer that you can check out.

hello bro, I want to try your async TCP library but HTTP library is missing? https://github.com/philbowles/H4AsyncHTTP link is not working, got any idea where can I get async HTTP library?

HamzaHajeir commented 1 year ago

Hi there, here's the HTTP client library:

https://github.com/HamzaHajeir/ArmadilloHTTP

On Sun, Nov 26, 2023, 06:25 dr3adx @.***> wrote:

Hi @HamzaHajeir https://github.com/HamzaHajeir is your library compatible with ESPAsyncWebServer?

Hi @johnnytolengo https://github.com/johnnytolengo, it's not, really.

But the stack comes with a stable H4AsyncWebServer https://github.com/hamzahajeir/h4asyncwebserver that you can check out.

hello bro, I want to try your async TCP library but HTTP library is missing? https://github.com/philbowles/H4AsyncHTTP link is not working, got any idea where can I get async HTTP library?

— Reply to this email directly, view it on GitHub https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1826478198, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA . You are receiving this because you were mentioned.Message ID: @.***>

dr3adx commented 1 year ago

Hi there, here's the HTTP client library: https://github.com/HamzaHajeir/ArmadilloHTTP … On Sun, Nov 26, 2023, 06:25 dr3adx @.> wrote: Hi @HamzaHajeir https://github.com/HamzaHajeir is your library compatible with ESPAsyncWebServer? Hi @johnnytolengo https://github.com/johnnytolengo, it's not, really. But the stack comes with a stable H4AsyncWebServer https://github.com/hamzahajeir/h4asyncwebserver that you can check out. hello bro, I want to try your async TCP library but HTTP library is missing? https://github.com/philbowles/H4AsyncHTTP link is not working, got any idea where can I get async HTTP library? — Reply to this email directly, view it on GitHub <#161 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA . You are receiving this because you were mentioned.Message ID: @.>

how do I get this running in platformio? do I need h4plugins ? do I need AardvarkTCP?

My current platformio config:

lib_deps = 
    https://github.com/HamzaHajeir/H4
    https://github.com/HamzaHajeir/H4Tools
    https://github.com/HamzaHajeir/H4AsyncTCP@^0.0.18
    https://github.com/HamzaHajeir/ArmadilloHTTP
    khoih-prog/ESPAsync_WiFiManager@>=1.15.1

I'm trying to compile example program and I get error during linking: https://pastebin.com/ZgStVidc

HamzaHajeir commented 1 year ago

Please checkout the platformio environment here: https://github.com/hamzahajeir/h4plugins_env

And it's important to utilize h4 APIs h4setup and h4UserLoop instead of setup and loop.

On Sun, Nov 26, 2023, 14:29 dr3adx @.***> wrote:

Hi there, here's the HTTP client library: https://github.com/HamzaHajeir/ArmadilloHTTP … <#m-6879621497364389165> On Sun, Nov 26, 2023, 06:25 dr3adx @.> wrote: Hi @HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir is your library compatible with ESPAsyncWebServer? Hi @johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo, it's not, really. But the stack comes with a stable H4AsyncWebServer https://github.com/hamzahajeir/h4asyncwebserver https://github.com/hamzahajeir/h4asyncwebserver that you can check out. hello bro, I want to try your async TCP library but HTTP library is missing? https://github.com/philbowles/H4AsyncHTTP https://github.com/philbowles/H4AsyncHTTP link is not working, got any idea where can I get async HTTP library? — Reply to this email directly, view it on GitHub <#161 (comment) https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1826478198>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA . You are receiving this because you were mentioned.Message ID: @.>

how do I get this running in platformio? do I need h4plugins https://github.com/philbowles/h4plugins ?

lib_deps = https://github.com/HamzaHajeir/H4 https://github.com/HamzaHajeir/H4Tools https://github.com/HamzaHajeir/H4AsyncTCP@^0.0.18 https://github.com/HamzaHajeir/ArmadilloHTTP khoih-prog/ESPAsync_WiFiManager@>=1.15.1

I'm trying to compile example program and I get error during linking: https://pastebin.com/ZgStVidc

— Reply to this email directly, view it on GitHub https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1826758214, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J5HVPCFVNVPL4OTZT3YGMR2JAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWG42TQMRRGQ . You are receiving this because you were mentioned.Message ID: @.***>

dr3adx commented 1 year ago

Please checkout the platformio environment here: https://github.com/hamzahajeir/h4plugins_env And it's important to utilize h4 APIs h4setup and h4UserLoop instead of setup and loop. … On Sun, Nov 26, 2023, 14:29 dr3adx @.> wrote: Hi there, here's the HTTP client library: https://github.com/HamzaHajeir/ArmadilloHTTP … <#m-6879621497364389165> On Sun, Nov 26, 2023, 06:25 dr3adx @.> wrote: Hi @HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir is your library compatible with ESPAsyncWebServer? Hi @johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo, it's not, really. But the stack comes with a stable H4AsyncWebServer https://github.com/hamzahajeir/h4asyncwebserver https://github.com/hamzahajeir/h4asyncwebserver that you can check out. hello bro, I want to try your async TCP library but HTTP library is missing? https://github.com/philbowles/H4AsyncHTTP https://github.com/philbowles/H4AsyncHTTP link is not working, got any idea where can I get async HTTP library? — Reply to this email directly, view it on GitHub <#161 (comment) <#161 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA . You are receiving this because you were mentioned.Message ID: @.> how do I get this running in platformio? do I need h4plugins https://github.com/philbowles/h4plugins ? lib_deps = https://github.com/HamzaHajeir/H4 https://github.com/HamzaHajeir/H4Tools https://github.com/HamzaHajeir/H4AsyncTCP@^0.0.18 https://github.com/HamzaHajeir/ArmadilloHTTP khoih-prog/ESPAsync_WiFiManager@>=1.15.1 I'm trying to compile example program and I get error during linking: https://pastebin.com/ZgStVidc — Reply to this email directly, view it on GitHub <#161 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J5HVPCFVNVPL4OTZT3YGMR2JAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWG42TQMRRGQ . You are receiving this because you were mentioned.Message ID: @.>

Got it. i downloaded your platformio project. However, I can only compile env:esp32s3-devkitc otherwise with other envs i get error:

.pio/libdeps/espwroom32/H4AsyncTCP/src/H4AsyncTCP.h:197:21: error: 'ip_addr_t' does not name a type; did you mean

but my board is not esp32s3-devkitc and i cannot upload firmware (even tho it compiles successfully), and it doesnt compile with board i normally use "az-delivery-devkit-v4" If I set -DLWIP_ALTCP=0 to zero I can compile just fine what should I do?

HamzaHajeir commented 1 year ago

If you don't need TLS it's fine, remove LWIP_ALTCP and everything will be fine.

If you do need it, please let me know to check it out at sooner.

On Sun, Nov 26, 2023, 19:13 dr3adx @.***> wrote:

Please checkout the platformio environment here: https://github.com/hamzahajeir/h4plugins_env And it's important to utilize h4 APIs h4setup and h4UserLoop instead of setup and loop. … <#m-3998183735438878006> On Sun, Nov 26, 2023, 14:29 dr3adx @.*> wrote: Hi there, here's the HTTP client library: https://github.com/HamzaHajeir/ArmadilloHTTP https://github.com/HamzaHajeir/ArmadilloHTTP … <#m-6879621497364389165> On Sun, Nov 26, 2023, 06:25 dr3adx @.> wrote: Hi @HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir is your library compatible with ESPAsyncWebServer? Hi @johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo, it's not, really. But the stack comes with a stable H4AsyncWebServer https://github.com/hamzahajeir/h4asyncwebserver https://github.com/hamzahajeir/h4asyncwebserver https://github.com/hamzahajeir/h4asyncwebserver https://github.com/hamzahajeir/h4asyncwebserver that you can check out. hello bro, I want to try your async TCP library but HTTP library is missing? https://github.com/philbowles/H4AsyncHTTP https://github.com/philbowles/H4AsyncHTTP https://github.com/philbowles/H4AsyncHTTP https://github.com/philbowles/H4AsyncHTTP link is not working, got any idea where can I get async HTTP library? — Reply to this email directly, view it on GitHub <#161 https://github.com/me-no-dev/AsyncTCP/issues/161 (comment) <#161 (comment) https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1826478198>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA . You are receiving this because you were mentioned.Message ID: @.> how do I get this running in platformio? do I need h4plugins https://github.com/philbowles/h4plugins https://github.com/philbowles/h4plugins ? lib_deps = https://github.com/HamzaHajeir/H4 https://github.com/HamzaHajeir/H4 https://github.com/HamzaHajeir/H4Tools https://github.com/HamzaHajeir/H4Tools https://github.com/HamzaHajeir/H4AsyncTCP@^0.0.18 https://github.com/HamzaHajeir/H4AsyncTCP@%5E0.0.18 https://github.com/HamzaHajeir/ArmadilloHTTP https://github.com/HamzaHajeir/ArmadilloHTTP khoih-prog/ESPAsync_WiFiManager@>=1.15.1 I'm trying to compile example program and I get error during linking: https://pastebin.com/ZgStVidc https://pastebin.com/ZgStVidc — Reply to this email directly, view it on GitHub <#161 (comment) https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1826758214>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J5HVPCFVNVPL4OTZT3YGMR2JAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWG42TQMRRGQ https://github.com/notifications/unsubscribe-auth/AH3O7J5HVPCFVNVPL4OTZT3YGMR2JAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWG42TQMRRGQ . You are receiving this because you were mentioned.Message ID: @.***>

Got it. i downloaded your platformio project. However, I can only compile env:esp32s3-devkitc otherwise with other envs i get error:

.pio/libdeps/espwroom32/H4AsyncTCP/src/H4AsyncTCP.h:197:21: error: 'ip_addr_t' does not name a type; did you mean

but my board is not esp32s3-devkitc and i cannot upload firmware, and it doesnt compile with board i normally use "az-delivery-devkit-v4" what should I do?

— Reply to this email directly, view it on GitHub https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1826824777, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7JYJAIYQOU7WIDKWQOLYGNTEDAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWHAZDINZXG4 . You are receiving this because you were mentioned.Message ID: @.***>

dr3adx commented 1 year ago

If you don't need TLS it's fine, remove LWIP_ALTCP and everything will be fine. If you do need it, please let me know to check it out at sooner. … On Sun, Nov 26, 2023, 19:13 dr3adx @.> wrote: Please checkout the platformio environment here: https://github.com/hamzahajeir/h4plugins_env And it's important to utilize h4 APIs h4setup and h4UserLoop instead of setup and loop. … <#m-3998183735438878006> On Sun, Nov 26, 2023, 14:29 dr3adx @.> wrote: Hi there, here's the HTTP client library: https://github.com/HamzaHajeir/ArmadilloHTTP https://github.com/HamzaHajeir/ArmadilloHTTP … <#m-6879621497364389165> On Sun, Nov 26, 2023, 06:25 dr3adx @.> wrote: Hi @HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir https://github.com/HamzaHajeir is your library compatible with ESPAsyncWebServer? Hi @johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo https://github.com/johnnytolengo, it's not, really. But the stack comes with a stable H4AsyncWebServer https://github.com/hamzahajeir/h4asyncwebserver https://github.com/hamzahajeir/h4asyncwebserver https://github.com/hamzahajeir/h4asyncwebserver https://github.com/hamzahajeir/h4asyncwebserver that you can check out. hello bro, I want to try your async TCP library but HTTP library is missing? https://github.com/philbowles/H4AsyncHTTP https://github.com/philbowles/H4AsyncHTTP https://github.com/philbowles/H4AsyncHTTP https://github.com/philbowles/H4AsyncHTTP link is not working, got any idea where can I get async HTTP library? — Reply to this email directly, view it on GitHub <#161 <#161> (comment) <#161 (comment) <#161 (comment)>>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA https://github.com/notifications/unsubscribe-auth/AH3O7J3WSZ6JNMCM7OK46RLYGKZEHAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWGQ3TQMJZHA . You are receiving this because you were mentioned.Message ID: @.> how do I get this running in platformio? do I need h4plugins https://github.com/philbowles/h4plugins https://github.com/philbowles/h4plugins ? lib_deps = https://github.com/HamzaHajeir/H4 https://github.com/HamzaHajeir/H4 https://github.com/HamzaHajeir/H4Tools https://github.com/HamzaHajeir/H4Tools https://github.com/HamzaHajeir/H4AsyncTCP@^0.0.18 https://github.com/HamzaHajeir/H4AsyncTCP@%5E0.0.18 https://github.com/HamzaHajeir/ArmadilloHTTP https://github.com/HamzaHajeir/ArmadilloHTTP khoih-prog/ESPAsync_WiFiManager@>=1.15.1 I'm trying to compile example program and I get error during linking: https://pastebin.com/ZgStVidc https://pastebin.com/ZgStVidc — Reply to this email directly, view it on GitHub <#161 (comment) <#161 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J5HVPCFVNVPL4OTZT3YGMR2JAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWG42TQMRRGQ https://github.com/notifications/unsubscribe-auth/AH3O7J5HVPCFVNVPL4OTZT3YGMR2JAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWG42TQMRRGQ . You are receiving this because you were mentioned.Message ID: @.**> Got it. i downloaded your platformio project. However, I can only compile env:esp32s3-devkitc otherwise with other envs i get error: .pio/libdeps/espwroom32/H4AsyncTCP/src/H4AsyncTCP.h:197:21: error: 'ip_addr_t' does not name a type; did you mean but my board is not esp32s3-devkitc and i cannot upload firmware, and it doesnt compile with board i normally use "az-delivery-devkit-v4" what should I do? — Reply to this email directly, view it on GitHub <#161 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7JYJAIYQOU7WIDKWQOLYGNTEDAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMRWHAZDINZXG4 . You are receiving this because you were mentioned.Message ID: @.>

got it, I dont need SSL for now

however I'm trying to make a simple request that returns json (verified) but i cant read the body, its empty

01:43:00.857 > code 200 response json: str:
01:43:00.863 > CONTENT-LENGTH : 20
01:43:00.863 > CONTENT-TYPE : application/json;charset=UTF-8
01:43:00.863 > Publishing  to response

any idea why the response body is empty? both asStdstring and asJsonString return empty. Even tho content-length is 20 i verified request with Postman it returns 20 length json body im using your test file

johnnytolengo commented 1 year ago

My question is how stable is the H4AsyncWebServer and if worth to replace the ESPAsyncWebServer?

As we know the huge problem of the ESPAsyncWebServer is that it crashes very often, and all that crashes are related to the AsyncTCP.h library, isn't it?

If I'm right maybe it's time to make an effort all togeather and fix the AsyncTCP.h in order to make it 100% stable without memory leaks and crashes.

What's the best fork of the AsyncTCP.h out there?

Plan B: switch to H4AsyncWebServer.

HamzaHajeir commented 1 year ago

got it, I dont need SSL for now

however I'm trying to make a simple request that returns json (verified) but i cant read the body, its empty
01:43:00.857 > code 200 response json: str:
01:43:00.863 > CONTENT-LENGTH : 20
01:43:00.863 > CONTENT-TYPE : application/json;charset=UTF-8
01:43:00.863 > Publishing  to response
any idea why the response body is empty? both asStdstring and asJsonString return empty. Even tho content-length is 20 i verified request with Postman it returns 20 length json body im using your test file

Per the ip_addr_t there's apparently a missing header only for esp32 "lwip/ip_addr.h', one can include in H4AsyncTCP.h file.

Per the issue, I've double checked the example, it successfully read the body response:

14:16:18.194 > code 200 response {
14:16:18.195 >   "userId": 1,
14:16:18.195 >   "id": 1,
14:16:18.195 >   "title": "delectus aut autem",
14:16:18.195 >   "completed": false
14:16:18.196 > }
14:16:18.196 > CF-RAY : 82c9f54d6fb235f1-FRA
14:16:18.196 > SERVER : cloudflare
14:16:18.196 > AGE : 15917
14:16:18.196 > ALT-SVC : h3=":443"; ma=86400
14:16:18.197 > CF-CACHE-STATUS : HIT
14:16:18.197 > VIA : 1.1 vegur
14:16:18.197 > ACCEPT-RANGES : bytes
14:16:18.197 > X-CONTENT-TYPE-OPTIONS : nosniff
14:16:18.198 > REPORTING-ENDPOINTS : heroku-nel=https://nel.heroku.com/reports?ts=1701018150&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=NiRy1FnssaN%2F8p4XTViBtH31WNCTyd9UpTCwmbfWAVg%3D
14:16:18.198 > X-RATELIMIT-REMAINING : 999
14:16:18.198 > CONTENT-TYPE : application/json; charset=utf-8
14:16:18.198 > CACHE-CONTROL : max-age=43200
14:16:18.198 > CONNECTION : close
14:16:18.198 > DATE : Mon, 27 Nov 2023 11:16:18 GMT
14:16:18.198 > NEL : {"report_to":"heroku-nel","max_age":3600,"success_fraction":0.005,"failure_fraction":0.05,"response_headers":["Via"]}
14:16:18.198 > CONTENT-LENGTH : 83
14:16:18.198 > PRAGMA : no-cache
14:16:18.198 > ETAG : W/"53-hfEnumeNh6YirfjyjaujcOPPT+s"
14:16:18.198 > X-POWERED-BY : Express
14:16:18.204 > X-RATELIMIT-LIMIT : 1000
14:16:18.204 > REPORT-TO : {"group":"heroku-nel","max_age":3600,"endpoints":[{"url":"https://nel.heroku.com/reports?ts=1701018150&sid=e11707d5-02a7-43ef-b45e-2cf4d2036f7d&s=NiRy1FnssaN%2F8p4XTViBtH31WNCTyd9UpTCwmbfWAVg%3D"}]}
14:16:18.206 > X-RATELIMIT-RESET : 1701018170
14:16:18.207 > VARY : Origin, Accept-Encoding
14:16:18.207 > EXPIRES : -1
14:16:18.207 > ACCESS-CONTROL-ALLOW-CREDENTIALS : true
14:16:18.207 > Publishing {
14:16:18.208 >   "userId": 1,
14:16:18.208 >   "id": 1,
14:16:18.208 >   "title": "delectus aut autem",
14:16:18.208 >   "completed": false
14:16:18.208 > } to response

Seems a different API you're requesting? More info is needed to help.

HamzaHajeir commented 1 year ago

My question is how stable is the H4AsyncWebServer and if worth to replace the ESPAsyncWebServer?

As we know the huge problem of the ESPAsyncWebServer is that it crashes very often, and all that crashes are related to the AsyncTCP.h library, isn't it?

If I'm right maybe it's time to make an effort all togeather and fix the AsyncTCP.h in order to make it 100% stable without memory leaks and crashes.

What's the best fork of the AsyncTCP.h out there?

Plan B: switch to H4AsyncWebServer.

I personally rely over the H4 stack in my product(s), I can tell you that the environment works as a charm*, including HTTP(s) requests, Webserver (even TLS**), and MQTT over TLS communication, with large payloads. completely async.

The testbed was working up to weeks (until I uploaded another sketch or something else), with no memory leak nor crashes.

* As long as there's available memory (ESP8266 is so limited). ** There's an issue that I've experienced for a TLS webserver, and until now it's open: I'm personally not interested because there's no reason for local webserver over TLS, even the routers don't run TLS webservers, MCUs are not a choice for public webservers.

dr3adx commented 12 months ago

My question is how stable is the H4AsyncWebServer and if worth to replace the ESPAsyncWebServer? As we know the huge problem of the ESPAsyncWebServer is that it crashes very often, and all that crashes are related to the AsyncTCP.h library, isn't it? If I'm right maybe it's time to make an effort all togeather and fix the AsyncTCP.h in order to make it 100% stable without memory leaks and crashes. What's the best fork of the AsyncTCP.h out there? Plan B: switch to H4AsyncWebServer.

I personally rely over the H4 stack in my product(s), I can tell you that the environment works as a charm*, including HTTP(s) requests, Webserver (even TLS**), and MQTT over TLS communication, with large payloads. completely async.

The testbed was working up to weeks (until I uploaded another sketch or something else), with no memory leak nor crashes.

As long as there's available memory (ESP8266 is so limited). ** There's an issue that I've experienced for a TLS webserver, and until now it's open: I'm personally not interested because there's no reason for local webserver over TLS, even the routers don't run TLS webservers, MCUs are not a choice for public webservers.

I will play around and let you know regarding JSON. But how can I disable logs like H=229600 M=110580 m=219940 S=5596 I tried defining H4AT_DEBUG 0 in config.h and it's not working, they're killing my console :/ I want to disable all logs in your example main.cpp

HamzaHajeir commented 12 months ago

I will play around and let you know regarding JSON. But how can I disable logs like H=229600 M=110580 m=219940 S=5596 I tried defining H4AT_DEBUG 0 in config.h and it's not working, they're killing my console :/ I want to disable all logs in your example main.cpp

These logs were setup for diagnostic, to disable you can comment out these lines:

    h4.every(300, []()
             {
#if defined(ARDUINO_ARCH_ESP32)
                Serial.printf("H=%u M=%u m=%u S=%u\n", _HAL_freeHeap(MALLOC_CAP_INTERNAL), _HAL_maxHeapBlock(MALLOC_CAP_INTERNAL), _HAL_minHeapBlock(MALLOC_CAP_INTERNAL), uxTaskGetStackHighWaterMark(NULL));
#else
                Serial.printf("H=%u M=%u m=%u\n", _HAL_freeHeap(), _HAL_maxHeapBlock(), _HAL_minHeapBlock());
#endif
                h4p["heap"] = _HAL_freeHeap();
                h4p["pool"] = mbx::pool.size();
                });

Also you can comment out any undesired Serial.printfs.

dr3adx commented 12 months ago

I will play around and let you know regarding JSON. But how can I disable logs like H=229600 M=110580 m=219940 S=5596 I tried defining H4AT_DEBUG 0 in config.h and it's not working, they're killing my console :/ I want to disable all logs in your example main.cpp

These logs were setup for diagnostic, to disable you can comment out these lines:
  h4.every(300, []()
           {
#if defined(ARDUINO_ARCH_ESP32)
              Serial.printf("H=%u M=%u m=%u S=%u\n", _HAL_freeHeap(MALLOC_CAP_INTERNAL), _HAL_maxHeapBlock(MALLOC_CAP_INTERNAL), _HAL_minHeapBlock(MALLOC_CAP_INTERNAL), uxTaskGetStackHighWaterMark(NULL));
#else
              Serial.printf("H=%u M=%u m=%u\n", _HAL_freeHeap(), _HAL_maxHeapBlock(), _HAL_minHeapBlock());
#endif
              h4p["heap"] = _HAL_freeHeap();
              h4p["pool"] = mbx::pool.size();
              });
Also you can comment out any undesired Serial.printfs.

Thanks a lot! Also I further debugged empty JSON response body. I'm banging my head because this is a one liner but its not working. Most websites return proper JSON with your H4 except when I try to GET http://192.168.1.105/app (the Tapo P105 remote switch) on my local network.

However, same request works in Postman and returns proper json body.

{
    "error_code": -1009
}

With your H4 weird thing is that status code returned is correct: 200 and content length is 20 (correct), but both asJsonstring & asStdstring methods return empty string.

My code & output is here: https://pastebin.com/hVtWtzmj Please help bro I really think the problem is in the library somewhere because content length and status code returned are fine.

mathieucarbou commented 12 months ago

@HamzaHajeir : I have an app using AsyncTCP, ESPAsyncWebServer and libraries using WebSocket. So I cannot use the H4 stack, but I was wondering if your AsyncTCP fork (https://github.com/HamzaHajeir/H4AsyncTCP) can be directly used as a replacement of the original AsyncTCP instead ? Thanks !

HamzaHajeir commented 12 months ago

With your H4 weird thing is that status code returned is correct: 200 and content length is 20 (correct), but both asJsonstring & asStdstring methods return empty string.

My code & output is here: https://pastebin.com/hVtWtzmj Please help bro I really think the problem is in the library somewhere because content length and status code returned are fine.

Can you insert those two lines and post the output:

    Serial.printf("data %p len %lu\n", reply.data, reply.length);
    dumphex(reply.data, reply.length);

HamzaHajeir commented 12 months ago

@HamzaHajeir : I have an app using AsyncTCP, ESPAsyncWebServer and libraries using WebSocket. So I cannot use the H4 stack, but I was wondering if your AsyncTCP fork (https://github.com/HamzaHajeir/H4AsyncTCP) can be directly used as a replacement of the original AsyncTCP instead ? Thanks !

Hi Matheiu, I'm really cannot tell exactly, there should be some similarity in APIs (onData/onRX) - (onConnect) - (onError). Philbowles had a different internal design.

Perhaps this comparison sketch shows some differences in APIs, I think one can try to port the aforementioned libraries to H4AsyncTCP and try out, for TCP stuff it's solid and works great up to my experience and knowledge (spent months to fully support it with fixes, but had to take extreme care dealing with TCP object in upper layers (MQTT/HTTP/Webserver) as it's expirable and nullable pointer that should be nullified on every error / disconnect / connectFail).

I'm really unaware of how ESPAsyncWebServer is designed, and whether there's a critical break in dealing to H4AsyncTCP.

Take a look at H4AsyncTCP documentation regarding dealing with the server.

But perhaps changing to H4AsyncWebServer might be easier, take a look at this comment, beside taking a look on how H4Plugins uses it.

Note: Basic authentication was added, and filesystem serving root is fixed (used within H4Plugins)

HamzaHajeir commented 12 months ago

As dealing with H4AsyncTCP as a client, one might take a look how H4 stack uses it:

MQTT.
HTTP.

dr3adx commented 12 months ago

  Serial.printf("data %p len %lu\n", reply.data, reply.length);
  dumphex(reply.data, reply.length);

This is output: data 0x3ffd47c9 len 0

HamzaHajeir commented 12 months ago

    Serial.printf("data %p len %lu\n", reply.data, reply.length);
    dumphex(reply.data, reply.length);

This is output: data 0x3ffd47c9 len 0

In the file dillo_config.h set the debug to 4.

#define ARMA_DEBUG  4

Also in ArmadilloHTTP.cpp:198, insert dumphex(d,s); line.

void ArmadilloHTTP::_rx(const uint8_t* d,size_t s){
    ARMA_PRINT1("RX 0x%08x len=%d FH=%u\n",d,s,_HAL_freeHeap());
    dumphex(d,s);
    ...

What the output would be?

dr3adx commented 12 months ago

  Serial.printf("data %p len %lu\n", reply.data, reply.length);
  dumphex(reply.data, reply.length);
This is output: data 0x3ffd47c9 len 0
In the file dillo_config.h set the debug to 4.
#define ARMA_DEBUG  4
Also in ArmadilloHTTP.cpp:198, insert dumphex(d,s); line.
void ArmadilloHTTP::_rx(const uint8_t* d,size_t s){
    ARMA_PRINT1("RX 0x%08x len=%d FH=%u\n",d,s,_HAL_freeHeap());
    dumphex(d,s);
    ...
What the output would be?

https://pastebin.com/FwCgKFzR

the json {"error_code":-1 displays now in the log at least! just have to find out why its not being returned in asJson method.

HamzaHajeir commented 12 months ago

This is because the webserver sends two separate TCP packets, both holding PUSH flag for immediate processing.

Other stream-based clients does "suck" data out of pool, unlike H4AsyncTCP which is completely event-driven.

Can you checkout Wireshark and validate it's only one TCP packet (not as I've described)?

HamzaHajeir commented 12 months ago

Alternatively, you can just turn H4AT_DEBUG 2 under h4async_config.h (Under H4AsyncTCP) and paste the output, it will print the flags by the following line:

    H4AT_PRINT2("_raw_recv %p data=%p L=%d f=0x%02x \n",rq,cpydata,cpylen,cpyflags);

dr3adx commented 12 months ago

H4AT_PRINT2("_raw_recv %p data=%p L=%d f=0x%02x \n",rq,cpydata,cpylen,cpyflags);

here's the new log: https://pastebin.com/0QnWfkTR

HamzaHajeir commented 12 months ago

H4AT_PRINT2("_raw_recv %p data=%p L=%d f=0x%02x \n",rq,cpydata,cpylen,cpyflags);
here's the new log: https://pastebin.com/0QnWfkTR

Yeah, two packets carrying PUSH flag:

H4AT:1: H=168092 M=110580 S=1196 _raw_recv 0x3ffd4a5c tpcb=0x3ffd4dac p=0x3ffd4f54 err=0 data=0x3ffd5b26 tot_len=85
H4AT:2: H=168004 M=110580 S=1196 * p=0x3ffd4f54 * FREE DATA 0x3ffd5b26 85 0x01 bpp=0x0
H4AT:2: H=170180 M=110580 S=5596 _raw_recv 0x3ffd4a5c data=0x3ffd5000 L=85 f=0x01
H4AT:1: H=170148 M=110580 S=1196 _raw_recv 0x3ffd4a5c tpcb=0x3ffd4dac p=0x3ffd34d8 err=0 data=0x3ffd643a tot_len=20
...
...
...
H4AT:2: H=172148 M=110580 S=5596 _raw_recv 0x3ffd4a5c data=0x3ffd5290 L=20 f=0x01

f=0x01 is PBUF_FLAG_PUSH.

The webserver behaviour should be corrected, either to send one TCP packet including body, or to detach PUSH flag from the first TCP packet.

dr3adx commented 12 months ago

H4AT_PRINT2("_raw_recv %p data=%p L=%d f=0x%02x \n",rq,cpydata,cpylen,cpyflags);
here's the new log: https://pastebin.com/0QnWfkTR
Yeah, two packets carrying PUSH flag:
H4AT:1: H=168092 M=110580 S=1196 _raw_recv 0x3ffd4a5c tpcb=0x3ffd4dac p=0x3ffd4f54 err=0 data=0x3ffd5b26 tot_len=85
H4AT:2: H=168004 M=110580 S=1196 * p=0x3ffd4f54 * FREE DATA 0x3ffd5b26 85 0x01 bpp=0x0
H4AT:2: H=170180 M=110580 S=5596 _raw_recv 0x3ffd4a5c data=0x3ffd5000 L=85 f=0x01
H4AT:1: H=170148 M=110580 S=1196 _raw_recv 0x3ffd4a5c tpcb=0x3ffd4dac p=0x3ffd34d8 err=0 data=0x3ffd643a tot_len=20
...
...
...
H4AT:2: H=172148 M=110580 S=5596 _raw_recv 0x3ffd4a5c data=0x3ffd5290 L=20 f=0x01
f=0x01 is PBUF_FLAG_PUSH.

The webserver behaviour should be corrected, either to send one TCP packet including body, or to detach PUSH flag from the first TCP packet.

Impossible to modify webserver, its a commercial device (Tapo P105) by TP-LINK :/ Is there any way this can be fixed software side? I dont care if its a dirty hack fix I just wanna get this running :/

johnnytolengo commented 12 months ago

mathieucar

@mathieucar Let me know if you were able to replace the original AsyncTCP with the H4AsyncTCP and how you did it with ESPAsyncWebServer.

Thanks.

HamzaHajeir commented 12 months ago

Impossible to modify webserver, its a commercial device (Tapo P105) by TP-LINK :/ Is there any way this can be fixed software side? I dont care if its a dirty hack fix I just wanna get this running :/

Real problem :\, as if they wanted to chunk it out, there's a standardized way to do in HTTP protocol and it's managed within the library*.

The issue is similar to this one except TLS isn't used here, we're out of the standards already.

I've gave it a thought, if to add a workaround to the library, it wont be a library anymore, as it will be custom to solve that case

But what you can do is something will be tough to some extent, is to work with H4AsyncTCP directly, manually write the request, and onRX you manage to combine up the fragments, with a simple trick to find the json by finding its start '{', then copy that to the end of your buffer. You'll need somehow to manage TCP pointer lifetime.

Alternatively you might use a stream-based HTTP client, as what esp32-arduino offers, which seems to me it solves your current problem if you don't really bother whether it's async or not.

* There's a current bug for handling very large responses.

dr3adx commented 12 months ago

Impossible to modify webserver, its a commercial device (Tapo P105) by TP-LINK :/ Is there any way this can be fixed software side? I dont care if its a dirty hack fix I just wanna get this running :/

Real problem :, as if they wanted to chunk it out, there's a standardized way to do in HTTP protocol and it's managed within the library*.

The issue is similar to this one except TLS isn't used here, we're out of the standards already.

I've gave it a thought, if to add a workaround to the library, it wont be a library anymore, as it will be custom to solve that case

But what you can do is something will be tough to some extent, is to work with H4AsyncTCP directly, manually write the request, and onRX you manage to combine up the fragments, with a simple trick to find the json by finding its start '{', then copy that to the end of your buffer. You'll need somehow to manage TCP pointer lifetime.

Alternatively you might use a stream-based HTTP client, as what esp32-arduino offers, which seems to me it solves your current problem if you don't really bother whether it's async or not.

There's a current bug for handling very large responses.

well its not really a "custom" problem, it works just fine in Postman and other HTTP clients how can i chain multiple async requests with h4? i need to check every IP on the subnet, so 255 requests, ill just use status code since json body isnt working

I tried queueing 255 reqs using h4.queue function but not a single req executes :O

HamzaHajeir commented 12 months ago

well its not really a "custom" problem, it works just fine in Postman and other HTTP clients

Other HTTP clients are either stream-based, or event-driven. I don't think this is a problem for Stream-based clients to a good extent, but if event driven they might insert the workarounds outside the standards. Some of other clients can do more advanced workarounds to deal with uncompliant servers, as correcting the order of out-of-order HTTP chunks (even worse than that), which happened in my webserver and found the browser just fixes that, before I fixed the bug here ofcourse. If I didn't fix it and you were to request by a moderate client, you will find it very hard to parse, as it will be a bunch of scrambled data to you.

Per the current issue, for any TCP packet which holds PUSH flag, it tells you "I'm finished, please process the message immediately", thus the client receives the pushed message and process it as per the HTTP standard, in your case the sender does just lie :).

how can i chain multiple async requests with h4? i need to check every IP on the subnet, so 255 requests, ill just use status code since json body isnt working I tried queueing 255 reqs using h4.queue function but not a single req executes :O

I'm not sure what you're aiming for, seems want to discover it? The common way to do is via mDNS, can you check the Tapo P105 documentation for such a feature?

Per requests, at least some does and the other just don't (you'll need to add callbacks onConnectFail/onError/onDisconnect to know why); but primarily because there's a limit of 16 simultaneous active pcbs (protocol control blocks) in ESP32.

dr3adx commented 12 months ago

well its not really a "custom" problem, it works just fine in Postman and other HTTP clients

Other HTTP clients are either stream-based, or event-driven. I don't think this is a problem for Stream-based clients to a good extent, but if event driven they might insert the workarounds outside the standards. Some of other clients can do more advanced workarounds to deal with uncompliant servers, as correcting the order of out-of-order HTTP chunks (even worse than that), which happened in my webserver and found the browser just fixes that, before I fixed the bug here ofcourse. If I didn't fix it and you were to request by a moderate client, you will find it very hard to parse, as it will be a bunch of scrambled data to you.

Per the current issue, for any TCP packet which holds PUSH flag, it tells you "I'm finished, please process the message immediately", thus the client receives the pushed message and process it as per the HTTP standard, in your case the sender does just lie :).

how can i chain multiple async requests with h4? i need to check every IP on the subnet, so 255 requests, ill just use status code since json body isnt working I tried queueing 255 reqs using h4.queue function but not a single req executes :O

I'm not sure what you're aiming for, seems want to discover it? The common way to do is via mDNS, can you check the Tapo P105 documentation for such a feature?

Per requests, at least some does and the other just don't (you'll need to add callbacks onConnectFail/onError/onDisconnect to know why); but primarily because there's a limit of 16 simultaneous active pcbs (protocol control blocks) in ESP32.

can 1 pcb only handle 1 simultaneous request at a time? or what does this mean in terms of performance?

also device im trying to ping doesnt use mDNS service so i have to http request every IP on subnet (for discovery), how would I go about doing this efficiently with H4?

HamzaHajeir commented 12 months ago

Each one controls only one TCP connection.

On Thu, Nov 30, 2023, 17:27 dr3adx @.***> wrote:

well its not really a "custom" problem, it works just fine in Postman and other HTTP clients

Other HTTP clients are either stream-based, or event-driven. I don't think this is a problem for Stream-based clients to a good extent, but if event driven they might insert the workarounds outside the standards. Some of other clients can do more advanced workarounds to deal with uncompliant servers, as correcting the order of out-of-order HTTP chunks (even worse than that), which happened in my webserver and found the browser just fixes that, before I fixed the bug here ofcourse. If I didn't fix it and you were to request by a moderate client, you will find it very hard to parse, as it will be a bunch of scrambled data to you.

Per the current issue, for any TCP packet which holds PUSH flag, it tells you "I'm finished, please process the message immediately", thus the client receives the pushed message and process it as per the HTTP standard, in your case the sender does just lie :).

how can i chain multiple async requests with h4? i need to check every IP on the subnet, so 255 requests, ill just use status code since json body isnt working I tried queueing 255 reqs using h4.queue function but not a single req executes :O

I'm not sure what you're aiming for, seems want to discover it? The common way to do is via mDNS, can you check the Tapo P105 documentation for such a feature?

Per requests, at least some does and the other just don't (you'll need to add callbacks onConnectFail/onError/onDisconnect to know why); but primarily because there's a limit of 16 simultaneous active pcbs (protocol control blocks) in ESP32.

can 1 pcb only handle 1 simultaneous request at a time? or what does this mean in terms of performance?

— Reply to this email directly, view it on GitHub https://github.com/me-no-dev/AsyncTCP/issues/161#issuecomment-1833882212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AH3O7J2F6FNYAAXLYWNOR63YHCJVDAVCNFSM6AAAAAAVJWOJLKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZTHA4DEMRRGI . You are receiving this because you were mentioned.Message ID: @.***>

HamzaHajeir commented 4 months ago

well its not really a "custom" problem, it works just fine in Postman and other HTTP clients

Other HTTP clients are either stream-based, or event-driven. I don't think this is a problem for Stream-based clients to a good extent, but if event driven they might insert the workarounds outside the standards. Some of other clients can do more advanced workarounds to deal with uncompliant servers, as correcting the order of out-of-order HTTP chunks (even worse than that), which happened in my webserver and found the browser just fixes that, before I fixed the bug here ofcourse. If I didn't fix it and you were to request by a moderate client, you will find it very hard to parse, as it will be a bunch of scrambled data to you. Per the current issue, for any TCP packet which holds PUSH flag, it tells you "I'm finished, please process the message immediately", thus the client receives the pushed message and process it as per the HTTP standard, in your case the sender does just lie :).

how can i chain multiple async requests with h4? i need to check every IP on the subnet, so 255 requests, ill just use status code since json body isnt working I tried queueing 255 reqs using h4.queue function but not a single req executes :O

I'm not sure what you're aiming for, seems want to discover it? The common way to do is via mDNS, can you check the Tapo P105 documentation for such a feature? Per requests, at least some does and the other just don't (you'll need to add callbacks onConnectFail/onError/onDisconnect to know why); but primarily because there's a limit of 16 simultaneous active pcbs (protocol control blocks) in ESP32.

can 1 pcb only handle 1 simultaneous request at a time? or what does this mean in terms of performance?

also device im trying to ping doesnt use mDNS service so i have to http request every IP on subnet (for discovery), how would I go about doing this efficiently with H4?

Hi. I'm just reading the thread again. Hope you have managed to solve your issue :-)

mathieucarbou commented 4 months ago

FYI I am maintaining since a while now (with some help):

AsyncTCP : https://github.com/mathieucarbou/AsyncTCP
ESPAsyncWebServer: https://github.com/mathieucarbou/ESPAsyncWebServer

These forks are used in some big projects out there (OpenDTU, Beelance, YaSolR, ESP-DASH, ElegantOTA, ESPConnect, WebSerial, etc). I have an app doing a LOT of websocket calls without any issue (at a rate of about 3-4 push per sec).

I've merged inside a couple of PRs standing there, plus added Arduino 3 support, ArduinoJson 7, IPv6 (improved from ESPHome draft), etc

You might want to check them out.

https://github.com/HamzaHajeir/H4AsyncTCP is not an option because of its too restrictive license.

HamzaHajeir commented 4 months ago

FYI I am maintaining since a while now (with some help):

AsyncTCP : https://github.com/mathieucarbou/AsyncTCP

ESPAsyncWebServer: https://github.com/mathieucarbou/ESPAsyncWebServer

These forks are used in some big projects out there (OpenDTU, Beelance, YaSolR, ESP-DASH, ElegantOTA, ESPConnect, WebSerial, etc). I have an app doing a LOT of websocket calls without any issue (at a rate of about 3-4 push per sec).

I've merged inside a couple of PRs standing there, plus added Arduino 3 support, ArduinoJson 7, IPv6 (improved from ESPHome draft), etc

You might want to check them out.

https://github.com/HamzaHajeir/H4AsyncTCP is not an option because of its too restrictive license.

Glad to hear you've fixed your issue.

Per the license, the author has passed in June 2022, leaving the work in the middle, then I took the lead to complete his work and make it useful to the world, I tried to get to his family (only one sister) regarding the license, and other people tried to reach her, but with no luck, she seems not so engaged on the internet and we have no other contact method.

Opening the license to the world is an option if we could not reach her, but it's the last resort for me.

BTW I'm running a reliable sketch that publishes 13k of MQTTv5.0 over TLS every 3 seconds, with running HTTPS and WSS, alongwith HTTPs Client that fetches every 15 seconds a TLS HTTP API.

me-no-dev / AsyncTCP

Crash on multiple async reqs #161