Really nice!

[hitecSmartHome]

That is a really usefull project. You are a genious!

Can this work with react or do you plan to create a react version of this?

[mhaberler]

yes it does work with React just fine

it's mentioned right in the repo headline

[hitecSmartHome]

Oh wonderfull. Will check it out for sure! Thanks

[BCsabaEngine]

Please check and give a feedback.

[hitecSmartHome]

It seems to work. Very well put together. I can even configure the initialisation function name. Wonderfull! I did not tested it yet, just compiled a basic react app for now. Will give additional feedback asap!

[hitecSmartHome]

Oh and I can easily copy and paste the build command to my build script with vite. Really nice indeed! I only have to type npm run build and it builds my app to the dist dir, and it also runs your script and makes a header file to the desired destination. I can't imagine better flow!

[BCsabaEngine]

I am so proud and happy to serve your requirement. Nice to hear about your success.

[hitecSmartHome]

I have to make a programmer cable and will test it on a device. Will give feedback!

[BCsabaEngine]

Please check etag usage, your browser and esp will use cached files, try out simple page reload and full reload. It works, I think.

[hitecSmartHome]

I was able to compile and upload my sketch. For some reason I got garbage data on my browser. I think there is no "/" handler

[hitecSmartHome]

(11456) psychic: New client connected 57
(11458) httpd_uri: httpd_uri: URI '/' not found
[0;33mW (11759) httpd_uri: httpd_uri: URI '/favicon.ico' not found
[0;33mW (16777) httpd_uri: httpd_uri: URI '/' not found
[0;33mW (17187) httpd_uri: httpd_uri: URI '/favicon.ico' not found

[hitecSmartHome]

I have added this endpoint after I called init

server.on("/", HTTP_GET, [](PsychicRequest * request) {
        if (request->hasHeader("If-None-Match") && request->header("If-None-Match") == String(etag2)) {
            PsychicResponse response304(request);
            response304.setCode(304);
            return response304.send();
        }
        PsychicResponse response(request);
        response.setContentType("text/html");
        response.addHeader("Content-Encoding", "gzip");
        response.addHeader("ETag", etag2);
        response.setContent(data2, sizeof(data2));
        return response.send();
    });

But no help. I got garbage

[hitecSmartHome]

Strange. I have added a plugin to my Vite config so it is compressing my files. The compression left the original files in the dist folder. I think your script compiled the regular files because when I excluded the original files and left in only the zipped (.gz) files your script did not see these.

import path from "path"
import react from "@vitejs/plugin-react-swc"
import { defineConfig } from "vite"
import { compression } from 'vite-plugin-compression2'

export default defineConfig({
  plugins: [
    react(),
    compression({
      deleteOriginalAssets: true
    })
  ],
  resolve: {
    alias: {
      "@": path.resolve(__dirname, "./src"),
    },
  },
})

This generated a dist folder with an index.html.gz inside and the script created only a png endpoint

//engine:   PsychicHttpServer
//cmdline:  -e psychic -s ../frontend/dist -o ../src/webServer/frontend.h --etag --espmethod initFrontend
//created:  2024. 07. 26. 1:13:16
//files:    1
//memory:   564

#include <Arduino.h>
#include <PsychicHttp.h>
#include <PsychicHttpsServer.h>

const uint8_t data0[564] = { 0x89, 0x50, 0x4e, 0x47, 0xd, 0xa, 0x1a, 0xa, 0x0, 0x0, 0x0, 0xd, 0x49, 0x48, 0x44, 0x52, 0x0, 0x0, 0x0, 0xc0, 0x0, 0x0, 0x0, 0xc0, 0x4, 0x3, 0x0, 0x0, 0x0, 0xa0, 0xf2, 0x71, 0x34, 0x0, 0x0, 0x0, 0x4, 0x67, 0x41, 0x4d, 0x41, 0x0, 0x0, 0xb1, 0x8f, 0xb, 0xfc, 0x61, 0x5, 0x0, 0x0, 0x0, 0x1, 0x73, 0x52, 0x47, 0x42, 0x0, 0xae, 0xce, 0x1c, 0xe9, 0x0, 0x0, 0x0, 0x18, 0x50, 0x4c, 0x54, 0x45, 0xff, 0xff, 0xff, 0x34, 0xbd, 0xeb, 0x48, 0xc4, 0xed, 0xe7, 0xf7, 0xfd, 0xca, 0xee, 0xfa, 0x7c, 0xd4, 0xf2, 0x59, 0xc9, 0xee, 0xa1, 0xe0, 0xf6, 0xf9, 0xd5, 0xa7, 0xdb, 0x0, 0x0, 0x1, 0x57, 0x49, 0x44, 0x41, 0x54, 0x78, 0xda, 0xed, 0xda, 0x31, 0x6b, 0xc2, 0x40, 0x18, 0xc6, 0xf1, 0x17, 0x2c, 0x74, 0x36, 0x43, 0x3a, 0x47, 0xd0, 0x59, 0x50, 0x70, 0x2d, 0x6d, 0xa5, 0x73, 0xc5, 0xc4, 0xd9, 0x42, 0x9c, 0xb, 0xd6, 0xef, 0xdf, 0x9a, 0xa1, 0xd5, 0xdc, 0xe5, 0xf2, 0xbe, 0x49, 0xce, 0xe, 0xfd, 0x3f, 0xab, 0xf1, 0x7e, 0xf0, 0x3c, 0xd3, 0xc1, 0xc9, 0x38, 0x72, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xe0, 0x4f, 0x80, 0xe9, 0x29, 0x8b, 0x9, 0xcc, 0xb6, 0x73, 0x59, 0x16, 0x59, 0x2c, 0x20, 0xd9, 0x3d, 0xcb, 0x77, 0x46, 0xaf, 0x9b, 0x28, 0x40, 0x52, 0x56, 0xc7, 0x57, 0xc4, 0xfa, 0x7d, 0x78, 0x60, 0xfa, 0x39, 0x97, 0xdf, 0x2c, 0xd5, 0x53, 0x88, 0xbe, 0xfc, 0xeb, 0x68, 0xa7, 0x10, 0x43, 0xf9, 0xd7, 0x51, 0x4e, 0x21, 0xa6, 0xf2, 0x6b, 0x84, 0x66, 0xa, 0x31, 0x96, 0x5f, 0xeb, 0xa9, 0x7d, 0xa, 0x31, 0x97, 0x5f, 0x9f, 0xa2, 0x17, 0xe0, 0x2d, 0xdf, 0x36, 0x45, 0x10, 0x28, 0x5b, 0x8f, 0xaf, 0x88, 0xd3, 0xbe, 0x1b, 0x10, 0x2a, 0x5f, 0x3f, 0x45, 0x23, 0x90, 0xee, 0xb4, 0xc7, 0x9f, 0xb3, 0x2a, 0x8c, 0x80, 0xa2, 0x7c, 0xe5, 0x14, 0x7e, 0xa0, 0x3c, 0x8a, 0x39, 0xd, 0x53, 0x48, 0xaf, 0xf2, 0x15, 0x53, 0x48, 0xcf, 0xf2, 0x5b, 0xa7, 0xb8, 0x0, 0xca, 0xfc, 0x1c, 0x6b, 0xf9, 0xf5, 0x29, 0xaa, 0x43, 0xa, 0x2f, 0xf0, 0x21, 0x83, 0xe5, 0x1e, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x20, 0x22, 0x70, 0x5c, 0x34, 0xc7, 0x73, 0x48, 0xe0, 0xeb, 0xc5, 0xca, 0xb, 0x4c, 0x9a, 0x33, 0x7b, 0x74, 0x81, 0x4d, 0xe0, 0xfb, 0x89, 0xf9, 0x1a, 0xeb, 0x1, 0xde, 0x6, 0xbd, 0x27, 0x3, 0x0, 0xfc, 0x17, 0x20, 0x39, 0x38, 0xd9, 0x6b, 0x0, 0xf7, 0x6f, 0x7, 0x3f, 0xf0, 0xe0, 0x1e, 0xf2, 0xa4, 0x0, 0x52, 0xf7, 0xce, 0x38, 0xca, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6e, 0xa, 0xdc, 0x51, 0x11, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x40, 0x27, 0xe0, 0xc5, 0x8d, 0x1f, 0x48, 0x73, 0x27, 0x3f, 0x8f, 0x33, 0xdc, 0x9f, 0xf2, 0x2c, 0xf0, 0x74, 0xcb, 0xfe, 0xf4, 0xa4, 0x7b, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6e, 0x4, 0x7c, 0x1, 0x6d, 0xb9, 0x7f, 0x1d, 0x5a, 0xf1, 0x41, 0x90, 0x0, 0x0, 0x0, 0x57, 0x7a, 0x54, 0x58, 0x74, 0x52, 0x61, 0x77, 0x20, 0x70, 0x72, 0x6f, 0x66, 0x69, 0x6c, 0x65, 0x20, 0x74, 0x79, 0x70, 0x65, 0x20, 0x69, 0x70, 0x74, 0x63, 0x0, 0x0, 0x78, 0x9c, 0xe3, 0xf2, 0xc, 0x8, 0x71, 0x56, 0x28, 0x28, 0xca, 0x4f, 0xcb, 0xcc, 0x49, 0xe5, 0x52, 0x0, 0x3, 0x23, 0xb, 0x2e, 0x63, 0xb, 0x13, 0x23, 0x13, 0x4b, 0x93, 0x14, 0x3, 0x13, 0x20, 0x44, 0x80, 0x34, 0xc3, 0x64, 0x3, 0x23, 0xb3, 0x54, 0x20, 0xcb, 0xd8, 0xd4, 0xc8, 0xc4, 0xcc, 0xc4, 0x1c, 0xc4, 0x7, 0xcb, 0x80, 0x48, 0xa0, 0x4a, 0x2e, 0x0, 0xea, 0x17, 0x11, 0x74, 0xf2, 0x42, 0x35, 0x95, 0x0, 0x0, 0x0, 0x0, 0x49, 0x45, 0x4e, 0x44, 0xae, 0x42, 0x60, 0x82 };
const char * etag0 = "e31457aaaac4a1a5483a8c219bc7cd56";

void initFrontend(PsychicHttpServer * server) {

  server->on("/littleLogo.png", HTTP_GET, [](PsychicRequest * request)
  {
    if (request->hasHeader("If-None-Match") && request->header("If-None-Match") == String(etag0)) {
      PsychicResponse response304(request);
      response304.setCode(304);
      return response304.send();
    }
    PsychicResponse response(request);
    response.setContentType("image/png");
    response.addHeader("ETag", etag0);
    response.setContent(data0, sizeof(data0));
    return response.send();
  });
}

[hitecSmartHome]

In my understanding, there is no need for compression since your lib does it already. But I see garbage data on my website.

[mhaberler]

@BCsabaEngine could you recommend some simple platformio project using svelteesp32 and ESPAsyncWebserver?

I also ran into some funnies with gzip compression and PsychicHTTP and I am fairly sure it's an issue with PsychicHTTP header handling

thanks for the great work, appreciated!

[hitecSmartHome]

I had my fights with gzip even when used the file system. I think the file name will be the problem. https://github.com/hoeken/PsychicHttp/issues/63

[mhaberler]

I had my fights with gzip even when used the file system. I think the file name will be the problem. hoeken/PsychicHttp#63

what I observed:

https, no compression/etags: works OK - headers from Chrome debug session:

https, gzip compression, no etags:

NB the weird line " : L" in the response header

the generated code looks perfectly fine:

  server->on("/svelte/index.html", HTTP_GET, [](PsychicRequest * request)
  {
    PsychicResponse response(request);
    response.setContentType("text/html");
    response.addHeader("Content-Encoding", "gzip");
    response.setContent(data3, sizeof(data3));
    return response.send();
  });

looks to me like the response.addHeader() just failed and added random garbage

suspice an undetected malloc failure here: https://github.com/mhaberler/PsychicHttp/blob/arduino3/src/PsychicResponse.cpp#L25-L36

[hitecSmartHome]

I can confirm that it works if I don't gzip the content. I added --no-gzip and it works.

[hitecSmartHome]

eTag doesnt work for me. It downloads the file all the time, does not cache it.

[mhaberler]

well if you cant send a header like "Content-Encoding : gzip" properly then there's a high chance something similar happens to the etags header

so get that working first

[hitecSmartHome]

Added some debug prints

void PsychicResponse::addHeader(const char *field, const char *value)
{
  //these get freed during send()
  HTTPHeader header;
  header.field = (char *)malloc(strlen(field)+1);
  header.value = (char *)malloc(strlen(value)+1);

  strlcpy(header.field, field, strlen(field)+1);
  strlcpy(header.value, value, strlen(value)+1);

  printf("Adding header: %s: %s\n", header.field, header.value);

  _headers.push_back(header);
}

void PsychicResponse::sendHeaders()
{
  //get our global headers out of the way first
  for (HTTPHeader header : DefaultHeaders::Instance().getHeaders())
    httpd_resp_set_hdr(_request->request(), header.field, header.value);

  //now do our individual headers
  for (HTTPHeader header : _headers){
    printf("Sending header: %s: %s\n", header.field, header.value);
    esp_err_t resp = httpd_resp_set_hdr(this->_request->request(), header.field, header.value);

    if (resp != ESP_OK){
      printf("Failed to set header: %s: %s. Error: %s\n", header.field, header.value, esp_err_to_name(resp));
    }
  }

  // clean up our header variables after send
  for (HTTPHeader header : _headers)
  {
    free(header.field);
    free(header.value);
  }
  _headers.clear();
}

[hitecSmartHome]

It seems to work

(9511) psychic: New client connected 51
(9513) httpd_uri: httpd_uri: URI '/' not found
Adding header: Content-Encoding: gzip
Adding header: ETag: cf77e49b1f78c9721d686cef353dfaf0
Sending header: Content-Encoding: gzip
Sending header: ETag: cf77e49b1f78c9721d686cef353dfaf0
(10015) httpd_uri: httpd_uri: URI '/favicon.ico' not found
Adding header: Content-Encoding: gzip
Adding header: ETag: cf77e49b1f78c9721d686cef353dfaf0
Sending header: Content-Encoding: gzip
Sending header: ETag: cf77e49b1f78c9721d686cef353dfaf0

[mhaberler]

did same, nothing unusual

then added -DSTREAM_CHUNK_SIZE=1450 to build_flags

The header is chopped a tad differently - NB it's now " : gzip"

the whole thing is reversible

[mhaberler]

we are barking up the wrong repo here ;)

[hitecSmartHome]

I also got garbage

[hitecSmartHome]

DefaultHeaders are working

[hitecSmartHome]

I have also created an issue :D https://github.com/hoeken/PsychicHttp/issues/108

[mhaberler]

let's continue with yours

[hitecSmartHome]

I'm gonna close this now. Thank you very much @BCsabaEngine for your work!

[BCsabaEngine]

So if I understand correctly, you can use my lib with an updated/patched physic lib?

[hitecSmartHome]

Yes! It works wondedfully. Will check with a massive react app soon