search

esphome / issues

Issue Tracker for ESPHome
https://esphome.io/
291 stars 34 forks source link

http_request.get broken(?) #5949

Open DJake42 opened 1 week ago

DJake42 commented 1 week ago

The problem

http_request.get seems to be somewhat broken (at least for me) especially in combination with max_response_buffer_size

using the http_request.get function without max_response_buffer_size causes the esp to hang, and then it just spams [02:38:44][E][http_request.arduino:132]: Stream pointer vanished! until it crashes

printing out the body via

char *ptr;
ptr = (char*)body.c_str();
ESP_LOGD("custom", "ptr: %s\n", ptr);

also shows that there are some weird characters in the beginning:

[12:59:25][D][custom:349]: ptr: a8

{"code":"XXX","product":{"brands":"XXX","code":"XXX","product_name":"XXX"},"status":1,"status_verbose":"product found"}

after the get call the body should only be composed of the json at the end, no idea where the "a8" and the line brak comes from

Also max_response_buffer_size does not seem to work as described in the documentation, for me, the number defined for max_response_buffer_size defines the number of characters the body holds, instead of the size of it in kB as mentioned in the doc

If I do this: max_response_buffer_size: 24

Instead of what I showed above, the log will show

[02:55:05][D][custom:349]: ptr: a8

{"code":"XXXXXXXXXXX

If I put in 28 instead of 24 it will show 4 more characters and so on. Increasing the size to over around 200 will result in a crash of the esp.

(I am not a C++ dev and this is my first time opening an issue on github, sorry if I messed anything up)

Which version of ESPHome has the issue?

2024.6.1

What type of installation are you using?

Home Assistant Add-on

Which version of Home Assistant has the issue?

2024.6.3

What platform are you using?

ESP32

Board

wemos_d1_mini32

Component causing the issue

http_request

Example YAML snippet

substitutions:
  friendly_name: "Barcode-Scanner-Custom"

  # Query ID for opentindb (https://opengtindb.org/api.php)
  rest_opentindb_queryid: "XXX"

  # I2C-Pins for Display
  display_i2c_sda: GPIO21
  display_i2c_scl: GPIO22

  # I2C-Adress OLED Display
  ssd1306_i2c_address: "0x3C"

  # UART-Pins for Barcode-Scanner
  uart_tx_pin: GPIO09
  uart_rx_pin: GPIO10

  # Pin Buzzer/Beeper
  beeper_pin: GPIO16

esphome:
  name: barcode-scanner-custom
  friendly_name: Barcode-Scanner-Custom
  on_boot:
    then:
      - lambda: id(last_ean).publish_state("Scan Barcode");

api:
  encryption:
    key: !secret api_key
  services:
    - service: request_ean
      variables:
        eancode: string
      then:
        - script.execute:
            id: request_ean
            ean: !lambda return eancode.c_str();

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

esp32:
  board: wemos_d1_mini32

# ----------------------------------
# ----       CONFIG END         ----
# ----------------------------------

ota:
- platform: esphome
  password: !secret ota_pw

improv_serial:

captive_portal:

logger:
  hardware_uart: UART0

# ---------------------------------------------

globals:
  - id: xpos_ean
    type: int
    initial_value: "0"
  - id: xpos_brand
    type: int
    initial_value: "0"
  - id: xpos_prod
    type: int
    initial_value: "0"
  - id: safetyCheck
    type: bool
    initial_value: 'true'

text_sensor:
  - platform: template
    id: barcode_scanner
    name: "${friendly_name} Sensor"
    on_value:
      then:
        - if:
            condition:
                and:
                    - lambda: 'return id(barcode_scanner).state != "unknown";'
                    - lambda: 'return id(safetyCheck) == true;'
            then:
              - script.execute: 
                  id: request_ean
                  ean: !lambda return x.c_str();

  - platform: template
    id: last_ean
    name: "${friendly_name} EAN"
  - platform: template
    id: last_brand
    name: "${friendly_name} Brand"
  - platform: template
    id: last_product
    name: "${friendly_name} Product"

uart:
  id: uart_bus
  baud_rate: 9600
  tx_pin: "${uart_tx_pin}"
  rx_pin: "${uart_rx_pin}"
  debug:
    direction: BOTH
    dummy_receiver: true
    after:
      delimiter: "\n"
    sequence:
      - lambda: UARTDebug::log_string(direction, bytes);
      - text_sensor.template.publish:
          id: barcode_scanner
          state: !lambda |
            std::string s(bytes.begin(), bytes.end());
            return s;

font:
  - file: "_fonts/arial.ttf"
    id: arial_font_18
    size: 18
  - file: "_fonts/arial.ttf"
    id: arial_font
    size: 20
    glyphs:  |
      ß!?"%()+*=,-_.:°ø0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZÄÜÖ abcdefghijklmnopqrstuvwxyzäüö€@<>/

i2c:
  sda: "${display_i2c_sda}"
  scl: "${display_i2c_scl}"
  scan: false
  frequency: 400kHz
  id: i2c_bus

display:
  - platform: ssd1306_i2c
    model: "SSD1306 128x64"
    address: "${ssd1306_i2c_address}"
    update_interval: 250ms
    lambda: |-
      it.print(id(xpos_ean), 0, id(arial_font_18), id(last_ean).state.c_str());
      it.print(id(xpos_brand), 19, id(arial_font), id(last_brand).state.c_str());
      it.print(id(xpos_prod), 40, id(arial_font), id(last_product).state.c_str());

      int x_start, y_start, height;
      int brand_width, prod_width, ean_width;
      it.get_text_bounds(0, 0, id(last_ean).state.c_str(), id(arial_font), TextAlign::TOP_LEFT, &x_start, &y_start, &ean_width, &height);
      it.get_text_bounds(0, 0, id(last_brand).state.c_str(), id(arial_font), TextAlign::TOP_LEFT, &x_start, &y_start, &brand_width, &height);
      it.get_text_bounds(0, 0, id(last_product).state.c_str(), id(arial_font), TextAlign::TOP_LEFT, &x_start, &y_start, &prod_width, &height);

      if (id(xpos_ean) > (-1*ean_width)-30 && ean_width>128) {
        id(xpos_ean) -= 10;
      } else {
        id(xpos_ean) = 0;
      }

      if (id(xpos_brand) > (-1*brand_width)-30 && brand_width>128) {
        id(xpos_brand) -= 10;
      } else {
        id(xpos_brand) = 0;
      }

      int prod_len = strlen(id(last_product).state.c_str());
      if (id(xpos_prod) > (-1*prod_width)-30 && prod_width>128) {
        id(xpos_prod) -= 10;
      } else {
        id(xpos_prod) = 0;
      }

switch:
  - platform: gpio
    name: "${friendly_name} Buzzer"
    pin: "${beeper_pin}"
    id: beeper

http_request:
  useragent: esphome/barcodescanner
  timeout: 10s
  id: http_request_data
  verify_ssl: false

script:
  - id: request_ean
    parameters:
      ean: std::string
    mode: single
    then:
      - globals.set:
          id: safetyCheck
          value: 'false'
      - switch.turn_on: beeper
      - delay: 50ms
      - switch.turn_off: beeper
      - script.execute: 
          id: getEanData
          ean: !lambda return ean.c_str();
      - delay: 500ms
      - text_sensor.template.publish:
          id: barcode_scanner
          state: unknown

  - id: getEanData
    parameters:
      ean: std::string
    mode: single
    then:
      - script.execute: clearProductData
      - lambda: |-
          ESP_LOGD("getEanData", "Scanned EAN: %s\n", ean);
      - lambda: id(last_ean).publish_state(ean);
      - script.execute: 
          id: refreshProductNameOpenfoodfacts
          ean: !lambda return ean.c_str();
      - lambda: |-
          ESP_LOGD("getEanData", "Openfoodfacts result: %s\n", id(last_product).state.c_str());
      - delay: 500ms
      - if:
          condition:
            or:
              - text_sensor.state:
                  id: last_product
                  state: 'null'
              - text_sensor.state:
                  id: last_product
                  state: ''
          then:
            - script.execute: 
                id: refreshProductNameOpenTin
                ean: !lambda return ean.c_str();
            - lambda: |-
                ESP_LOGD("getEanData", "OpenTinDb result: %s\n", id(last_product).state.c_str());
      - delay: 2s
      - if:
          condition:
            text_sensor.state:
              id: last_product
              state: 'null'
          then:
              - switch.turn_on: beeper
              - delay: 1000ms
              - switch.turn_off: beeper
          else:
              - switch.turn_on: beeper
              - delay: 50ms
              - switch.turn_off: beeper
      - delay: 2s
      - globals.set:
          id: safetyCheck
          value: 'true'
      - script.execute: resetDisplay

  - id: resetDisplay
    mode: restart
    then:
      - delay: 10s
      - script.execute: clearProductData
      - lambda: id(last_ean).publish_state("Scan Barcode");

  - id: clearProductData
    mode: single
    then: 
      - lambda: id(last_ean).publish_state("");
      - lambda: id(last_brand).publish_state("");
      - lambda: id(last_product).publish_state("");

  - id: refreshProductNameOpenTin
    mode: single
    parameters:
      ean: std::string
    then:
      - http_request.get:
          capture_response: true
          max_response_buffer_size: 172
          url: !lambda |-
                  return (std::string) "https://opengtindb.org/?ean=" + ean + "&cmd=query&queryid=${rest_opentindb_queryid}";
          on_response:
            then:
              - lambda: |-
                  char delimiter[] = "\n";
                  char *ptr;
                  ptr = strtok((char*)body.c_str(), delimiter);
                  ESP_LOGD("opengtin", "ptr: %s\n", ptr);
                  int i = 0;
                  char *name_line = NULL;
                  char *vendor_line = NULL;
                  while(ptr != NULL) {
                    if(i == 4){
                      name_line = ptr;
                    }
                    if(i == 5){
                      vendor_line = ptr;
                    }                    
                    ptr = strtok(NULL, delimiter);
                    i++;
                  }
                  ptr = strtok(name_line, {"="});
                  i=0;
                  while(ptr != NULL) {
                    if(i == 1){
                      ESP_LOGD("opengtin", "Name: %s\n", ptr);
                      id(last_product).publish_state(ptr);
                    }                    
                    ptr = strtok(NULL, delimiter);
                    i++;
                  }
                  ptr = strtok(vendor_line, {"="});
                  i=0;
                  while(ptr != NULL) {
                    if(i == 1){
                      ESP_LOGD("opengtin", "Firma: %s\n", ptr);
                      id(last_brand).publish_state(ptr);
                    }                    
                    ptr = strtok(NULL, delimiter);
                    i++;
                  }

  - id: refreshProductNameOpenfoodfacts
    mode: single
    parameters:
      ean: std::string
    then:
      - http_request.get:
          capture_response: true
          max_response_buffer_size: 172
          headers:
            Content-Type: application/json          
          url: !lambda |-      
            std::string url = "https://world.openfoodfacts.org/api/v2/product/" + ean + "?fields=brands,product_name,code";
            ESP_LOGD("openFood", "API: %s\n", url.c_str());
            return url;
          on_response:
            then:
              - if:
                  condition:
                     lambda: |-
                        return response->status_code == 200;
                  then:              
                    - lambda: |-
                        char *ptr;
                        ptr = (char*)body.c_str();
                        ESP_LOGD("custom", "ptr: %s\n", ptr);
                        json::parse_json(ptr, [](JsonObject root) -> bool {
                        std::string jsonStr;
                        id(last_brand).publish_state(root["product"]["brands"]);
                        id(last_product).publish_state(root["product"]["product_name"]);
                        return true;
                        });

Anything in the logs that might be useful for us?

This is the log with max_response_buffer_size: 172
[12:59:24][D][uart_debug:158]: <<< "XXXXXXXXXXXXX"
[12:59:24][D][text_sensor:064]: 'Barcode-Scanner-Custom Sensor': Sending state 'XXXXXXXXXXXXX'
[12:59:24][D][switch:012]: 'Barcode-Scanner-Custom Buzzer' Turning ON.
[12:59:24][D][switch:055]: 'Barcode-Scanner-Custom Buzzer': Sending state ON
[12:59:24][D][switch:016]: 'Barcode-Scanner-Custom Buzzer' Turning OFF.
[12:59:24][D][switch:055]: 'Barcode-Scanner-Custom Buzzer': Sending state OFF
[12:59:24][D][text_sensor:064]: 'Barcode-Scanner-Custom EAN': Sending state ''
[12:59:24][D][text_sensor:064]: 'Barcode-Scanner-Custom Brand': Sending state ''
[12:59:24][D][text_sensor:064]: 'Barcode-Scanner-Custom Product': Sending state ''
[12:59:24][D][getEanData:215]: Scanned EAN: \xxx#\xxx?

[12:59:24][D][text_sensor:064]: 'Barcode-Scanner-Custom EAN': Sending state 'XXXXXXXXXXXXX'
[12:59:24][D][openFood:337]: API: https://world.openfoodfacts.org/api/v2/product/XXXXXXXXXXXXX?fields=brands,product_name,code

[12:59:25][D][http_request.arduino:119]: Content-Length: -1
[12:59:25][D][custom:349]: ptr: a8

{"code":"XXXXXXXXXXXXX","product":{"brands":"XXX","code":"XXXXXXXXXXXXX","product_name":"XXX"},"status":1,"status_verbose":"product found"}

[12:59:25][E][json:103]: JSON parse error: InvalidInput
[12:59:25][D][getEanData:221]: Openfoodfacts result: 

[12:59:25][W][component:237]: Component script took a long time for an operation (1286 ms).
[12:59:25][W][component:238]: Components should block for at most 30 ms.
[12:59:28][D][http_request.arduino:119]: Content-Length: 13
[12:59:28][D][opengtin:290]: ptr: error=1

[12:59:28][D][getEanData:237]: OpenTinDb result: 

[12:59:28][W][component:237]: Component script took a long time for an operation (1870 ms).
[12:59:28][W][component:238]: Components should block for at most 30 ms.
[12:59:28][D][text_sensor:064]: 'Barcode-Scanner-Custom Sensor': Sending state 'unknown'
[12:59:30][D][switch:012]: 'Barcode-Scanner-Custom Buzzer' Turning ON.
[12:59:30][D][switch:055]: 'Barcode-Scanner-Custom Buzzer': Sending state ON
[12:59:30][D][switch:016]: 'Barcode-Scanner-Custom Buzzer' Turning OFF.
[12:59:30][D][switch:055]: 'Barcode-Scanner-Custom Buzzer': Sending state OFF
[12:59:42][D][text_sensor:064]: 'Barcode-Scanner-Custom EAN': Sending state ''
[12:59:42][D][text_sensor:064]: 'Barcode-Scanner-Custom Brand': Sending state ''
[12:59:42][D][text_sensor:064]: 'Barcode-Scanner-Custom Product': Sending state ''
[12:59:42][D][text_sensor:064]: 'Barcode-Scanner-Custom EAN': Sending state 'Scan Barcode'

This is the log with max_response_buffer_size not defined at all, defined with a value of over 200 or with it only being defined in one of the get functions:
[13:25:54][D][text_sensor:064]: 'Barcode-Scanner-Custom EAN': Sending state ''
[13:25:54][D][text_sensor:064]: 'Barcode-Scanner-Custom Brand': Sending state ''
[13:25:54][D][text_sensor:064]: 'Barcode-Scanner-Custom Product': Sending state ''
[13:25:54][D][text_sensor:064]: 'Barcode-Scanner-Custom EAN': Sending state 'Scan Barcode'
[13:26:16][D][uart_debug:158]: <<< "XXXXXXXXXXXXX"
[13:26:16][D][text_sensor:064]: 'Barcode-Scanner-Custom Sensor': Sending state 'XXXXXXXXXXXXX'
[13:26:16][D][switch:012]: 'Barcode-Scanner-Custom Buzzer' Turning ON.
[13:26:16][D][switch:055]: 'Barcode-Scanner-Custom Buzzer': Sending state ON
[13:26:16][D][switch:016]: 'Barcode-Scanner-Custom Buzzer' Turning OFF.
[13:26:16][D][switch:055]: 'Barcode-Scanner-Custom Buzzer': Sending state OFF
[13:26:16][D][text_sensor:064]: 'Barcode-Scanner-Custom EAN': Sending state ''
[13:26:16][D][text_sensor:064]: 'Barcode-Scanner-Custom Brand': Sending state ''
[13:26:16][D][text_sensor:064]: 'Barcode-Scanner-Custom Product': Sending state ''
[13:26:16][D][getEanData:215]: Scanned EAN: \xxx#\xxx?

[13:26:16][D][text_sensor:064]: 'Barcode-Scanner-Custom EAN': Sending state 'XXXXXXXXXXXXX'
[13:26:16][D][openFood:335]: API: https://world.openfoodfacts.org/api/v2/product/XXXXXXXXXXXXX?fields=brands,product_name,code

[13:26:17][D][http_request.arduino:119]: Content-Length: -1
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
[13:27:32][E][http_request.arduino:132]: Stream pointer vanished!
and this goes on infinitley until the device crashes

Additional information

I have two http_request.get functions that do different things. Only if both of them have the max_response_buffer_size defined the system won't run into the [02:38:44][E][http_request.arduino:132]: Stream pointer vanished!-Error upon executing them

Chris112 commented 1 week ago

I too have have a very similiar project with the same URL lookup and tried swapping from the arduino framework to esp-idf and using SSL with no luck. I've rolled esphome back until this is fixed 👀

andrewjswan commented 4 days ago

https://github.com/esphome/issues/issues/5945#issuecomment-2187027867

MisterRadish commented 3 days ago

1) Until 2024.6.1 the get_string method worked fine for me on multiple ESP8266 making http requests Upgrading to 2024.6.1 forced the use of the capture_response: true and use the the "body" variable as get_string was no longer supported. I rolled out to 1 ESP8266 that communicates directly with another ESP8266 running ESPHome and imediately ran into the memory leak issue so I waited until 2024.6.2 #(5955) and the fix before continuing my roll out ro other ESPs. 2) Following the 2024.6.2 fix and when rolling out to other ESP8266 devices I have run into the "[E][http_request.arduino:132]: Stream pointer vanished!" error and the ESP8266 locks up. I have tried setting the max_response_buffer_size to a value for all HTTP requests as mentioned above but this did not solve the issue.

In summary : The inital ESP (1) is ESPHome to ESPHome and works on 2024.6.2 while the other ESPs (2) do not. The only difference is that the ESPs in (2) are using HTTP request to access a tasmota plug rather than another ESPHome device. Both devices provide a JSON response which caused no issue in 2024.5.5.

MisterRadish commented 3 days ago

Adding reference to this comment which states.

One thing that was overlooked with the rewrite was servers sending chunked responses with no known Content-Length header. This will need more code added to handle these kind of responses.

andrewjswan commented 3 days ago

https://github.com/esphome/esphome/pull/6953

DJake42 commented 17 hours ago

FYI Tested everything again on 2024.6.4, still the same issue.

andrewjswan commented 17 hours ago

PR is not ready yet.:(