external_components not properly fetching (before yaml check) / sometimes not working at all

RubenKelevra commented 2 weeks ago

The problem

The external_components function is a bit flaky for me, when overwriting an already existing component with an updated one for testing.

Issues I've noticed

Using just a branch, will lead to no updates being done, after pushing new commits to Github to the branch.
Using tags will sometimes not work at all, without any notification about not using the external_component. This will lead to complains by the yaml checker if there were changes to the syntax as the only way to find out that something was going wrong. If there were no changes to the syntax, there's no way of knowing that this happened.
After pushing a new tag, even when waiting two minutes, it will still complain that this path does not exist. A retry will immediately fetch the new tag and build (in most cases).

Observations

Sometimes it helps to wait just half an hour, sometimes it won't.
Cleaning the build files don't help
Pushing the same commit to a new tag will help sometimes.

Which version of ESPHome has the issue?

2024.9.2 / 2024.9.1

What type of installation are you using?

Home Assistant Add-on

Which version of Home Assistant has the issue?

2024.10.0b3 / 2024.9.0

What platform are you using?

ESP8266

Board

Wemos D1 mini

Component causing the issue

external_components

Example YAML snippet

external_components:
  - source: github://RubenKelevra/esphome_esphome@tcs34725_spec_based_irradiance_output_v0.23
    components: [ tcs34725 ]

Anything in the logs that might be useful for us?

#### Not fetching the update at all:

INFO ESPHome 2024.9.2
INFO Reading configuration /config/esphome/ambient-light-sensor.yaml...
INFO Detected timezone 'Europe/Berlin'
Failed config

sensor.tcs34725: [source /config/esphome/ambient-light-sensor.yaml:88]
  platform: tcs34725
  i2c_id: bus_a

  [red_channel_irradiance] is an invalid option for [sensor.tcs34725]. Did you mean [red_channel]?
  red_channel_irradiance: 
    name: Ambient Light Sensor Red Channel Irradiance
    accuracy_decimals: 1
    filters: 
      - or: 
          - throttle: 300s
          - delta: 0.01

  [green_channel_irradiance] is an invalid option for [sensor.tcs34725]. Did you mean [green_channel]?
  green_channel_irradiance: 
    name: Ambient Light Sensor Green Channel Irradiance
    accuracy_decimals: 1
    filters: 
      - or: 
          - throttle: 300s
          - delta: 0.01

  [blue_channel_irradiance] is an invalid option for [sensor.tcs34725]. Did you mean [blue_channel]?
  blue_channel_irradiance: 
    name: Ambient Light Sensor Blue Channel Irradiance
    accuracy_decimals: 1
    filters: 
      - or: 
          - throttle: 300s
          - delta: 0.01

  [sensor_saturation] is an invalid option for [sensor.tcs34725]. Please check the indentation.
  sensor_saturation: 
    name: Ambient Light Sensor Sensor Saturation
    accuracy_decimals: 1

#### First try of fetching a newly published tag:

INFO ESPHome 2024.9.2
INFO Reading configuration /config/esphome/ambient-light-sensor.yaml...
Failed config

external_components: [source /config/esphome/ambient-light-sensor.yaml:2]
  - 
    Could not find components folder for source. Please check the source contains a 'components' or 'esphome/components' folder.
    source: |-
      github://RubenKelevra/esphome_esphome@tcs34725_spec_based_irradiance_output_v0.24
    components: 
      - tcs34725

### Hitting retry won't work:

INFO ESPHome 2024.9.2
INFO Reading configuration /config/esphome/ambient-light-sensor.yaml...
INFO Detected timezone 'Europe/Berlin'
Failed config

sensor.tcs34725: [source /config/esphome/ambient-light-sensor.yaml:88]
  platform: tcs34725
  i2c_id: bus_a

  [red_channel_irradiance] is an invalid option for [sensor.tcs34725]. Did you mean [red_channel]?
  red_channel_irradiance: 
    name: Ambient Light Sensor Red Channel Irradiance
    accuracy_decimals: 1
    filters: 
      - or: 
          - throttle: 300s
          - delta: 0.01

  [green_channel_irradiance] is an invalid option for [sensor.tcs34725]. Did you mean [green_channel]?
  green_channel_irradiance: 
    name: Ambient Light Sensor Green Channel Irradiance
    accuracy_decimals: 1
    filters: 
      - or: 
          - throttle: 300s
          - delta: 0.01

  [blue_channel_irradiance] is an invalid option for [sensor.tcs34725]. Did you mean [blue_channel]?
  blue_channel_irradiance: 
    name: Ambient Light Sensor Blue Channel Irradiance
    accuracy_decimals: 1
    filters: 
      - or: 
          - throttle: 300s
          - delta: 0.01

  [sensor_saturation] is an invalid option for [sensor.tcs34725]. Please check the indentation.
  sensor_saturation: 
    name: Ambient Light Sensor Sensor Saturation
    accuracy_decimals: 1

#### Changing to v0.23, which is the same commit, does work:

INFO ESPHome 2024.9.2
INFO Reading configuration /config/esphome/ambient-light-sensor.yaml...
INFO Detected timezone 'Europe/Berlin'
INFO Generating C++ source...
INFO Compiling app...
Processing ambient-light-sensor (board: d1_mini; framework: arduino; platform: platformio/espressif8266@4.2.1)
--------------------------------------------------------------------------------
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
Dependency Graph
|-- ESP8266WiFi @ 1.0
|-- ESP8266mDNS @ 1.2
|-- noise-c @ 0.1.6
|-- Wire @ 1.0

Additional information

No response

nkinnan commented 1 week ago

external_components:
  - source:
    refresh: 1s  <------------------- add this

RubenKelevra commented 1 week ago

@nkinnan I'll try this tomorrow. But why isn't the source refreshed automatically, if the tag is not found? And why does it not abort the build process, but just continues without the specified source? 🤔

nkinnan commented 1 week ago

I couldn't say, I just run with the refresh specified and have had no issues.

speculation: I suppose it doesn't know that the tag wasn't found because it didn't check because it wasn't refreshing from git in the first place since it's "timeout" to refresh wasn't hit. And if you manually go into the filesystem and delete what it downloaded, and it's not hit the refresh "timeout" it again doesn't check git, and doesn't know you went and deleted it behind it's back.

RubenKelevra commented 6 days ago

Well, that sounds pretty broken ;D

If I got time I'll take a look

nkinnan commented 3 days ago

It's just a guess ;) But I'm sure any improvements would be welcomed by esphome.

blacknell commented 2 days ago

I was seeing this when specifying a PR number and after pushing new commits it just wasn't being download. I ended up using local files

esphome / issues