BigBobbas / ESP32-S3-Box3-Custom-ESPHome

Custom ESPHome config for ESP32-S3-Box-3 with sensors and touchscreen
112 stars 19 forks source link

NEW Features I want to add and other enhancements #49

Open BigBobbas opened 3 months ago

BigBobbas commented 3 months ago

Here is a list of things I want to add or change - but haven't had time to complete. Some may not be possible ... but can't hurt to try. This is my 'to do' list. Please feel free to comment or even take on a task and contribute ;) Without the input of you guys this would be a very boring project. If you would like to contribute at all , please use the dev branch and make a PR with any proposed changes. The 2 files to look at are the sensor version no sensor version

These are in no particular order of priority, just straight out of my head.

  1. create a package with core config - giving users less YAML to have to scroll through to make their own tweaks and customisations. also use more substitutions etc. use subs for adding entities. look into using ota updates to just update changes - recently added to ESPHome but haven't had the time to investigate how it all works

  2. Create a web installer for non-sensor version

  3. Create a DND option to turn off screen and mute mic / speakers - preferably with a time from - until selection from the device

  4. Add visuals for additional timers and also naming of timers

  5. Add user configurable 'time out' duration for screen saver and brightness to dim to to be released

  6. add colour selector for backgrounds - would need to be a set number of colours but would like to do this from the device.

  7. Pull in calendar events from HA as visual notifications.

  8. change interval method for screen brightness... don't like having to run it continually every 1 sec added condition to check current state

  9. remove unnecessary entities in HA for the device. entities tidied and re-vamped - TBR

  10. Update guides and add an operation guide.

  11. add additional sounds with option to select for wakeword

  12. add IR blaster config

  13. fix mute!!! STILL πŸ™ƒ re-written and now working~~ TBR

  14. add STT and TTS text sensor. https://github.com/BigBobbas/ESP32-S3-Box3-Custom-ESPHome/issues/50 βœ…

  15. Add switch to change audio from internal speaker to external media player. as per feature request implemented and TBR

  16. add continued conversation as per this request

  17. add switch for continued conversation as per request from Sir_Hartley on discord here

    • any other bug fixes etc that get thrown in along the way :)
MetalOnes commented 3 months ago

Hope you don't mind me just commenting the code here, I've never done a PR before πŸ˜†

I have the IR blaster working on mine. There's 2 caveats: 1) The IR receiver needs to be switched on for the blaster to work. 2) The blaster is very weak. Its reach is maybe 5ft.

First I added a switch for the IR Receiver. I reckon it should be set to ALWAYS_ON, but likely there's a better way of doing this.

switch:
  - platform: gpio
    id: enable_ir_rx
    name: IR Receiver
    pin:
      number: GPIO44
      inverted: true
    restore_mode: ALWAYS_ON

Add this as well. If you don't need to see the received IR codes in the logs, you can remove the dump: all line for the receiver.

remote_receiver:
  id: ir_rx
  pin: 
    number: 38
    inverted: true
  dump: all # Comment out or remove this line if you don't need to see received codes in the logs. Leaving it on results in a lot of log spam.
  rmt_channel: 4

remote_transmitter:
  id: ir_tx
  pin: 
    number: 39
    inverted: false
  carrier_duty_percent: 50%
  rmt_channel: 0

My use case for this was toggling a fan and TV that's close to my desk (where the Box S3 sits). I didn't incorporate learning or anything, just created buttons that send the commands.

button:
  - platform: template
    name: TV Power
    icon: "mdi:television"
    id: tv_power
    on_press:
      then:
        - remote_transmitter.transmit_samsung:
            transmitter_id: ir_tx
            data: 0xE0E040BF

  - platform: template
    id: fan_power_btn
    name: Fan Power Button
    icon: "mdi:fan"
    on_press:
      then:
        - remote_transmitter.transmit_raw:
            carrier_frequency: 38kHz
            code: [Very long sequence code was here]

To get the codes, start viewing the logs in ESPHome, then just point your remote at the receiver and hit the button you'd like the code for. More examples here

It does all work, however the blaster is so weak it's practically useless. I'm not sure if there's a way to increase it's power?

BigBobbas commented 3 months ago

Hi, That is awesome , thank you for sharing! Who needs PR's lol I can pretty much copy this straight in and that's good enough for me πŸ‘ I've been doing a few more tweaks and trying to tick a few more off the list, so this will definitely be a big help, and I think that this is the last of the functions that the box has , to make it complete, as far as the hardware it has available. So a massive thank you for contributing. I will make sure to include this in the next dev-update. πŸ™
With regards to the range it has, I think it is what it is unfortunately, also the fact that the IR LED's are facing you is also a bit of a pain. I did have an idea of making a small device , with multiple LED's and to use that as a repeater, perhaps will look at that in the future, I also have in my mind to print a new case and dock, so maybe there would be scope to upgrade the LED's and the casing... but that's getting more advanced at this stage lol (but the 3d printing STL files are available. and the box is very easily dismantled and modular, so quite tempting lol.... food for thought!

All the best Rob

wappinghigh commented 3 months ago

Add easy firmware upgrades from within the ESPHome Dashboard( straight from within the S3-Box3 card)

I would really love this! 😊

Notes from HA release 2024.7.3

"Besides the new logo, ESPHome has added a new feature that works with this release of Home Assistant: The ability for makers of ready-made devices or projects running on ESPHome, to provide updates for their devices directly to Home Assistant.

This means that if you have a device that is powered-by or Made for ESPHome, and the maker of that device provides updates for it, you can now update that device directly from Home Assistant; without the need for you to build, adopt, or set up the ESPHome software yourself! 😎"

BigBobbas commented 3 months ago

This is included in number 1 in the todo list ;) It will happen ... just will take time as it is not a quick straightforward thing for me to implement. I need to re-organise the whole code structure and at the moment on the whole, the updated configs are still experimental and changes made currently aren't released as stable updates and therefore require manual implementation. Once I have a complete stable update ready to push out I intend to push this out as an OTA update but small fixes, etc at the moment will have to be manually implemented. I have quite a few other features, tweaks etc to complete before there is a 'stable' update, this will also require testing and feedback from users , prior to being released as stable. In order to implement OTA firmware updates there will also need to be an initial update that has to be added manually to allow OTA updates thereafter ;)

I am also providing support and replying to questions etc which all takes time :) Please bare in mind that there is only me doing this and not a company providing consumer products and support. I do this as an unpaid 'hobby' and to share with the community, This project is in no way supported, aided , assisted or affiliated with HA , ESPHome or Nabucasa .

But one thing for sure... IT WILL HAPPEN ;) Best Wishes Rob πŸ‘

BigBobbas commented 3 months ago

After a little research this morning, one limitation with enabling OTA updates is that 50% of the total flash size of the device has to be reserved for the OTA image. This will greatly reduce the amount of available flash for the actual config. Currently the flash size required is around 42% of the available 16mb so enabling OTA updates will greatly limit the available space for developing the config further. However this doesn't imply that it won't happen, it will require some thought and streamlining to try and squeeze everything in without compromising performance and running flash to the max, as flash space is also required for some variables and no doubt others in the future.

kucau0901 commented 3 months ago

is possible to add option to play sound audio response to any chromecast speaker. i know this can be done with editing the yaml but very cool if it is can be done from UI?

BigBobbas commented 3 months ago

is possible to add option to play sound audio response to any chromecast speaker. i know this can be done with editing the yaml but very cool if it is can be done from UI?

Just so I am clear. Would you like an option 'switch' that you can tap to change if the audio is outputted to the internal speaker or to an external HA media player ?

kucau0901 commented 3 months ago

is possible to add option to play sound audio response to any chromecast speaker. i know this can be done with editing the yaml but very cool if it is can be done from UI?

Just so I am clear. Would you like an option 'switch' that you can tap to change if the audio is outputted to the internal speaker or to an external HA media player ?

yup. that would be cool

BigBobbas commented 3 months ago

is possible to add option to play sound audio response to any chromecast speaker. i know this can be done with editing the yaml but very cool if it is can be done from UI?

Just so I am clear. Would you like an option 'switch' that you can tap to change if the audio is outputted to the internal speaker or to an external HA media player ?

yup. that would be cool

OK! thanks for confirming, I will see if there is a way to make that happen πŸ‘ Thanks for the suggestion.

ecto1a2003 commented 3 months ago

Any chance of adding continued conversation mode al la

https://github.com/jaymunro/esphome_firmware

BigBobbas commented 3 months ago

Any chance of adding continued conversation mode al la

https://github.com/jaymunro/esphome_firmware

It's something that I can look into, I am conscious of the available flash space on the device and still have a few things that need adding. But I will add it to the list and see what can be done.

Or if anyone else wants to try and incorporate it then feel free ;) and raise a PR

ecto1a2003 commented 3 months ago

i had an idea about the upgradeability, if we use packages on the endpoints with substitutions it could just point back to the repo. wed just need to specify the mandatory substitutions on the end user end. I'm running like this rn actually.

BigBobbas commented 3 months ago

i had an idea about the upgradeability, if we use packages on the endpoints with substitutions it could just point back to the repo. wed just need to specify the mandatory substitutions on the end user end. I'm running like this rn actually.

if this is something that you currently have running, at the end of a compile it will show a percentage of flash used. what is yours showing ? as it is my understanding that even with the use of packages etc these are still pulled and compiled into the firmware. therefore still taking up flash space. Unless I am misunderstanding what you mean :)

ecto1a2003 commented 3 months ago

heres my output

`INFO ESPHome 2024.7.2 INFO Reading configuration /config/esphome/esp32-s3-box-3-5a93e0.yaml... INFO Updating https://github.com/BigBobbas/ESP32-S3-Box3-Custom-ESPHome@None INFO Updating https://github.com/gnumpi/esphome_audio@dev-next INFO Detected timezone 'America/Los_Angeles' WARNING GPIO45 is a strapping PIN and should only be used for I/O with care. Attaching external pullup/down resistors to strapping pins can cause unexpected failures. See https://esphome.io/guides/faq.html#why-am-i-getting-a-warning-about-strapping-pins WARNING GPIO46 is a strapping PIN and should only be used for I/O with care. Attaching external pullup/down resistors to strapping pins can cause unexpected failures. See https://esphome.io/guides/faq.html#why-am-i-getting-a-warning-about-strapping-pins WARNING GPIO3 is a strapping PIN and should only be used for I/O with care. Attaching external pullup/down resistors to strapping pins can cause unexpected failures. See https://esphome.io/guides/faq.html#why-am-i-getting-a-warning-about-strapping-pins WARNING GPIO0 is a strapping PIN and should only be used for I/O with care. Attaching external pullup/down resistors to strapping pins can cause unexpected failures. See https://esphome.io/guides/faq.html#why-am-i-getting-a-warning-about-strapping-pins WARNING Found and merged multiple configurations for ota platform esphome port(s) [3232] INFO Generating C++ source... INFO Updating https://github.com/espressif/esp-adf.git@v2.5 INFO Updating submodules (components/esp-adf-libs, components/esp-sr) for https://github.com/espressif/esp-adf.git@v2.5 INFO Updating https://github.com/jesserockz/esp32-s3-box-3-board.git@main INFO Updating https://github.com/espressif/esp-tflite-micro@v1.3.1 INFO Compiling app... Processing esp32-s3-box-3-5a93e0 (board: esp32s3box; framework: espidf; platform: platformio/espressif32@5.4.0)

/data/cache/platformio/packages/framework-espidf Already Patched: yes HARDWARE: ESP32S3 240MHz, 320KB RAM, 16MB Flash

ecto1a2003 commented 3 months ago

and how im applying it

substitutions: name: esp32-s3-box-3-5a93e0 friendly_name: Foyer Jarvis micro_wake_word_model: hey_jarvis packages: esphome.voice-assistant: url: https://github.com/BigBobbas/ESP32-S3-Box3-Custom-ESPHome file: s3b.yaml refresh: 0d esphome: name: ${name} name_add_mac_suffix: false friendly_name: ${friendly_name} api: encryption: key: just between you and me ota:

wifi: ssid: !secret wifi_ssid password: !secret wifi_password domain: !secret domain

ecto1a2003 commented 3 months ago

so this likely wont save space but endusers could use the above config and anytime the repository is updated an enduser could just re run the install instead of copy and pasting and recustomizing

BigBobbas commented 3 months ago

ah get you now , yes this is what i am looking to do soon, and to add substitutions for each entity in the same config file and then just have all the core no-need to edit stuff in the package. So each of the 6 screen icons would have a substitution for the display and that would also reflect to the binary for touch and also text sensor. will take a little while to sort it all - but yes agree this is the way to go. and as you say can just install when there's any change and all the customisation will remain untouched πŸ‘. I haven't played with ota updates yet.. but I will look into that too.

I do need to figure out though why, even when i remove the ota: component completely and flash over serial it is not allocating the full 16mb flash. I was under the assumption that ota reserved 50% flash for the ota image so if it failed it would always use the previous image that is stored in flash and therefore very hard to 'brick' the device... but thinking about it... that may be something to do with safe mode.. i need to research a little more.

But to summarise... yes.. you are spot on with what we should do. :)

wappinghigh commented 3 months ago

Hi Bob I don't really mind what you do but the install process really is confusing and time consuming and if it has to be done from scratch each time it is a nightmare for people like me who have moved to OSX and don't have an old computer with direct USB and a https HA webserver. Trust me I know I am now trying a new install from scratch and am having problems again due to the above constraints of ESPHome. There are also so many hidden steps with trial and error and no manual for the device itself what to do how to reboot etc. swapping between cables, wifi not knowing whether the screen is blank because its not rebooting or because it;s being swtiched off - Trust me this is not a project for the masses, not even boffins like me who have a fair bit of practical knowledge with both HA, networks, computers etc what to do, right now. I just can;t seem to get yr firmware easily onto the device. Not yr fault mate. The whole device itself (having to be flashed by USB) and ESPhome for me totally sucks! Cheers

wappinghigh commented 3 months ago

Thanks for all the efforts Bob...ok so I am up and running (again)- you know what the problem/s combined were? 1/old drivers in the mac that contained USB (most macs don't now) - PS who wants to add a driver from the PRC that is blocked by OSX? (Not me!) 2/ The USB cable shipped with the device not 100% suitable for data. I say 100% because the 1st time it may have worked maybe not - I think I recall I got it to load via wifi but not this time I compiled then loaded yr program with a different USB cable I knew was data OK (past experience with Junkies Ulanzi clock build) and then uploaded it to the SBox 3 direct off my HA running linux server and all was good. Trial and error figuring out what the bottlenecks were and pathways to use I guess - See what I mean Bob? Anyway now I have a reliable way of uploading yr updated YAML file all will be good until you find a quicker and easier pathway to firmware upgrades ( which would be needed for mere mortals Bob! ) Just MO!

BigBobbas commented 3 months ago

HI, first of all thank you for pointing out the pitfalls and the issues that you have had, it is you guys that will help make this project better and easier to implement as I have limited capacity for testing so your feedback is greatly appreciated. IF the device has had an initial installation using USB (serial) after that point then providing the device name doesn't change or and networking options then subsequent 'uploads' should work over wifi and not require USB. Personally I use linux (ubuntu) cli to do all of my compiling and uploading which works flawlessly. I am in the final steps of completing the first update to the 'stable' main branch release which will implement the steps discussed in a couple of posts above. To streamline the user facing config to allow for easier setting up of entities etc for use with the device. I will also be looking further into pushing ota updates to hopefully include into the release. Currently I would estimate that this will take possibly up to two weeks as I have some family commitments in this coming week which will slow down progress. However I will need to in addition to the code part have to re-write / update the instructions etc to include the changes and also to add in some troubleshooting tips that have been brought to light by users. Now that there are more features and pages I also need to add an operation manual for the device. I have recently setup a discord server to hopefully provide a more fluent realtime chat for helping troubleshoot issues. https://discord.gg/94auHnG4

Thank you for your patience and understanding and I hope that despite the hiccups, you will find the project more useful than the device being 'just a voice assistant'

Best Wishes Rob

rcblackwell commented 1 month ago

Rob,

The discord invite has expired. Can you sent out a fresh one?

BigBobbas commented 1 month ago

I have added a new link , I didn't realise there was a time expiry. The new link should stay permanently active. https://discord.gg/U3SRYCG6Kp cheers for letting me know,

rcblackwell commented 1 month ago

I have added a new link , I didn't realise there was a time expiry. The new link should stay permanently active. https://discord.gg/U3SRYCG6Kp cheers for letting me know,

Works like a charm. Thank you.

archef2000 commented 1 month ago

Would you accept a PR to allow the Status bar to work everywhere or have a internal toggle for fullscreen apps?

BigBobbas commented 1 month ago

Would you accept a PR to allow the Status bar to work everywhere or have a internal toggle for fullscreen apps?

I'm not quite sure what you mean by to have it work everywhere? it should show on all normal operation pages and be able to tap for info etc.. if it isn't then i can fix it.

But if you could explain in a bit more detail that would be great

archef2000 commented 1 month ago

Currently when for example the wifi page is shown you can't switch to the timer page.

BigBobbas commented 1 month ago

ah ok ... i get you ... by all means if you wanted to make a PR i will certainly take a look. it's a good idea πŸ‘

archef2000 commented 1 month ago

The status bar could also be put into a script so the config is not as big

BigBobbas commented 1 month ago

I'm open to any suggestions and contribution . I am not a dev or coder so have done everything with the limited knowledge I have

archef2000 commented 1 month ago

can s3b_import.yaml be deleted can't see it mentioned anywhere?

BigBobbas commented 1 month ago

it is the config that is imported after flashing from the web installer - i'm currently changing the way that is organised

archef2000 commented 1 month ago

It is the same as s3b.yaml so you could just point to that. I also found a way to just make one config and then extend that with the sensor version as a package. Currently testing it out.

BigBobbas commented 1 month ago

yes, i am in the middle of re-organising i am just having a few hours off from it today :) it is all in hand. The manifest is pointing to import and will be changed in the next day or so. Yes packages will work it is just not something I have got around to doing as yet.

archef2000 commented 1 month ago

I am currently in the progress of displaying the sensor version text when the package is imported. have a look at the progress: https://github.com/archef2000/ESP32-S3-Box3-Custom-ESPHome

BigBobbas commented 1 month ago

sorry - i'm being stupid... what should i be looking at 🀣

archef2000 commented 1 month ago

WIP so just this currently as I am experimenting with the universal config.

BigBobbas commented 1 month ago

ok i'm with you now, users will need to be able to have access from the dashboard to the following:

substitutions: wifi: api: ota: sensors: text_sensors: binary_sensors switch: light: images: color: fonts: display: off the top of my head i think tht's all of the components required to be exposed to the dashboard

archef2000 commented 1 month ago

How should I go about adding the universal file?

archef2000 commented 1 month ago

They will have access to all except the s3b_sensors.yaml as that will be imported from the github repo as a package:

packages:
  sensors:
    url: https://github.com/archef2000/ESP32-S3-Box3-Custom-ESPHome/
    files: [s3b_sensors.yaml]  # optional; if not specified, all files will be included
    ref: main #fe3833c83aa44ee6269ad89ef3c4bf0925947eb1  # optional
    refresh: 0s  # optional
BigBobbas commented 1 month ago

I need to sit down and properly look and go through it, as I am also creating separate web installers for the different versions. I will hopefully have some time tomorrow to have a proper look and test.

archef2000 commented 1 month ago

The s3b_universal.yaml now has the display script and package import. I only have the 3B version so please test it with your sensors.

BigBobbas commented 1 month ago

Hi, I have tested and all works as expected. I'm liking the status bar script very neat. the only issue i had - but my approach may be incorrect. but when removing the package to essentially provide the 'no-sensor' config. I get errors compiling where the display lambda is referencing the now missing sensors. I know this is a WIP so you probably have this in mind or I am going about it the wrong way.

jerome83136 commented 1 month ago

Hi @BigBobbas , Thank you for the amazing work you did over there :)

I would like to share a suggestion; but I'm not sure where to speak about it. So I try here; but let me know if you think this is out of scope of this thread or this repo :)

I'm planning to use several esp32-s3-box3 in my house. They will all use the same wakewords list and they will be located in several rooms.

My question: do you have any plan for adding some kind of "anti-concurrency" mechanism in your firmware ?

Concurrency problem I have in mind:

Example of problems I have in mind:

I'm wondering if some kind of "clustering/quorum" feature could maybe help us solving this problem ? Or maybe synchronizing devices thanks to MQTT to decide:

I would be glad getting your feedback guys :) Is this question relevant ? Is this thread/repo the right place to talk about it ?

Thank you for your feedback Have a great day

Kelerion-UK commented 1 month ago

Hi @BigBobbas , Thank you for the amazing work you did over there :)

I would like to share a suggestion; but I'm not sure where to speak about it. So I try here; but let me know if you think this is out of scope of this thread or this repo :)

I'm planning to use several esp32-s3-box3 in my house. They will all use the same wakewords list and they will be located in several rooms.

My question: do you have any plan for adding some kind of "anti-concurrency" mechanism in your firmware ?

Concurrency problem I have in mind:

* I'm standing between 2 rooms and I say "okay nabu"

* There is a chance 2 devices will catch the wakeword and start listening at the same time

* So the 2 devices will try to manage the intent in concurrency; which can lead to problems

Example of problems I have in mind:

* okay nabu

* device 1 & device 2 wake up

* toggle kitchen light

* device 1 will toggle the kitchen light

* device 2 will toggle the kitchen light

I'm wondering if some kind of "clustering/quorum" feature could maybe help us solving this problem ? Or maybe synchronizing devices thanks to MQTT to decide:

* which device was the first to catch the wakeword

* ensure this device will be (the only one) that will handle the intent

I would be glad getting your feedback guys :) Is this question relevant ? Is this thread/repo the right place to talk about it ?

Thank you for your feedback Have a great day

My first instinct would be to not call the "turn on"/"turn off" functions.. but to call a custom script that checks if it has been toggled in the last second.. and if not? then do it. That would probably stop 2 devices doing the same thing at the same time.

archef2000 commented 4 weeks ago

@BigBobbas I have now fixed the compilation errors. It is not the cleanest but the python script befor the compiler is complaining about the missing ID's so the no sensor version needs dummies. explained here I would like it if you could test it on your side and then I would clean the repo and make a PR.

badnetmask commented 3 weeks ago

Hey there! First of all, let me say that I bumped into this project because I was frustratingly looking for a semi-pre-built-but-customizable solution, and this was the best one I could find. I will keep an eye on this repo, and provide some feedback/contribution when I feel like I have something useful to share.

Now, to comment on @jerome83136 question:

I'm planning to use several esp32-s3-box3 in my house. They will all use the same wakewords list and they will be located in several rooms.

My question: do you have any plan for adding some kind of "anti-concurrency" mechanism in your firmware ?

I'm also interested in this, so I decided to comment here because, before finding this repo, I found this other project called Willow, which sounded a lot more complicated for what I wanted initially. It seems like that project was able to implement the "device concurrency" by using a central controller (which they call Willow Application Server), so I have a feeling this would be a little hard for @BigBobbas to implement.

But hope is never lost, and I'm willing to be corrected. Just thought about sharing what I have seen out there to see if it helps with the discussion.

Keep up the good work!