Feedback required: New sets of plugins/controllers

[TD-er]

The last few weeks we're running out of the space available for the sketch. The test and dev builds now approach their max size so something has to change.

I think the sets of normal, test and dev does not really make sense anymore.

We already have some builds specific for some use case, like specific hardware or just a single purpose. For example the Sonoff POW as a specific build for off the shelf hardware and the IR builds for the IR RX/TX and AC control.

Maybe we can make more logical build sets.

For this we need some user input to get an idea of sensors often used together or so often they must be included in any build.

Other options are to disable some of the webserver parts if they are not needed.

So please let's get some proper lists of used plugins to redefine the build scripts.

[TD-er]

Well I tried the "build based on product" idea a while ago, see the Sonoff POW builds. But then we would end up with so many builds.

Also it does not fix the problem with the "many sensors build" definition.

I'm now working on splitting the stuff as much as possible, to make it all definable from the Custom.h file as much as possible. I also did make a feature request to the ESP8266/Arduino repo for an option to even let go of flash-size related build definitions. (determine SPIFF location at run time, not compile time)

Also I do think the new core 2.6.0, which will be released in about a week, will be enough for 90% of the builds, so that can reduce the build images a lot.

[Misiu]

@TD-er I agree that "build based on a product" won't be good for GitHub releases page, but I think it would be helpful to preconfigure the web-based build tool. Let's say you select Shelly 2, then the ESP8285 will get selected (I think they use ESP8285), this way you will know that you can select plugins that won't increase the build size above 1M.

EDIT: @TD-er there is one more benefit of selecting the device. According to this issue there will be some limitation when auto-size more will be enabled:

Warnings, when the option is enabled:

user must ensure sketch will fit in flash (cannot check) not fully tested, especially with 512K and 1MB flash chips, may not work (there is this "magic" word at physical address 0x0 in flash which contains "software" flash size...) (reading it is disabled in this PR, replaced by the "physical" result ESP.getFlashChipRealSize())

So you must know how many plugins you can select to fit them all in flash.

[henkiejan1]

I am using the most: BME280 sensor Domoticz controller MQTT controller ESP Easy P2P controller PWM functionality (no settings via webinterface. Should be great ;)) SSD1306 CCS811/HDC1080 BME680

And also more and more ESP32. So it should be fine that the ESP32 get more priority in my opinion. :)

[fmuntean]

question: what are the two methods: addPredefinedRules and addButtonRelayRule used for ? they seems to be unused and take some firmware size due to the rule file defined there.

[TD-er]

They are used when you "factory default" to some Sonoff devices. They generate rules and tasks for the push button + relay. It is called from void ResetFactory()

I guess they can be put behind some #define

[ShardanX]

Plugins in use here, I'll split list by device FYI: Weather Station unit:

• PCF8574
• Pulse Counter
• DS18B20
• Dummy Device
• TLS2561
• BMx280
• System Info Room Sensors
• DS18B20
• TLS2651 or BH1750
• SI7021/HTU21D
• System Info Weather Sensor (Old version)
• BMP180
• BH1750 / TLS2561
• SHT1x Dust Sensors
• PMSx003
• BMx280
• NeoPixel Basic
• Oled 1306 Remote box
• PCF8574 / MCP23017
• NeoPicel Basic

[TD-er]

Just a suggestion, so please let me know what you all think of it.

Some sensors/devices have quite a specific purpose and are not likely to also have a number of other sensors connected. For example displays and power meters.

Although it is unlikely you will connect an e-ink display and a Nextion for example, so at first it may seem not making sense to combine them in the same build, they do have in common that they are not likely to be combined with a lot of other sensors. So I was wondering, would it make sense to make a build containing the "normal" set of plugins + all displays and one of the "normal" set + all energy meters? This way you still have builds present to get people started with those and we no longer have "zombie" plugins that are not part of a nightly build. Later when we have a build server operational, it doesn't matter anymore what build combinations we have.

So let me hear what you think about it.

[tonhuisman]

That should be good enough as a workaround for now.

[berbergh]

Ik think this would work for me, but what do you mean with 'normal' set?

[TD-er]

Ik think this would work for me, but what do you mean with 'normal' set?

Every plugin marked here as "normal"

[berbergh]

I see. I guess it is virtually impossible to make the perfect prefab build. I can't. So, I my guess is that your proposal is a good one.

[berbergh]

Just updated a Display and and an energy version. So for all's well.

Can I use ESP_Easy_mega_20201227_minimal_core_274_sdk3_ESP8266_1M_OTA_Domoticz.bin tp update my 4 channel Sonoff? I do not understand what this sentence means: Minimal OTA builds (and other builds with reduced build size) link to online CSS and JavaScript files

[TD-er]

Until this change for linking to external CSS/JS files, all these static files were included in the build. Meaning the build size was larger due to files which your browser could also fetch from another site.

I linked to a CDN URL of GitHub, which hosts copies of our repository. Meaning your browser will fetch them from the GitHub servers when needed. However the "TTL" of those values (the time they can be safely cached by your browser) is set to 1 year.

So what does this al mean? If you operate your units linking to an external hosted CSS or JS file, your browser must be able to reach the GitHub servers. But this only has to be done once and after that the browser will cache those files for 1 year (or until you clear your cache). This means you may need to have an internet connection on the device you use to access the ESPEasy node if those files are not in your browser cache.

There is a way to overcome this limitation, by saving the files using the same names onto your SPIFFS file system of the node. If the files are present on the local file system, the ESP will serve them to you.

The files are located here: https://github.com/letscontrolit/ESPEasy/tree/mega/static

• espeasy_default.css
• favicon.ico
• update_sensor_values_device_page.js
• fetch_and_parse_log.js
• rules_save.js
• github_clipboard.js
• reboot.js
• toasting.js

[clumsy-stefan]

@TD-er this is a great improvement!! My build are ~4K smaller, the 1M build even twice as much!

One suggestion: the URI could be in a static #define or probabky even as config option so it would be possible to easily change it to a local server (eg. home-automation server) which would work even without having access to the internet and not serving it from the node itself (and therefore have a central place to make changes)...

Thanks a lot for your great work and this update!

[TD-er]

I will indeed add it to a define so you can set it yourself in Custom.h for example. I only have to find out how to update the URL to fetch it based on the last tag to make sure a node always uses the files intended for the specific build.

[berbergh]

@TD-er Thank you for this explanation, but I still need to bee sure. Can I use ESP_Easy_mega_20201227_minimal_core_274_sdk3_ESP8285_1M_OTA_Domoticz.bin to update my 4 channel Sonoff? If it fails, I have to unmount the Sonoff from a difficult place to reset.

[tonhuisman]

I have done an OTA update test using an ESP-01S unit, in the usual 2-step process, but not with that specific build tough. My test was done using a beta release. It should work, as the size is nearly the same as the minimal_core_274_sdk3_ESP8266_1M_OTA_Domoticz file I used for my tests. The usual differences between ESP8266 and ESP8285 should not get in your way here, just select the correct bin file. I'm assuming that the 2step_uploader is capable of running correctly on an ESP8285, but I have no ESP8285's so I can't confirm that. If you have done this with older builds, it should still be possible though.

[TD-er]

The 2-step OTA bin has not been changed for years, and never heard of anyone having trouble with it on ESP8285. There is not that much of a difference between ESP8285 and ESP8266. The main difference is the availability of GPIO pins (ESP8285 has more unused) and the internal flash. If a build is made to use the flash with at most 2 pins (DIO mode for example) then I am almost certain any ESP8266 build will run on ESP8285 (given it does not use > 1 MB flash).

Only other thing that may be an issue is when you update from a very old version. The settings should be compatible, but I cannot guarantee it from very old builds. So better also make a backup of your settings before updating.

[berbergh]

Just what I was affraid of, happened.

My 4 chanel Sonoff gave this in the web interface: Update error: ERROR[4]: Not Enough Space

After I first uploaded: ESPEasy_2step_UploaderMega_1024.bin and then: ESP_Easy_mega_20201227_minimal_core_274_sdk3_ESP8285_1M_OTA_Domoticz.bin

But after a reboot, it seems to have updated correctly.

No, there are some issues. I try to find out what they are, but some of the settings are gone.

[berbergh]

Sonoff is no longer acting on messages like http://DEVICE-IP/control?cmd=GPIO,0,1 I send to it from Domoticz

[TD-er]

Hmm I just tested it on my breadboard and it does reply with a JSON reply showing the new state. Can you try that on your node?

[berbergh]

per haps this is not the right place to continue with this specific problem. But I had the same results with JSON, so I dug in a bit deeper. Now I directly control the relay ports of the Sonoff, It seems to be okay again. Beforehand I used to send commands to the switch ports, that in turn toggled the relays accordingly via a rule, but that does not work as it used to anymore. Most important is that I can reach the Sonoff and manipulate the relays. That gives me time to find out why things do what they do.

[TD-er]

Maybe you should also have a look at Domoticz MQTT helper

[berbergh]

Thank you, I just did and followed the directions. However, if I do not put http://SONOFF-IP/control?cmd=GPIO,12,1 in the action on line of the switch in Domoticz, nothing is switched to on.

[tonhuisman]

nothing is switched to on.

Hm, that could be true if you don't have a MQTT server in your network, that Domoticz is subscribed to. Did you install a new MQTT server to enable that? In that case a reboot may be required to get it all up and running.

[berbergh]

It was even worse than that. I forgot to enable the MQTT controller

[berbergh]

In one of the ESPeasy modules, I use dust sensor PMSx003, which is still being tested. It is in neither Display nor Energy. What can I use instead to update?

[TD-er]

I can move it to "normal", as it is for sure stable for years. Let's hope it will not break other builds then for becoming too large.

[berbergh]

I am using the sensor for some 2 years now. I see it is also still here ESP_Easy_mega_20201227_test_ESP8266_4M1M_VCC. That could work for me, I guess.

[Grumpfbe]

My two cents on this; for what I use right now :

In production I have those different D1 mini or Pro:

• BH1750 & AM2320 with an Oled screen P036 so the framed one.
• 3 DS18b20 as a calibration device
• A LCD 2004 to post domoticz message about other probes & values from the net (Weather) and a 7Seg to do a clock. This is my clock to display time date, inside T/H et outside T/H pressure tendency, windspeed & rain. I have a custom build for this one for special LCD characters so I think having to drive a 7SEG & a LCD will never be in any sane person project.
• A relay in a light appliance
• Driving two leds for status and a RCWL for movement detection light appliance. I also need to be able to send orders through rules from Domoticz to "force on" status.
• Some BMx280 with analog input for battery status

Several ESP32Cams (not Espeasy yet). Can see them in Domoticz and a static HTML page on a network drive for a mosaic of them.

In Testing projects (work but not in production yet)

• A LCD 2004 (see clock above) with PIR (maybe replaced by a RCWL) & a button.
• A Capicity Moisture sensor with a framed Oled. I got a PCB project that'll use ADS1115 for the sensors, a RCWL to turn on a framed Oled to display values and the analog for voltage of the battery.
• A SI1145 + BMx280 & analog for battery (nearly in production I just need to print an enclosure).

In paused/abandonned projects

• Framed Oled & Bmx280 & Serial server. The serial out is to communicate with an arduino to drive a color LCD to display domoticz logs about security devices (PIR & door/windows reed switches (RFXCOM)
• ADS1115 with several MQ2 3 & 7 to which I'd like to add my CCS811 or MZH19. This will never be on battery, too power hungry.

For special HW : Several Sonoffs touch. Those will need rules and nothing will ever be added to them. Using sensors or I²C devices is not advised due to the voltage and the way they are used so just generic switch & rules is enough. Several sonoff / Shelly that'll need generic switch & IOs for the relay plus some rules. If it fits them all, the current & Pulses for the POW R2 in the same build will be a NTH.

[Grumpfbe]

Some general considerations :

A weather station/probe oriented release that will need A/D conversion, internal analog, probes support for voltage monitoring, deep sleep. Same for air quality probes. Checking if adding LCD or Oled will fit although just for debug as you want your probe to not waste power to displays or LEDs.

A generic release for Sonoff/Touch/Shelly & other devices that will need only generic switch, IO, rules & maybe current. Those guys; once in prod will never be updated. too risky for many reasons :

OTH is hardly available and nobody fancy unscrewing a wall switch to program it serially with soldered wires. Once installed, it'll not be moved.

[Grumpfbe]

And lastly; we can approach the space issue by looking at the hungriest plugins. Best will be to know what size each .ino file takes once compiled but that'll be very complicated.

We can supposed that biggest .Ino will be the one taking the most memory in the release (bad shortcut I know) so in this case:

P073 7SEG, nice for a clock but by far the biggest. C014 Homie, dunno if it can be optimized or what it does. looks like a H/A system but never heard of it. P091 Dunno what it does, what's the difference with P020 can they be merged maybe ? P036, Framed OLED, I like it but it's very big. Removing some functions maybe ? C018 LORA TTN, doesn't that need it's own release too with the usual suspects probes ? That'll be for very specifics projects. P016 IR, probably can use a release with things that'll be used with it too, dunno, I haven't seen an IR comm/remote used this century. P099 & P095, no clue, can they be merged perhaps ?

So to alleviate the load, maybe creating some BINs for sets that will never really be updated like IR coms & specific HW like wall switches. Including them in new release but not really bothering to recompile for them. That is because a new release should never make them incompatible; see post above, nobody will unscrew a placed wallswitch.

Last consideration (promise) :

• Tinkerers will probably use the same little aliexpress modules together to have fun with Espeasy. Some can't program at all.

• Someone creating his own PCBs and wanting to release weather balloons with GPS tracking in the atmosphere will be able to compile his own release. So plugins for hardware that will require experience in programming or designing/soldering could probably be left out.

[TD-er]

I hoped to make a new build this week, which is already way past the interval I planned a while ago. But right now none of the "test" builds and also the single "dev" build cannot be built as they don't fit (not even 4M ESP32 builds)

So I would like to see a suggestion to split the test (and dev) builds which makes sense.

@tonhuisman did make a nice "max" build setup which includes all plugins and controllers, but you need a 16 MB flash for it. ESP8266 is still limited to just a few kB less than 1MB as maximum build size, regardless the flash size.

[tonhuisman]

I have been working on multiple testing builds, numbered 1..4, but I didn't complete that yet, each containing a separate set of plugins. Part of that plan was to update the Plugins documentation page, showing TESTING 1..4 for where each plugin can be found.

Is that an acceptable path to continue?

[TD-er]

Yep, I guess so. We simply cannot make a selection that will fit all needs, so just take the one that makes the most sense. I was also thinking about maybe splitting Domoticz and OpenHAB as it is unlikely those will be used together, but that's just a short term fix as it will only make it fit for now.

[tonhuisman]

I'll then put my effort into that, the next couple of evenings and weekend. (responding here I can do sometimes during work 😛) Splitting between Domoticz and OpenHAB indeed does look practical, except for the people using both (but how likely is that...). For the hard-core users/testers, there is always the MAX build 😉

[TD-er]

Yep, you made my ESP32 units with 4MB obsolete by creating the "Max" build :)

[Grumpfbe]

I think I already abused my privileges with my WOT :) but I was also thinking that maybe a non OTA version with a max of plugins could work for prototyping as you don't know yet what sensors you'll finish using and you'll probably program it with usb if only to increase the speed as you re-program every 5 minutes.

I've seen some two-steps flashing .bin but didn't find the docs for it so I imagine you thought of it loooong ago but just in case :

Instead of OTA+CODE 50% & 50% Space we go for

OTA+CODE X% + whatever is needed for OTA+Hardly any code

So you when a new version is released you first upload OT+Hardly anything in the dedicated space then use it to UPload the new OTA+CODE.

It's probably what you have done already right ?

Cheers !

[TD-er]

ESP8266 and ESP32 perform OTA slightly different.

ESP8266 flash layout:

• Sketch + OTA area
• File system + some area for core library scratch pad for WiFi settings

ESP32 flash layout:

• Bootloader (which marks what sketch is active)
• Sketch 1 area
• Sketch 2 area
• (optional, we don't use it) Sketch N area
• file system
• Some extra blocks for NVS storage

As you can see the flash is somewhat divided into partitions, which have hard set limits. If you go over those limits you may corrupt data in the next partition, like your file system with settings etc.

ESP8266 does share the partition with the active sketch + OTA image. This means the active sketch + OTA image should fit in that partition. We can use compression on the OTA image, so during upload of the OTA image we can have the compressed image on the flash, which will be decompressed in the boot right after uploading it. It will copy/decompress the OTA image to the start of the partition.

Thus all these combinations must fit:

1. Active sketch (uncompressed) + OTA sketch (compressed)
2. OTA sketch (uncompressed) + new sketch (compressed)
3. new sketch (uncompressed)

You can read this table as the normal OTA update steps (minus step 2), or the complete table for the 2-step OTA update. For 4MB builds the 2-step OTA is hardly an issue, but the maximum sketch size is. For ESP8266, you cannot go above (1M - 4k) as sketch size, so you cannot have all plugins included in a build for ESP8266 as we will go way above this limit. The "max" build for ESP32 is already > 2.5 MB, just to give an indication of how big it has become.