search

letscontrolit / ESPEasy

Easy MultiSensor device based on ESP8266/ESP32
http://www.espeasy.com
Other
3.28k stars 2.22k forks source link

Dynamic build environment & short build definition #3262

Open TD-er opened 4 years ago

TD-er commented 4 years ago

As stated numerous times before in issues here and on the forum, it is quite important to have a more dynamic build setup.

We now have:

But this still isn't easy to use for anyone.

Another feature I really would like to have is to let a module request a new build based on its own build parameters.

One of the things I'm thinking of is to generate a single string representing a build configuration. For example a HEX string separated by some character separating the definition of plugins, controllers and notifiers.

The HEX values then represent a binary bitfield with the ID. For example 0x78 = 0111 1000 = P001 & P002 & P003 & P004

Just to not limit ourselves to max. number of IDs (e.g. RPiEasy uses plugin IDs > 512) there should be some way of extending these strings without loosing compatibility the separate substrings must be separated.

Since the range of plugins could become quite long, we might need a way to shorten it. For example IPv6 addressing uses some separator to split every 16 bit and also allow to use less than 4 digits to describe the 16 bit. This is still open for better ideas :)

A string could be something like this: P7800:0:1T3000N0

So we need a way to encode this string from a build on an ESP and some web page. It should also be usable as input to generate a build.

Creating an encoder to generate it in ESPEasy isn't that hard and also the decoder in the build environment is do-able as we now use Python in combination with PlatformIO in the build process.

What's needed is a build environment able to receive the build definition and making the generated build available for download. Some kind of REST API comes to mind here as the most suitable solution. I also can imagine it would be useful to select a platform (ESP8266/ESP32/ESP8285), flash size and flash layout and maybe some other useful parameters.

Maybe someone has more good ideas for this?

Example of a full build definition: ESP8266_4M2M_P7800:0:1T3000N0

As we can't use a colon in a filename, I guess it should be replaced with an underscore: ESP8266_4M2M_P7800_0_1T3000N0.bin

For an online build environment I thought the web frontend to select a build should generate the build string along with some way to notify the requester of the build and store it in a database. The build environment should then fetch the next build definition and start building it. If the build already exists on the file system, it should be marked as done immediately, otherwise it should build it.

tonhuisman commented 4 years ago

Example of a full build definition: ESP8266_4M2M_P7800:0:1T3000N0

As we can't use a colon in a filename, I guess it should be replaced with an underscore: ESP8266_4M2M_P7800_0_1T3000N0.bin

I think it will be useful to include the build date in the filename, like ESP8266_4M2M_P7800_0_1T3000N0_20200920.bin, with the date at the end to sort similar configurations together in a files list.

Using hex representation for the configuration is quite readable, and possibly appealing to us, 'technical guys', but the Pnnnnn configuration string will be rather long when using the higher Pxxx plugins (we just reached 100...). Would it be feasible to use base64 (without padding) to shorten it a bit (~25% shorter), but still make it some kind of 'recognizable/readable'? It does have the disadvantage that it will be quite hard to mock up a configuration by hand 😆. The Tnnnn and Nnn configuration arguments won't need base64 encoding because they are short enough, for now, and may be a reason to not adopt base64 for the Pnnnnn configuration, though, as we have the 'powwa of Python' available, that shouldn't actually be an issue.

Edit: Using either Hex or Base64 encoding could be 'configurable' by replacing P with H for hexadecimal encoding, and use another character, maybe even the P, for base64 encoded configuration. And maybe this encoding stuff is even too much trouble, so you could just leave it with hex. It's just a suggestion 😃

TD-er commented 4 years ago

Well the base64 encoding does need capitalization, which may be an issue on some OS'es. Also its ending may be padded by 1 or 2 = characters, which also is a bit difficult ;)

By using the grouping per 16 bit, you only need 2 characters per 16 IDs, so 128 IDs can be written using only 18 characters + a few needed per actually used plugin.

Maybe we can also shorten it even more using some notation to mark lots of 0's to quickly skip a lot of numbers. For example :0(4): can be read as 4 blocks of 0. This allows to make it even shorter, even when using a number of 'playground' plugins. (ID 128 ... 255)

I do agree with the need of adding the date stamp, although I'm not yet sure as of what it should represent. Is it a tag in the Git branch? (most useful I guess)

TD-er commented 4 years ago

Just added a test PR to see if it is a bit usable.

Initially I made an error by not swapping the bits per 16 bits, but maybe that's not a bad idea after all as it may result in shorter strings as you will not end with 1...3 zeroes. So not sure yet which bit order we should use here.

N.B. this was the used build config:

    "CONTROLLER_SET_ALL",
    "NOTIFIER_SET_NONE",
    "PLUGIN_SET_ONLY_SWITCH",
    "USES_P001",  # Switch
    "USES_P002",  # ADC
    "USES_P004",  # Dallas DS18b20
    "USES_P100",  # Pulse Counter - DS2423

    "USES_C016",  # Cache Controller
    "USES_C018",  # TTN/RN2483
    "USES_C015",
tonhuisman commented 4 years ago

Well the base64 encoding does need capitalization, which may be an issue on some OS'es.

I was probably hoping this wouldn't clash (often), though it's indeed possible.

Also its ending may be padded by 1 or 2 = characters, which also is a bit difficult ;)

My intent was to leave these off, as it may be a reserved character, and base64 can be interpreted without 'm.

The 'bitness' is something to carefully consider, most human readable would be left to right, serialized with the plugin numbers. And to avoid long names, maybe the colons could be removed (from the hex bitmap only) instead of replaced by underscores when creating the filename, though that may compromise readability. Just thinking out loud here 🤔

TD-er commented 4 years ago

When I remove the colons, the string will become larger as the colons allow to leave out characters when the value (in hex) starts with a 0 (or more than 1) And since it is very likely there will be 'gaps' between the used plugins, there is likely to be several zeroes in a row. Without the colons you have no idea what bit position you are at.

Or we should use the () notation not for sets of 4 zeroes, but per zero. Thing is you will then need 3 bytes to maybe save 3 or 4 characters