Jev1337 / NiceFlor-Encoder

Nice Flor-s Remote Encoder
Apache License 2.0
3 stars 1 forks source link

Possible integration with ESPHome remote_transmitter.transmit_raw #1

Closed TimelessNL closed 8 months ago

TimelessNL commented 8 months ago

Hi @Jev1337.

Great work on the NiceFlor protocol! I'm looking through your code and are trying to understand how to use this in my own project. My plan is to use the remote_transmitter component from ESPHome controlled using its API or MQTT. This makes the transmitter really simple and as a bonus integrates nicely with HomeAssistant.

One thing I'm struggling with is the send() command and its output, I suspect the hexadecimal output needs to be converted to binary with the correct timings to accommodate for manchester tribit encoding. These raw timings should then be possible to be used with remote_transmitter.transmit_raw Action.

Can you give my any tips on using the output of send() and convert it over to raw timings? Thanks in advance!

TimelessNL commented 8 months ago

I was able to port your python code to ESPHome using the following code:

#include <stdint.h>
#include <stdio.h>

uint64_t nice_flor_s_encode(uint32_t serial, uint32_t code, uint8_t button_id, uint8_t repeat)
{
    uint8_t snbuff[4] = {0};
    snbuff[0] = serial & 0xff;
    snbuff[1] = (serial & 0xff00) >> 8;
    snbuff[2] = (serial & 0xff0000) >> 16;
    snbuff[3] = (serial & 0xff000000) >> 24;

    uint64_t encbuff[7] = {0};
    uint16_t enccode    = pgm_read_word_near(nice_flor_s_table_decode + code);
    uint8_t ki          = nice_flor_s_table_ki[code & 0xff] ^ (enccode & 0xff);

    encbuff[0] = button_id & 0x0f;
    encbuff[1] = ((repeat ^ button_id ^ 0x0f) << 4) | ((snbuff[3] ^ ki) & 0x0f);
    encbuff[2] = enccode >> 8;
    encbuff[3] = enccode & 0xff;
    encbuff[4] = snbuff[2] ^ ki;
    encbuff[5] = snbuff[1] ^ ki;
    encbuff[6] = snbuff[0] ^ ki;

    uint64_t encoded = 0;
    encoded |= ((encbuff[6] << 0x4) & 0xF0) << 0;
    encoded |= (((encbuff[5] & 0x0F) << 4) | ((encbuff[6] & 0xF0) >> 4)) << 8;
    encoded |= (((encbuff[4] & 0x0F) << 4) | ((encbuff[5] & 0xF0) >> 4)) << 16;
    encoded |= (((encbuff[3] & 0x0F) << 4) | ((encbuff[4] & 0xF0) >> 4)) << 24;
    encoded |= (((encbuff[2] & 0x0F) << 4) | ((encbuff[3] & 0xF0) >> 4)) << 32;
    encoded |= (((encbuff[1] & 0x0F) << 4) | ((encbuff[2] & 0xF0) >> 4)) << 40;
    encoded |= (((encbuff[0] & 0x0F) << 4) | ((encbuff[1] & 0xF0) >> 4)) << 48;
    encoded = encoded ^ 0xFFFFFFFFFFFFF0;
    encoded = encoded >> 4;

    return encoded;
}

void nice_flor_s_tx_10(std::vector<int>* tx_buffer)
{
    tx_buffer->push_back(500);
    tx_buffer->push_back(-1000);
}

void nice_flor_s_tx_11(std::vector<int>* tx_buffer)
{
    tx_buffer->push_back(1000);
    tx_buffer->push_back(-500);
}

void nice_flor_s_tx_sync(std::vector<int>* tx_buffer)
{
    tx_buffer->push_back(1500);
    tx_buffer->push_back(-1500);
}

void nice_flor_s_tx_gap(std::vector<int>* tx_buffer)
{
    tx_buffer->push_back(15000);
}

std::vector<int> nice_flor_s_tx_generate(uint64_t data)
{
    std::vector<int> tx_buffer;

    nice_flor_s_tx_sync(&tx_buffer);

    unsigned char byte;
    int i, j = 0;
    unsigned char *b = (unsigned char*) &data;
    for (i = sizeof(data)-1; i >= 0; i--) {
        for (j = 7; j >= 0; j--) {
            byte = (b[i] >> j) & 1;
            if(byte){
               nice_flor_s_tx_10(&tx_buffer);
            }else{
               nice_flor_s_tx_11(&tx_buffer);
            }
        }
    }
   nice_flor_s_tx_gap(&tx_buffer);
   return tx_buffer;
}

With the following ESPHome config

esphome:
  name: niceflors
  friendly_name: niceflors

  includes:
    - ./niceflors/codes.h
    - ./niceflors/nice.h

remote_transmitter:
  pin: GPIO5
  id: transmitter
  carrier_duty_percent: 100%

number:
  - platform: template
    name: "Current Code"
    id: current_code
    optimistic: true
    min_value: 0
    max_value: 65535
    step: 1

  - platform: template
    name: "Repeat"
    id: repeat
    optimistic: true
    min_value: 1
    max_value: 7
    step: 1

  - platform: template
    name: "Button"
    id: button_id
    optimistic: true
    min_value: 1
    max_value: 2
    step: 1

button:
  - platform: template
    name: "Send signal"
    on_press:
      - remote_transmitter.transmit_raw:
          code: !lambda |-
            uint64_t encoded = nice_flor_s_encode(0x00E48DCA, id(current_code).state, id(button_id).state, id(repeat).state);
            return nice_flor_s_tx_generate(encoded);

It is still a work in progress, especially with the button, code, and repeat as input numbers. Using this method I can test my outcome against your examples. And so far they seem to match.

I'm still uncertain if the nice_flor_s_tx_generate() implementation is correct though as I'm currently unable to test the resulting signal.

I thought you might be interested, as this eliminates the need for a separate code generating API server as the entire thing can run on the ESP itself.

Jev1337 commented 8 months ago

One thing I'm struggling with is the send() command and its output, I suspect the hexadecimal output needs to be converted to binary with the correct timings to accommodate for manchester tribit encoding. These raw timings should then be possible to be used with remote_transmitter.transmit_raw Action.

There is a library to digitally outputs (high/low outputs on GPIO), called RF433send. Static Usage Example for Nice Flor:

#include "RF433send.h"
RfSend* tx_whatever;
//The following setup works for me when it comes to the Nice Flor Remote
tx_whatever = rfsend_builder(
    RfSendEncoding::TRIBIT,      //Encoding
    PIN_RFOUT,     //Output Pin
    RFSEND_DEFAULT_CONVENTION,  // Do we want to invert 0 and 1 bits? No.
    1,                          // Number of sendings
    nullptr,                    // No callback to keep/stop sending (if you want to send
                                // SO LONG AS a button is pressed, the function reading the
                                // button state is to be put here).
    17888,                      // initseq
    1432,                       // lo_prefix
    1424,                       // hi_prefix
    0,                          // first_lo_ign
    474,                        // lo_short
    952,                        // lo_long
    0,                          // hi_short (0 => take lo_short)
    0,                          // hi_long  (0 => take lo_long)
    1400,                       // lo_last
    19324,                      // sep
    52                          // nb_bits
  );
byte data[] = { 0x1 , 0xf6 , 0x60 , 0xba , 0x62 , 0xb , 0x4c };
tx_whatever->send(sizeof(data), data);

Dynamically speaking you can either generate all 4 codes, or generate 1 and determine the next 3 with your own algorithm (I also implemented it here). image Code1 being the first button and Code2 being the second button, however you can just generate all 4 using the encoder.

The send() (of the rf433send library) works by using delays() to represent continuous sequences of 1s and 0s. The data that you put is the same data that the encoder gives you as encbuff.

I thought you might be interested, as this eliminates the need for a separate code generating API server as the entire thing can run on the ESP itself.

Yes, I actually do have a C/C++ Version of the Python code, I did plan to implement it into one server, however my current ESP Module does not have enough memory to store all the table hex codes, on top of that, I need to have an external storage such as a SD memory card + Reader to store the last code reached, this could also be done via EEPROM memory but it's unreliable and it's more like a temporary solution. On top of all the mentionned issues, Implementing everything into the ESP could cause added delays, although negligable.

TimelessNL commented 8 months ago

There is a library to digitally outputs (high/low outputs on GPIO), called RF433send. Static Usage Example for Nice Flor:

Thanks for the tip, I saw your NiceOpener implementation using this library, but as ESPHome offers almost identical functionality with its remote_transmitter component and is a native functionality this would be preferred for now.

Dynamically speaking you can either generate all 4 codes, or generate 1 and determine the next 3 with your own algorithm (I also implemented it here). Code1 being the first button and Code2 being the second button, however you can just generate all 4 using the encoder.

This part is unclear to me, maybe I'm not understanding the rolling code encryption enough. But when a button is pressed does it send multiple sequences?

Yes, I actually do have a C/C++ Version of the Python code,

Great! Are you willing to share that implementation?

I did plan to implement it into one server, however my current ESP Module does not have enough memory to store all the table hex codes, on top of that,

My current ESPHome build is around 690KB, how big are your builds?

RAM:   [====      ]  43.7% (used 35776 bytes from 81920 bytes)
Flash: [=======   ]  66.1% (used 690873 bytes from 1044464 bytes)

I need to have an external storage such as a SD memory card + Reader to store the last code reached, this could also be done via EEPROM memory but it's unreliable and it's more like a temporary solution.

Although the ESP8266 indeed does not have an EEPROM, its emulation with flash memory is quite stable from what I know. But true, it's up to personal preferences.

On top of all the mentionned issues, Implementing everything into the ESP could cause added delays, although negligable.

I've not yet tested it myself, but is the receiver really picky on the timings? this is my 3rd 433MHz project although the first with a rolling code. But the other 2 work fine on ESPHome and so far they have never failed any command (yet).

Jev1337 commented 8 months ago
/*
    * Protocol Nice Flor-S
    * Packet format Nice Flor-s: START-P0-P1-P2-P3-P4-P5-P6-P7-STOP
    * P0 (4-bit)    - button positional code - 1:0x1, 2:0x2, 3:0x4, 4:0x8;
    * P1 (4-bit)    - batch repetition number, calculated by the formula:
    * P1 = 0xF ^ P0 ^ n; where n changes from 1 to 15, then 0, and then in a circle
    * key 1: {0xE,0xF,0xC,0xD,0xA,0xB,0x8,0x9,0x6,0x7,0x4,0x5,0x2,0x3,0x0,0x1};
    * key 2: {0xD,0xC,0xF,0xE,0x9,0x8,0xB,0xA,0x5,0x4,0x7,0x6,0x1,0x0,0x3,0x2};
    * key 3: {0xB,0xA,0x9,0x8,0xF,0xE,0xD,0xC,0x3,0x2,0x1,0x0,0x7,0x6,0x5,0x4};
    * key 4: {0x7,0x6,0x5,0x4,0x3,0x2,0x1,0x0,0xF,0xE,0xD,0xC,0xB,0xA,0x9,0x8};
    * P2 (4-bit)    - part of the serial number, P2 = (K ^ S3) & 0xF;
    * P3 (byte)     - the major part of the encrypted index
    * P4 (byte)     - the low-order part of the encrypted index
    * P5 (byte)     - part of the serial number, P5 = K ^ S2;
    * P6 (byte)     - part of the serial number, P6 = K ^ S1;
    * P7 (byte)     - part of the serial number, P7 = K ^ S0;
    * K (byte)      - depends on P3 and P4, K = Fk(P3, P4);
    * S3,S2,S1,S0   - serial number of the console 28 bit.
*/

For more context, you can read the above for more information, there are online russian articles explaining how it works with images but the important part is that you understand P0 and P1, the others are calculated using the tables.

This part is unclear to me, maybe I'm not understanding the rolling code encryption enough. But when a button is pressed does it send multiple sequences?

A button press of the Nice Flor-s remote will send at least (if held, it's just going up to 6 codes and keep looping from 1 to 6), if we look closely, the 4 codes that are sent have the same bytes except one; P1 changes with each code, here is an example of button 1 press of serial "0x00E48DCA": image As you can see here, the only thing that changes is P1, to find out what changes, you can do arithmetic operations (subtract etc...) and find out what is the hex number to add/subtract in order to find the next sequence. However note that I do it this way because in my API, to lessen the length of the GET request (and lessen the chance of losing packets) I just require one code and the arduino will just do the math within microseconds to find out the rest.

Great! Are you willing to share that implementation?

It's basically the same python one, I forgot where I stored it on my pc but I quickly redone it and it should work (hopefully).

I've not yet tested it myself, but is the receiver really picky on the timings? this is my 3rd 433MHz project although the first with a rolling code. But the other 2 work fine on ESPHome and so far they have never failed any command (yet).

I meant delays within the ESP's processing.

My current ESPHome build is around 690KB, how big are your builds?

Honestly no clue, right now I don't have Arduino IDE installed it's been a while since I last opened the project, when I finished the project I just made sure everything works then I reinstalled windows, since then I haven't ran into issues with the current setup, but like I said using an external API satisfies my needs, especially that that API happens to be a Python Flask website.

TimelessNL commented 8 months ago

For more context, you can read the above for more information, there are online russian articles explaining how it works with images but the important part is that you understand P0 and P1, the others are calculated using the tables.

Thank you for the additional information, this makes it a lot clearer.

It's basically the same python one, I forgot where I stored it on my pc but I quickly redone it and it should work (hopefully).

That's great, thanks for the effort!

I do have some remaining questions:

if held, it's just going up to 6 codes and keep looping from 1 to 6

Jev1337 commented 8 months ago

I've noticed that your table_encode values in codes.h are different than the table_decode in this pull request . Is that because one is for encoding and the other one for decoding?

The tables are for encoding, they can be used to decode, however I think the values in that pull request might be incorrect because the way the tables are right now, they work just fine.

Currently there are 65536 codes to be used, do I assume correctly that when the last code is used the counter resets and thus the first entry from nice_flor_s_table_encode will be reused?

I haven't tested, but there is a discussion about it here on how it acts when it overflows. There was a link explaining what could happen, but the website is down.

Other than the pairing process, did you ever need to send more than one batch repetition per command? I'm specifically asking this because I've noticed that some 433MHz devices send one command 2 to 4 times as it takes time for that receiver to distinguish the signal from noise and therefor only responds when it has received a command multiple times.

If I understood correctly no, but in some instances where the sender is far away, I did have to repeat multiple times.

TimelessNL commented 8 months ago

I haven't tested, but there is a discussion about it here on how it acts when it overflows. There was a link explaining what could happen, but the website is down.

I had some trouble getting google translate to work due to the cloudflare protection, but the cached version worked. What will happen when it overflows will remain vague for now, even by people on that forum. But I guess with 65536 options based on 4 actions a day it will last for around 45 years. I think other components will wear out before that time 😅

Jev1337 commented 8 months ago

I had some trouble getting google translate to work due to the cloudflare protection, but the cached version worked. What will happen when it overflows will remain vague for now, even by people on that forum. But I guess with 65536 options based on 4 actions a day it will last for around 45 years. I think other components will wear out before that time 😅

I see, I hope you find a solution to getting the raw data, I honestly don't have time for it now, but please if you find a way to generate raw 433Mhz signals or an algorithm to replicate the tribit code of 433Mhz of the Nice remote share them here!

TimelessNL commented 8 months ago

an algorithm to replicate the tribit code of 433Mhz of the Nice remote share them here!

Sure will! I think my ESPHome implementation is complete 🥳, thanks to your help. Because of bad weather I've not been able to test it though 😅. But when I compare your solution against mine the pattern is identical:

The pairing process is still a bit vague to me, when I watch video's like this and these the authors seem so imply that a button is 'copied' and with this logic one has to pair each button individually.

But when looking at your code and this repository the button used is 0x02 (aka stop?) which means to my understanding that all the buttons 0x01 0x02 0x04 and 0x08 on the new remote should work from that point on.

Could you by any chance confirm this?

Jev1337 commented 8 months ago
// P0 (4-bit)    - button positional code - 1:0x1, 2:0x2, 3:0x4, 4:0x8;

Button 1: 0x01 Button 2: 0x02 Button 3: 0x04 Button 4: 0x08

On my code I use 0x01 for pairing, which is button 1, on this repository, 0x02 is used, meaning the second button is used.

The pairing process is simple:

  1. Enable pairing mode on the receiver by using the new remote (holding the button, a.k.a loop through the 6 codes) for 5 seconds (with code repeat/sync 1)
  2. Letting the receiver know that the new remote should be added to the database, by clicking the button of the old remote 3 times
  3. Repeat step 1 (with code repeat/sync 2)
TimelessNL commented 8 months ago

Thanks for explaining. So if I understand correctly, after the pairing process (using any button on the new and old remote) all 4 buttons on the new remote should work? No need to pair each button individually?

Jev1337 commented 8 months ago

Thanks for explaining. So if I understand correctly, after the pairing process (using any button on the new and old remote) all 4 buttons on the new remote should work? No need to pair each button individually?

No, each remote has a serial number, and if you pair button 1 with a receiver, you can't pair the other buttons with that same receiver. There is a video explaining that in Russian on Youtube, but I didn't save it.

Also just to clarify, there is no copying going on, those technicians don't understand what's going on so they say its "cloning" and "copying" the old remote, but in reality it's just telling the receiver to add a new serial number to the database linked with the button number.

(the timings are a bit different as I'm using these)

I read the timings of my remote and got those values, but factory values are the same ones you put, so yeah those are more correct.

EDIT: I think I'm a bit out of context, Why do you need to pair all 4 buttons with the same receiver?

TimelessNL commented 8 months ago

EDIT: I think I'm a bit out of context, Why do you need to pair all 4 buttons with the same receiver?

I'm currently in a quest to control the gate with automations from HomeAssistant, and for that I need dedicated open and close commands.

For example: when the gate is physically opened already (by the user using a remote) and HomeAssistant wants to open the gate (again) because of an automation and because it does not know it was opened I don't want the gate to close but instead leave it open. When HomeAssistant sends an open command, the gate will simply do noting and remain open.

Right now I have a remote with 2 buttons: remote

And as I understand from this random manual I found on page 15 there are remotes with 4 buttons that do: afbeelding

And looking at the different pairing processes, there is a way of pairing all buttons on a remote: afbeelding I hope by pairing the remote this way, button 3 and 4 will be become open and close buttons. So I can use 0x04 and 0x08 in my request to specifically tell the gate to open and close.

The pairing method you explained seems to duplicate the functionality of the original remote, as they say: (the new transmitter can receive the same settings as those of the memorized transmitter) afbeelding As I don't know exactly how the original remote was paired and since the remote only has 2 buttons I don't know if the receiver will respond to button 3 and 4.

Jev1337 commented 7 months ago

EDIT: I think I'm a bit out of context, Why do you need to pair all 4 buttons with the same receiver?

I'm currently in a quest to control the gate with automations from HomeAssistant, and for that I need dedicated open and close commands.

For example: when the gate is physically opened already (by the user using a remote) and HomeAssistant wants to open the gate (again) because of an automation and because it does not know it was opened I don't want the gate to close but instead leave it open. When HomeAssistant sends an open command, the gate will simply do noting and remain open.

Right now I have a remote with 2 buttons: remote

And as I understand from this random manual I found on page 15 there are remotes with 4 buttons that do: afbeelding

And looking at the different pairing processes, there is a way of pairing all buttons on a remote: afbeelding I hope by pairing the remote this way, button 3 and 4 will be become open and close buttons. So I can use 0x04 and 0x08 in my request to specifically tell the gate to open and close.

The pairing method you explained seems to duplicate the functionality of the original remote, as they say: (the new transmitter can receive the same settings as those of the memorized transmitter) afbeelding As I don't know exactly how the original remote was paired and since the remote only has 2 buttons I don't know if the receiver will respond to button 3 and 4.

Oh I never even knew that there was a way to program it in a way where you can program each button to do one command instead of a toggle button.

My remote is the same as the one you shown with two buttons, however it is programmed in a way where the first button toggles (open/close) the outdoor garage door and the second button toggles (open/close) the indoor garage door.

As for the document saying that it "duplicates the settings", I don't think that there are settings, it just adds the new remote serial number to the authorized serial numbers database.

So I guess you try and find out whether the rest of the buttons would get automatically paired or not, I'm going to bet that they wont be automatically paired.

TimelessNL commented 7 months ago

So I guess you try and find out whether the rest of the buttons would get automatically paired or not, I'm going to bet that they wont be automatically paired.

Will do! And if they don't I probably need to add a sensor to the gate to detect if it is open or not. 🤷

Jev1337 commented 7 months ago

So I guess you try and find out whether the rest of the buttons would get automatically paired or not, I'm going to bet that they wont be automatically paired.

Will do! And if they don't I probably need to add a sensor to the gate to detect if it is open or not. 🤷

Hey there, have you tried it? If yes did it automatically recognize the buttons? Or you had to pair them one by one? I'm planning to make my project work with 4 different buttons that way if I click open I know for sure it's going to be open

TimelessNL commented 7 months ago

Unfortunately I've not been able to test the 4 button pairing method yet. But when I do I'll post the results 👍. Great to see that you've also implemented support for all 4 buttons.

TimelessNL commented 7 months ago

I finally had time to test the all button single remote paring method and can confirm it works 🎉 It works exactly like it states in the manual with dedicated Step-by-Step, OPEN and CLOSE commands. Don't know what AUX does but probably enable a specific output I suppose.

I had to change my implementation to send the pairing (in my case button 1) command for at least 5 seconds for it to be successful.

Jev1337 commented 7 months ago

You paired each button is that correct?

TimelessNL commented 6 months ago

No there was no need to. At first I thought to use button 0x08 for the pairing process. Me thinking this would imply to the receiver there is a 4 button remote. But the curiosity got to me and used button 0x01 instead to see what would happen. So after sending button 0x01 for at least 5 seconds in repeat I saw the red indicator in the receiver blinking 3 times indicating a successful pair and to my surprise it immediately responded to all 4 buttons with Step-by-Step, OPEN and CLOSE.

I have no idea how one would pair just 1 button to a receiver for both OPEN/CLOSE as the manual only mentions to pair a single action to a button. But eh 🤷‍♂️ I only interested in the 4 button pairing anyway. afbeelding

One thing to note, in my case when the gate is moving (opening or closing) repeating the instruction will cause it to pause momentarily until another command is given.

Jev1337 commented 6 months ago

Thanks for the response, So for button 4 you used code 0x08? For some reasons it didn't work for me knowing that I already paired the 0x01 and it works just fine by opening/closing (like a toggle). I might be missing something, it might be that the receiver is configured in a way it doesn't accept other buttons except the 0x01.

TimelessNL commented 6 months ago

So for button 4 you used code 0x08?

Yes.

it might be that the receiver is configured in a way it doesn't accept other buttons except the 0x01.

Might be 🤔, as you paired it using another remote. According to the manual it will take of the other remotes configuration. Have you tried to unpair the ESP transmitter? afbeelding Or maybe delete all transmitters and start over? afbeelding Or use a new serial number for the all button pairing method?

Jev1337 commented 6 months ago

The remote I copied has only two buttons, first button for the exterior garage door and the second one for the interior one

TimelessNL commented 6 months ago

I think that might be your problem, I suspect if you remove this/all remotes and pair them again it should work.

Jev1337 commented 3 months ago

I think that might be your problem, I suspect if you remove this/all remotes and pair them again it should work.

I tried with the other buttons and repaired, (pair with 0x01 with another remote's 0x01 button), however only one button works after doing so, the same button closes/opens the door, like a toggle.

I think there are different models of the garage door perhaps or different configurations for the receiver.

In any case, I changed to Home Assistant on my old Raspberry Pi 3B+, reasons are because my ESP was not reliable and starts giving up crash codes every now and then. I did consider ESPHome however when I was adding it to the addon, it kept generating connection errors, so eventually I gave up entirely on the ESP. I don't have any usages for my Home Assistant so I used my RPi purely for this project, I used the code that was available here. That code had some issues such as deprecations and the timings for some reason had to be halved, I think the clock of my RPi is doubled or something (I haven't really experienced with RPi enough).

So far it has been good with the RPi and I can track downtimes from outages, as well as create a DDNS without having to confirm hostname for free No-IP. Now using "Cloudflared" addon with a paid domain (3$ a year) to secure my host against DDoS and other attacks.

TimelessNL commented 2 months ago

pair with 0x01 with another remote's 0x01 button

Are you still using another remote to pair the new transmitter? That is not how the 'all button' method works. You have to use the radio button on the motor itself and look at the LED next to it: afbeelding And follow these steps: pairing