Alert flooding

[mavishak]

03/05/2021

We have been aware of the following bug for some time and decided the time has come to solve it. Bug description: When the motion sensor goes off multiple alerts are sent to the database at the same time.

At first, we thought it was a logic problem in the 'record Alert()' function, we created a safety net that insured alerts were sent - meaning if an alert was not sent successfully by a certain timeout we sent another one after closing the connection with Firebase. The problem with this sequence of events is that we sent another alert even if the connection was not closed properly, thus creating a situation where a second alert was sent while the first was pending.

When we saw this did not help we thought the problem might be with the hardware promised delay, so we added a software safety net in the shape of a 'timeout' meaning that each time an alert was sent a timeout was set and no alert could be sent before the timeout was over. This did not help either.

Then we became creative, we thought that the interrupt might be interrupting itself before the timeout was set so we rearranged the order of the commands so that the timeout would be set first, and only then would the handler be queued. Needless to say, this did not work either:)

As a last resort, we came to the conclusion that the event queue contains a bug or that the problem has to do with HTTP/TCP networking behavior that has nothing to do with us.

To check if the last assumption was true we created the next main:

#include "stm32f103xb.h"
#include "usart.h"
#include "esp8266_Firebase.h"
#include "esp8266_WiFi.h" //for testing usart1...
#include "timers.h"
#include "common.h"

int main(void)
{
    init_usart2(); 
    init_timer4(); //for ESP8266 timeout
    init_usart1(); // for ESP8266

    write_usart2((uint8_t*)("\r\n_______________\r\n"));//For test

    while(1)
    {
        alert_Handler();
        delay_with_timer4(20);
    }
}

The result was as follows, we received alerts in the following times:

03/05/2021 22:55:20 03/05/2021 22:55:20 03/05/2021 22:55:20

03/05/2021 22:54:54 03/05/2021 22:54:54 03/05/2021 22:54:54 03/05/2021 22:54:54 03/05/2021 22:54:54

03/05/2021 22:54:30 03/05/2021 22:54:30 03/05/2021 22:54:30 03/05/2021 22:54:30 03/05/2021 22:54:30

03/05/2021 22:54:04 03/05/2021 22:54:04 03/05/2021 22:54:04 03/05/2021 22:54:04 03/05/2021 22:54:04

03/05/2021 22:53:38 03/05/2021 22:53:38 03/05/2021 22:53:38 03/05/2021 22:53:38 03/05/2021 22:53:38

03/05/2021 22:53:14 03/05/2021 22:53:14 03/05/2021 22:53:14 03/05/2021 22:53:14 03/05/2021 22:53:14

03/05/2021 22:52:48 03/05/2021 22:52:48 03/05/2021 22:52:48 03/05/2021 22:52:48 03/05/2021 22:52:48

03/05/2021 22:52:24 03/05/2021 22:52:24 03/05/2021 22:52:24 03/05/2021 22:52:24 03/05/2021 22:52:24

We concluded the problem was due to networking behavior as the main sends one alert every 20 seconds and the results show otherwise.

[mavishak]

04/05/2021

We made an important discovery today. The alert flooding is not a cause of the embedded side, rather it is due to functionality we are using on the web application 'onSnapshotChange()' that returns the full list of alerts every time an alert is added and therefore makes it seem as if there are duplicate alerts.

However, we added a line of code in event_queue.c:

res = queue.eq[queue.readIndex].handler();
queue.eq[queue.readIndex].handler = NULL; // cleanup

To make sure no trace of the handler is found after it is complete.