Generation of EventAccumulatedPic and Event-Threshold comparison

Hi,

I am developing my first application based on the CeleX 04-S DVS sensor. I have a few questions about that.

1) Generation of the EventAccumulatedPic

According to the documentation, the brightness value from each event is taken and saved into a cv::Mat. My own implementation of this procedure:

this->ownAccumulatedMat = cv::Mat(cv::Size(768, 640), CV_8UC1, char(0));
// ...
auto eventVector = pSensorData->getEventDataVector();

for (auto e : eventVector) {
    int currRow = PIXELS_PER_ROW - e.row - 1;
    int currCol = PIXELS_PER_COL - e.col - 1;

    this->ownAccumulatedMat.at<unsigned char>(currRow, currCol) = (unsigned char) e.brightness;
}

results in a visible difference compared to the result provided by the SDK (cv::Mat currentSdkEventAccumulatedMat = pSensorData->getEventPicMat(EventAccumulatedPic); ) . My generated frame seems significantly noisier. Can you explain me the difference in the generation of this accumulated Mat?

accumulated_noise_diff

2) Comparison with the threshold

The documentation of the threshold-function (CeleX4::setThreshold) indicates that the triggering of events is determined by this value. So I'm assuming that each event has a absolute difference to the last known gray-value above this threshold-value.

Comparing the brightness of each event in the EventDataVector to the corresponding pixel in (previous, current and own) EventAccumulatedPic shows, however, that this is not necessarily the case:

Here the example output of a comparison (matching / not matching):

// [...] previousSdkAccumulatedMat: 24327 / 104185 (18.9297%) currentSdkEventAccumulatedMat: 18575 / 109937 (14.4539%) ownAccumulatedMat: 11476 / 117036 (8.92991%)

previousSdkAccumulatedMat: 26141 / 103533 (20.159%) currentSdkEventAccumulatedMat: 18908 / 110766 (14.5812%) ownAccumulatedMat: 13679 / 115995 (10.5488%) // [...]

Is my source for the difference calculation wrong?

For a better understanding, I have attached a minimal code example that clarifies my questions and my calculations:

int main(int argc, char **argv) {
    CeleX4 *pCelex = new CeleX4;
    // ...
    pCelex->openSensor("");
    pCelex->setFpnFile(FPN_PATH);
    // ...
    pCelex->setEventFrameTime(60);
    pCelex->setOverlapTime(0);
    pCelex->setThreshold(30);
    // ...
    pCelex->setSensorMode(EventMode);
    auto *pSensorData = new SensorDataObserver(pCelex->getSensorDataServer(), pCelex);
    // ...
    while (true)
    {
        pCelex->pipeOutFPGAData();
        usleep(1000 * 2);
    }
}

SensorDataObserver::SensorDataObserver(CX4SensorDataServer *pServer, CeleX4 *sensor) {
    m_pServer = pServer;
    m_pCelex = sensor;
    m_pServer->registerData(this, CeleX4DataManager::CeleX_Frame_Data);

    this->ownAccumulatedMat = cv::Mat(cv::Size(768, 640), CV_8UC1, char(0));
}

void SensorDataObserver::onFrameDataUpdated(CeleX4ProcessedData* pSensorData) {
    // ...
    if (EventMode == sensorMode) {
        this->frameCount += 1;

        // ******************************************************************
        // Interpretation of applied threshold
        // Assumption: Each event should have an intensity change higher as current threshold in
        //             comparison to last known pixel intensity
        // ******************************************************************
        cv::Mat currentSdkEventAccumulatedMat = pSensorData->getEventPicMat(EventAccumulatedPic);

        auto eventVector = pSensorData->getEventDataVector();
        int diff, eventCountMatching, eventCountNotMatching;

        // 1) compare to previous 'frame' SDK accumulated data
        if (this->frameCount >= 2) {
            eventCountMatching = eventCountNotMatching = 0;
            for (auto e : eventVector) {
                int currRow = PIXELS_PER_ROW - e.row - 1;
                int currCol = PIXELS_PER_COL - e.col - 1;

                diff = (int) this->previousSdkAccumulatedMat.at<unsigned char>(currRow, currCol) - (int) e.brightness;
                if ( abs(diff) >= m_pCelex->getThreshold() ) {
                    eventCountMatching += 1;
                } else {
                    eventCountNotMatching += 1;
                }
            }
            std::cout << "previousSdkAccumulatedMat: " << eventCountMatching << " / " << eventCountNotMatching <<
                " (" << eventCountMatching / (float)(eventCountMatching+eventCountNotMatching)*100.0 << "%)" << std::endl;
        }

        // 2) compare to current SDK 'frame' accumulated data
        if (this->frameCount >= 2) {
            eventCountMatching = eventCountNotMatching = 0;
            for (auto e : eventVector) {
                int currRow = PIXELS_PER_ROW - e.row - 1;
                int currCol = PIXELS_PER_COL - e.col - 1;

                diff = (int) currentSdkEventAccumulatedMat.at<unsigned char>(currRow, currCol) - (int) e.brightness;
                if ( abs(diff) >= m_pCelex->getThreshold() ) {
                    eventCountMatching += 1;
                } else {
                    eventCountNotMatching += 1;
                }
            }
            std::cout << "currentSdkEventAccumulatedMat: " << eventCountMatching << " / " << eventCountNotMatching <<
                " (" << eventCountMatching / (float)(eventCountMatching+eventCountNotMatching)*100.0 << "%)" << std::endl;
        }

        // 3) compare to own accumulated 'frame' data
        if (this->frameCount >= 2) {
            eventCountMatching = eventCountNotMatching = 0;
            for (auto e : eventVector) {
                int currRow = PIXELS_PER_ROW - e.row - 1;
                int currCol = PIXELS_PER_COL - e.col - 1;

                diff = (int) this->ownAccumulatedMat.at<unsigned char>(currRow, currCol) - (int) e.brightness;
                if ( abs(diff) >= m_pCelex->getThreshold() ) {
                    eventCountMatching += 1;
                } else {
                    eventCountNotMatching += 1;
                }
            }
            std::cout << "ownAccumulatedMat: " << eventCountMatching << " / " << eventCountNotMatching <<
                " (" << eventCountMatching / (float)(eventCountMatching+eventCountNotMatching)*100.0 << "%)" << std::endl;
        }
        std::cout << "*********************************************************" << std::endl;

        // ******************************************************************
        // *** Difference between SDK accumulation and own implementation ***
        // ******************************************************************
        for (auto e : eventVector) {
            int currRow = PIXELS_PER_ROW - e.row - 1;
            int currCol = PIXELS_PER_COL - e.col - 1;

            this->ownAccumulatedMat.at<unsigned char>(currRow, currCol) = (unsigned char) e.brightness;
        }

        cv::imshow("currentSdkEventAccumulatedMat", currentSdkEventAccumulatedMat);
        cv::imshow("ownAccumulatedMat", this->ownAccumulatedMat);

        this->previousSdkAccumulatedMat = currentSdkEventAccumulatedMat.clone();
        char key = (char) cvWaitKey(2);
    }
    // ...
}

CelePixel / CeleX4-OpalKelly