Implementation of the IMU inside the StreamRecorder App

[ozgunkaratas]

Hello everybody!

i am aware that the StreamRecorder app must be modified to ask consent for the IMU both in app manifest and also by editing for asking consent, however i still have some questions regarding this matter. Since update loops are only for the cameras, i was wondering if we have to individually implement some update and write threads (for example, similar to RMCameraReader.cpp) so that we can save the data obtained from the IMU. If that is the case, i would appreciate some tips and hints !

thanks in advance!

[Sayoj]

Hey @ozgunkaratas,

Sadly I can't help you, since I am working on the same problem. Would you be so kind and share your code with us? So we can maybe help each other.

Kind regards

[ozgunkaratas]

hello Sayoj, i am trying to imitate the same code from the camera streams but so far without success. i am attaching the accelerometer code that i changed (which is essentially the same for gyro and mag) maybe this will spark up some ideas for us all :)

// Loads vertex and pixel shaders from files and instantiates the cube geometry. void AccelRenderer::AccelUpdateLoop() { uint64_t lastSocTick = 0; uint64_t lastHupTick = 0; LARGE_INTEGER qpf; uint64_t lastQpcNow = 0;

// Cache the QueryPerformanceFrequency
QueryPerformanceFrequency(&qpf);

winrt::check_hresult(m_pAccelSensor->OpenStream());

//create storage related vars 
//start recording

std::lock_guard<std::mutex> storage_guard(m_storageMutex);

//set storage folder
m_storageFoldera = storageFolder;
wchar_t fileName[MAX_PATH] = {};
swprintf_s(fileName, L"folder", m_storageFoldera.Path().data(), m_pAccelSensor->GetFriendlyName());
m_storageCondVar.notify_all();

//comes from appmain.cpp on stream rec
StorageFolder localFolder = ApplicationData::Current().LocalFolder();
auto archiveSourceFolder = co_await localFolder.CreateFolderAsync(L"accelfolder", CreationCollisionOption::ReplaceExisting);

while (!m_fExit)
{
    char printString[1000];

    IResearchModeSensorFrame* pSensorFrame = nullptr;
    IResearchModeAccelFrame *pSensorAccelFrame = nullptr;
    ResearchModeSensorTimestamp timeStamp;
    const AccelDataStruct *pAccelBuffer = nullptr;
    size_t BufferOutLength;

    winrt::check_hresult(m_pAccelSensor->GetNextBuffer(&pSensorFrame));

    winrt::check_hresult(pSensorFrame->QueryInterface(IID_PPV_ARGS(&pSensorAccelFrame)));

    {
        std::lock_guard<std::mutex> guard(m_sampleMutex);

        winrt::check_hresult(pSensorAccelFrame->GetCalibratedAccelaration(&m_accelSample));
    }

    winrt::check_hresult(pSensorAccelFrame->GetCalibratedAccelarationSamples(
        &pAccelBuffer,
        &BufferOutLength));

    lastHupTick = 0;
    std::string hupTimeDeltas = "";

    for (UINT i = 0; i < BufferOutLength; i++)
    {
        pSensorFrame->GetTimeStamp(&timeStamp);
        if (lastHupTick != 0)
        {
            if (pAccelBuffer[i].VinylHupTicks < lastHupTick)
            {
                sprintf(printString, "####ACCEL BAD HUP ORDERING\n");
                OutputDebugStringA(printString);
                DebugBreak();
            }
            sprintf(printString, " %I64d", (pAccelBuffer[i].VinylHupTicks - lastHupTick) / 1000); // Microseconds

            hupTimeDeltas = hupTimeDeltas + printString;

        }
        lastHupTick = pAccelBuffer[i].VinylHupTicks;
    }

    hupTimeDeltas = hupTimeDeltas + "\n";
    //OutputDebugStringA(hupTimeDeltas.c_str());

    pSensorFrame->GetTimeStamp(&timeStamp);
    LARGE_INTEGER qpcNow;
    uint64_t uqpcNow;
    QueryPerformanceCounter(&qpcNow);
    uqpcNow = qpcNow.QuadPart;

    if (lastSocTick != 0)
    {
        uint64_t timeInMilliseconds =
            (1000 *
                (uqpcNow - lastQpcNow)) /
            qpf.QuadPart;

        if (timeStamp.HostTicks < lastSocTick)
        {
            DebugBreak();
        }

        char currentDate[30];

        time_t t = time(NULL);

        struct tm tm = *localtime(&t);

        sprintf(printString, "####Accel: %d-%d-%d %d:%d:%d  % 3.4f % 3.4f % 3.4f %f %I64d %I64d\n",
            tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec,
            m_accelSample.x,
            m_accelSample.y,
            m_accelSample.z,
            sqrt(m_accelSample.x * m_accelSample.x + m_accelSample.y * m_accelSample.y + m_accelSample.z * m_accelSample.z),
                (((timeStamp.HostTicks - lastSocTick) * 1000) / timeStamp.HostTicksPerSecond), // Milliseconds
            timeInMilliseconds);

        //OutputDebugStringA(printString);

        // the storage functions should come over here
        auto path = m_storageFoldera.Path().data();
        std::wstring fullName(path);
        fullName = L"accel.txt";

        std::ofstream accel_data(fullName);
        accel_data << m_accelSample.x << " ," << m_accelSample.y << " ," << m_accelSample.y << " ,"
            << tm.tm_year + 1900 << " ," << tm.tm_mon + 1 << " ," << tm.tm_mday << " ," << tm.tm_hour << " ,"
            << tm.tm_min << " ," << tm.tm_sec << " ," << sqrt(m_accelSample.x * m_accelSample.x + m_accelSample.y * m_accelSample.y + m_accelSample.z * m_accelSample.z) << " ,"
            << (((timeStamp.HostTicks - lastSocTick) * 1000) / timeStamp.HostTicksPerSecond) << " ,"
            << timeInMilliseconds << " END OF LINE \n";

        accel_data.close();

    }
    lastSocTick = timeStamp.HostTicks;
    lastQpcNow = uqpcNow;

    if (pSensorFrame)
    {
        pSensorFrame->Release();
    }

    if (pSensorAccelFrame)
    {
        pSensorAccelFrame->Release();
    }
}

winrt::check_hresult(m_pAccelSensor->CloseStream());

}

[Sayoj]

Hey @ozgunkaratas

with a lot of help of a good friend of mine, we managed to make it work. You can finde the code here.

I hope this may help you! :) Best Regards

[ozgunkaratas]

Hello @Sayoj

thank you for your response, i will have some time with the Hololens tomorrow and i will try your app on it. I was working on some extensions myself to include accel,gyro and mag on separate files and i will try to upload this code as well!

if i get it working i will close the issue and open up a new one with gyro and mag extensions so that everyone can benefit!

best regards

[ozgunkaratas]

Hello @Sayoj,

i extended the code into three different threads and the sample can be found here

https://github.com/ozgunkaratas/HoloLens2ForCV

i hope this helps other people as well.

additionally, have you tried to write the files into a single txt file instead of creating a .txt file for every timestamp? The tarball produces multiple .txt files for each timestamp. i tried using std::ofstream but the txt file is never created to begin with. i will focus on this and update the fork accordingly,

best regards

[Sayoj]

Hey @ozgunkaratas

Having it in three different threads makes the code definitely much more clearly and easier to read! :D

Yeah, I am working on the same problem. Can't figure out how to just write the IMU data in one text file. If you happens to solve this, please let me know.

Best regards

[Sayoj]

Hey @ozgunkaratas

Anything new on your front? Did you managed to integrate it into one file with timestamp?

Meanwhile I tried to translate the code into HoloLens2-Unity-ResearchModeStreamer, but still to no avail yet.

Kind Regards

[ozgunkaratas]

Hello, @Sayoj i used a linux terminal command to merge all of the txt files together in a folder according to their creation date and at the end this produces a big txt file with all of the IMU data from the beginning to the end. This work-around is efficient so far, find it below here:

cat $(find . -maxdepth 1 ! -name "." -type f -printf "%T+ %p\n" | sort | sed 's/^. //') >> BIGFILE

additionally, there is another timing problem that i came across. since the timestamping is directly related with the relative ticks, it becomes inconsistent as the IMUs cannot maintain their refresh rates of 12Hz for the accelerometer and 20Hz for the gyroscope. I have run experiments where the IMU and the PV camera were simultaenously capturing data and IMU sensor rate dropped to as low as 5 Hz. So far this is random and dependent on device temperature and other factors. There have been experiments where the PV timestamps were showing correct values whereas the last IMU timestamp was off by a few minutes.

Inside TimerConverter.cpp there is a UniversalToUnixTime function that i wanted to use, but i currently do not have access to the HL. I presume that this function will not use the relative ticks happening in the sensor to stamp the data but rather directly use FILETIME. Maybe you can also try this out and update the repo on your findings,

kind regards

Özgün

[kianwei96]

Hey @ozgunkaratas

with a lot of help of a good friend of mine, we managed to make it work. You can finde the code here.

I hope this may help you! :) Best Regards

Hi @Sayoj ! I'm interested in the device temperature data and have been trying to use your modifications to save the IMU frames, but am running into issues accessing the IMU data (have tried ACCEL and GYRO). I'm able to work with the DEPTH_AHAT data, so I think device side everything should be alright. While debugging, this is where the exception arises (in RMCameraReader.cpp):

Would you have any idea or hints on where to look? Thanks in advance!