More compact recorded track data storage

[biodranik]

Some of our users would love to store all their movements/tracks in OM. Storing recent/recorded track data in the compact format will not only save space for year-long trips but also can be loaded/processed faster in some cases.

It would be great to discuss and design/estimate efforts for a more efficient storage format. Refactoring should include migration from the current format to a new one.

This is how points are stored in C++ now:

void Pack(char * p, location::GpsInfo const & info)
{  
  MemWrite<double>(p + 0 * sizeof(double), info.m_timestamp);
  MemWrite<double>(p + 1 * sizeof(double), info.m_latitude);
  MemWrite<double>(p + 2 * sizeof(double), info.m_longitude);
  MemWrite<double>(p + 3 * sizeof(double), info.m_altitude);
  MemWrite<double>(p + 4 * sizeof(double), info.m_speedMpS);
  MemWrite<double>(p + 5 * sizeof(double), info.m_bearing);
  MemWrite<double>(p + 6 * sizeof(double), info.m_horizontalAccuracy);
  MemWrite<double>(p + 7 * sizeof(double), info.m_verticalAccuracy);
  ASSERT_LESS_OR_EQUAL(static_cast<int>(info.m_source), 255, ());
  uint8_t const source = static_cast<uint8_t>(info.m_source);
  MemWrite<uint8_t>(p + 8 * sizeof(double), source);
}

void Unpack(char const * p, location::GpsInfo & info)
{
  info.m_timestamp = MemRead<double>(p + 0 * sizeof(double));
  info.m_latitude = MemRead<double>(p + 1 * sizeof(double));
  info.m_longitude = MemRead<double>(p + 2 * sizeof(double));
  info.m_altitude = MemRead<double>(p + 3 * sizeof(double));
  info.m_speedMpS = MemRead<double>(p + 4 * sizeof(double));
  info.m_bearing = MemRead<double>(p + 5 * sizeof(double));
  info.m_horizontalAccuracy = MemRead<double>(p + 6 * sizeof(double));
  info.m_verticalAccuracy = MemRead<double>(p + 7 * sizeof(double));
  uint8_t const source = MemRead<uint8_t>(p + 8 * sizeof(double));
  info.m_source = static_cast<location::TLocationSource>(source);
}

Each point takes 65 bytes. The data can be packed more tightly to be always kept in memory and to be processed more efficiently. Saving a track from these already collected points can be as simple as marking start and end points on it.

1. A simple idea is to use a more compact data format based on the range of possible values and good enough precision. For example, coordinates with 10^-6 precision (~10cm) would be enough for mobile GPS devices. Or using a quad tree cell id like is used for data indexing. Min/max Earth elevation is also limited, we may include submarines and airplanes, of course, then it will anyway fit into 16 bit signed value with a meter precision (even with sub-meter precision the size will be smaller than sizeof(double) = 64 bits now). If each stored record has the same size, it would be possible to do binary search and random reads on the stored track data, like it is possible now.
2. Use delta encoding. It may be very compact compared to other options, but random reading becomes more complicated without a separate indexing mechanism. Does it worth introducing it?

Originally posted by @biodranik in https://github.com/organicmaps/organicmaps/pull/7951#pullrequestreview-2062440326

CC @cyber-toad @vng @kirylkaveryn @rtsisyk @kavikhalique @itfarrier

[rtsisyk]

65 bytes per point is about 5 megabytes per day (65 60 60 * 24 / 1024.0 / 1024.0 = 5.35). 24h is the maximum recording interval in the current implementation. What problem are we trying to solve here?

[biodranik]

There is a request to store/render local data for months or even years of travel. Why it should be limited only to a day? Then you can "save" a recorded track at any time later, without any hassle or time pressure.

[rtsisyk]

data for months

10 megabytes?

even years of travel

I guess the rendering of millions of points will die earlier.

Not a priority for "The Track Recorder for Android (GSoC 2024)" project.

[biodranik]

Almost 40 Mb of data were collected in a week. More data means slower data loading, which means faster battery depletion.

It is not necessary to render everything at all times.

Why was it marked as a "Raw idea"? The idea is pretty clear and simple to implement.