preloaded world level map data into the app bundle

[nicHoch]

at the latest update of the app i recognized that the map data is not available on app installation. it takes some seconds before the map presents thecrag content. i would highly suggest that the app already is shipped with a prefilled map database (same content that will be loaded after start up). Th entire map data for the world is around 70mb (if i remember right). The computational power and bandwidth to provide that several thousands times on the first release will most likely freeze our servers. First the large data and second the expected slow server response wight course a very bad first user experience. envision it will take 10minutes or more (id the server is not dead at all) to see anything on the app (with poor mobile connection maybe longer).

so i would suggest that the app is already sipped with the world index map data and only does a "refresh" after first time startup.

[andrasbari]

We will not need to preload world level map into the app bundle. There is enough time to download the starter json from your servers. On the first start we download only neccessary datas for the map usage. We use since parameter for world downloads too. For the first time we use this eg.: https://www.thecrag.com/api/index/detail/7546063?withdata=NodeID&withdata=ParentID&withdata=Depth&withdata=SiblingSequence&withdata=NodeType&withdata=CountryNodeID&withdata=AreaType&withdata=Name&withdata=TLC&withdata=NumberRoutes&withdata=Point&withdata=BoundedBox&withdata=Fence&withdata=Styles&withdata=Webcover&withdata=LastUpdated&to=tlc&since=1&pretty=0&abbr=0&topoidonly=1&webcover-dimensions=%5B%5B1080%2C720%5D%2C%5B540%2C360%5D%2C%5B192%2C192%5D%5D&key=abc123 The result is about 8MB uncompressed JSON. Network transfer is only 2.2MB because your server use gzip compression for transfer. This is the initial request. After that, we use the received LastUpdated value for the next request's since parameter. So we only download the changed datas on every app start.

[nicHoch]

hm.. i am not so sure about this the numbers are ok but the the answer that this is not an issue is false.

i just tried it on the new version with entire world level enabled. I had to wait quite some time (20sec) until the map shows any data (with wlan and normal server load). in that moment the user has no idea what to do (bad first user experience) and i am very concerned about our server usage. there is a real possibility that we will have several 1000 app downloads after first shout out. Our current server serves app, api and web and if we kill the server that will have serious impact on this app release and reputation.

so i would be happy if this issue is not ignored. The prefilled data base in the app bundle come more or less for free (beside dev effort) if the app is 10 or 20mb large does not matter at all - the app store will scale for sure.

[andrasbari]

@nicHoch I think, put starter dataset into the app bundle is not a good method for this. The data is changing frequently. We can't update the app bundle as often as needed. I think the best method is caching several endpoint responses to reduce your server's load. The most time that you feel is the parsing of the json response. It consumes a lot of memory and cpu power for the first start of the app because we need to use custom json parsing algorithm for received data array. I think I can make some optimizations to speed up this point. If you would be a new user, you can't feel it, because you need to login/register first. At this time the json parsing ends on low-end devices too. The data sync is started with the application.

@scd Can you set memcached on server side? In our TG server there is a 15minute cache for json responses, if cache exists and not expired we serve the client from the cache. It is very fast and consume only memory resources. Only the starter dataset need to be cached. for eg.:

if (isset($since)) { // if since parameter is set
  $response= .............. // sql query
  return $response;
} else { // if since parameter is not set
  $key=$nodeId . "_" . md5($_SERVER['REQUEST_URI']);
  $m = new Memcached();
  $m->addServer('localhost', 11211);
  $cached_response = $m->get($key);
  if (!$cached_response) {
    if ($m->getResultCode() == Memcached::RES_NOTFOUND) { // if not cached
        $response = ......................... // sql query
        $m->set($key, $response, time() + 3600); // cache for 1 hour
        return $response;
    } else {
        /* log error */
    }
  } else { // if cached content found
    return $cached_response;
  }
}

Or we can use static json files too. We need to create a cron job for eg. every hour, and generate a json file for the started dataset. But I think the memory cache is the best for this.

[rouletout]

@nicHoch and @scd are we happy with this now?

[rouletout]

Seems we are ok technically, there is one minor improvement to be made on server side, closing