Many fields in objects - Githubissues

jas1211 commented 4 years ago

Hi. I am using tile38. Works fast, everything suits. But there is one problem. I have objects with geo-coordinates, and many layers. a layer is a geofence (polygon) and a set of filters over the attributes of objects. I decided to store the coordinates of the objects in tile38 and, as the name of the fields, the identifiers of the layers into which the object belongs, well, in the value 0 or 1. The calculation is included in the layer or not, another process is calculated that pushes data into tile38. To which clients subscribe (SETHOOK) with a filter by fields.

on the test with 5 layers everything worked fine. but adding 500 layers for 25,000 objects, tile38 ate 5GB of RAM should it be so?

tidwall commented 4 years ago

Hi. That 5GB ram does seem like a lot.

I'm curious what you get back from running the SERVER command? Issue a GC beforehand to make sure that the heap is as small as possible.

tile38-cli> GC
tile38-cli> SERVER

For me, I just ran a quick test on my side and dumped 25,000 points into a collection, each with 12 fields, and I'm the tile38-server process is only consuming 35MB.

My SERVER response is:

{"aof_size":13895435,
"avg_item_size":1246,
"cpus":16,
"heap_released":100466688,
"heap_size":27269424,
"http_transport":true,
"id":"5b352d2f097f70355bdb047cc71e82ae",
"in_memory_size":3087500,
"max_heap_size":0,
"mem_alloc":27269424,
"num_collections":1,
"num_hooks":0,
"num_objects":25000,
"num_points":21875,
"num_strings":3125,
"pid":66239,
"pointer_size":8,
"read_only":false,
"threads":28,
"version":"1.19.3"}

For my test, each object was a simple point, like: SET fleet truck1 FIELD ... FIELD ... POINT 33 -112. More complex objects, such as GeoJSON features like Polygons with properties and such, will take up more ram.

jas1211 commented 4 years ago

I create 25k simple objects (POINT) with 460 Fields

{
   "ok": true,
   "stats": {
      "aof_size": 34823873,
      "avg_item_size": 10182,
      "cpus": 8,
      "heap_released": 3268608,
      "heap_size": 3333309928,
      "http_transport": true,
      "id": "4758e4686bab14bfc9d75be745f2b30a",
      "in_memory_size": 2475003984,
      "max_heap_size": 0,
      "mem_alloc": 3333309928,
      "num_collections": 2,
      "num_hooks": 0,
      "num_objects": 26904,
      "num_points": 327342,
      "num_strings": 0,
      "pid": 20656,
      "pointer_size": 8,
      "read_only": false,
      "threads": 14,
      "version": "1.19.2"
   },
   "elapsed": "997.8µs"
}

After invoke command GC 5GB ram -> 3GB

tidwall / tile38

Many fields in objects #518