API memory usage is proportional to dataset size

VizierDB / web-api-async

Vizier Asynchronous Web API

Apache License 2.0

2 stars 2 forks source link

API memory usage is proportional to dataset size #37

Open okennedy opened 5 years ago

okennedy commented 5 years ago

Snapshotted while loading a 26GB Parquet file:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND     
23956 vizier_+  20   0 12.868g 0.012t    180 D   1.0 79.1   0:06.68 python

The load happens entirely in Mimir. Why does the API have 12GB resident?