ETL query OOM - Githubissues

cisco-ie / tdm

Telemetry Data Mapper to ease data discovery, correlation, and usage with YANG, MIBs, etc.

Apache License 2.0

37 stars 4 forks source link

ETL query OOM #38

Open remingtonc opened 5 years ago

remingtonc commented 5 years ago

The dbms service is getting OOMurdered during ETL query. #36

Likely due to flattening in query. Should move to streaming results. https://github.com/cisco-ie/tdm/blob/05eb9c91ddefc8dfa21c919bec880a4d3a2d9fcc/etl/src/search.py#L39

remingtonc commented 4 years ago

Seeing this hit ~60 GB of RAM usage. It appears that ArangoDB is computing the entire query, and then streaming the results in the batch size. My impression was that computation and results would both be on the fly and per batch size - need to investigate.

remingtonc commented 4 years ago

We could move the search entry into the YANG parsing section...

remingtonc commented 4 years ago

Tracking in etl-search-stream.