vurusovs
commented
10 months ago I confirm memory leak, it's reproducible w/o threads execution on CPU. Please, find code snippet below:
#include <string>
#include <thread>
#include "openvino/openvino.hpp"
#include "openvino/pass/serialize.hpp"
size_t getSystemDataByName(char* name) {
auto parseLine = [](std::string line) -> size_t {
std::string res = "";
for (auto c : line)
if (isdigit(c))
res += c;
if (res.empty())
throw std::runtime_error("Can't get system memory values");
return std::stoul(res);
};
FILE* file = fopen("/proc/self/status", "r");
size_t result = 0;
bool status = false;
if (file != nullptr) {
char line[128];
while (fgets(line, 128, file) != NULL) {
if (strncmp(line, name, strlen(name)) == 0) {
result = parseLine(line);
status = true;
break;
}
}
fclose(file);
}
if (!status)
throw std::runtime_error("Can't get system memory values");
return result;
}
size_t getVmRSSInKB() {
return getSystemDataByName(const_cast<char*>("VmRSS:"));
}
ov::Core core;
std::shared_ptr<ov::Model> model;
std::string device_name = "CPU";
auto xrand = []() {
unsigned long align = 32;
return (rand() % (2048-align) & ~(align-1)) + align;
};
void StartThreadForRun() {
auto compiled_model = core.compile_model(model, device_name);
for (;;) {
ov::InferRequest request = compiled_model.create_infer_request();
request.infer();
std::cout << std::this_thread::get_id() << " " << getVmRSSInKB() << std::endl;
}
}
int main() {
model = core.read_model("./MaskRCNN-12.onnx");
ov::serialize(model, "./MaskRCNN-12.xml", "./MaskRCNN-12.bin");
model = core.read_model("./MaskRCNN-12.xml", "./MaskRCNN-12.bin");
model->reshape(ov::Shape{3, xrand(), xrand()});
StartThreadForRun();
}It produces next results:
v-Golubev
notaz
OpenVINO Version
2023.1.0
Operating System
Ubuntu 20.04 (LTS)
Device used for inference
CPU
Framework
None
Model used
Mask R-CNN
Issue description
We need to infer using a custom model with unlimited amount of input shapes that are not known in advance from multiple threads. Dynamic shape support is used for that purpose. Large amount (but not unlimited if course) of RAM is available, so a separate model is compiled for each thread. I'm well aware of larger memory usage for dynamic shapes, however as the program runs the memory usage never seems to stop growing. I'm aware about the
CPU_RUNTIME_CACHE_CAPACITYparameter, but setting it to 0 doesn't seem to be helping here.I can't share the model, however I was able to reproduce this with Mask R-CNN public model.
On a 72 thread machine, if a fixed shape of [3,1024,1024] is repeatedly used, the memory usage peaks at around 76.3G. With random shapes it reaches 128G in 2-3 minutes, 192G in ~1h and keeps growing at a rate of ~600M/minute rss and ~7G/minute virtual. The amount of mappings (/proc/[pid]/maps) is also growing, many new regions have x permission set so it's probably some generated code cache that keeps growing.
Also happens in master branch as of 755651cd34c6c6195039fd17dbc3eac1619e2a88.
Step-by-step reproduction
Run the following program, preferable on a machine with large amount of CPU cores. Observe memory usage growing endlessly with
topor another tool of your choice.Relevant log output
No response
Issue submission checklist