The script runs pytorch_inference, loads the model then continuously runs inference on it. Logging is to std out, the model output is written to a json file. Every 100 request the script asks the pytorch_inference how much memory it is using and this is written to the same json file. grep mem out.json will show that data.
--num_threads_per_allocation and --num_allocations are the parameters to tweak. Increasing either of those will make inference faster and changes in memory should be seen sooner.
First download the elser model locally. Either
The script runs pytorch_inference, loads the model then continuously runs inference on it. Logging is to std out, the model output is written to a json file. Every 100 request the script asks the pytorch_inference how much memory it is using and this is written to the same json file.
grep mem out.jsonwill show that data.Run with
--num_threads_per_allocationand--num_allocationsare the parameters to tweak. Increasing either of those will make inference faster and changes in memory should be seen sooner.