Open funkaikai opened 4 months ago
However, on another Windows machine, it was run directly. [INFO ] - Number of inter-op threads is 1 [INFO ] - Number of intra-op threads is 1 [INFO ] - Load PyTorch (2.1.1) in 0.019 ms. [INFO ] - Running MultithreadedBenchmark on: [cpu()]. [INFO ] - Multithreading inference with 2 threads. Loading: 100% |========================================| [INFO ] - Model traced_resnet18 loaded in: 1583.521 ms. [INFO ] - Warmup with 2 iteration ... [INFO ] - Warmup latency, min: 37.997 ms, max: 119.212 ms [INFO ] - Completed 100 requests [INFO ] - Inference result: [-0.06938224, 0.616994, -1.9312545 ...] [INFO ] - Throughput: 46.66, completed 100 iteration in 2143 ms. [INFO ] - Model loading time: 1583.521 ms. [INFO ] - total P50: 42.663 ms, P90: 44.797 ms, P99: 55.650 ms [INFO ] - inference P50: 42.545 ms, P90: 44.657 ms, P99: 55.518 ms [INFO ] - preprocess P50: 0.074 ms, P90: 0.092 ms, P99: 0.259 ms [INFO ] - postprocess P50: 0.041 ms, P90: 0.054 ms, P99: 0.346 ms
why ?
hi, We are trying to use the DJL (Deep Java Library) framework and have conducted benchmark tests on different servers, finding that the throughput performance varies on different machines. One can reach a throughput of 60/s, which is a latest Windows machine with i7-13700H. Another machine can reach a throughput of 10/s, which is a 2019 MacBook Pro with i9.
1.I would like to know about the approximate time consumption of model prediction for our reference. 2.As you may know, which companies are currently using it in a production environment?