charlesollion
commented
3 years ago Multiprocessing is not fully supported yet, can work on it, but for now we should send a single video per AI instance
Open cl3m3nt opened 3 years ago
charlesollion
commented
3 years ago Multiprocessing is not fully supported yet, can work on it, but for now we should send a single video per AI instance
Dear mot developpers,
when using the mot AI inference service, I sometimes faced an error related to multiprocessing capabilities. Although the error may happen over the time, it was not necessarily possible to reproduce from a deterministic perspective. However, a common observed pattern of this error was when the AI server was processing 3 or more requests at a time. See error details below:
[2021-04-26 09:05:41,487] ERROR in app: Exception on / [POST] [00:11<01:35, 2.58s/it] multiprocessing.pool.RemoteTraceback: | 2/41 [00:07<02:06, 3.25s/it] """ Traceback (most recent call last): File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, **kwds)) File "/src/mot/serving/inference.py", line 233, in process_image return localizer_tensorflow_serving_inference(image, SERVING_URL, return_all_scores=True) File "/src/mot/object_detection/query_server.py", line 85, in localizer_tensorflow_serving_inference signature, ratio = preprocess_for_serving(image) File "/src/mot/object_detection/preprocessing.py", line 58, in preprocess_for_serving resized_image, scale_ratio = resize_to_min_dimension(image, min_dimension, max_dimension) File "/src/mot/object_detection/preprocessing.py", line 24, in resize_to_min_dimension image = image.astype(np.float32, copy=False) AttributeError: 'NoneType' object has no attribute 'astype' """
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise raise value File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request return self.view_functionsrule.endpoint File "/src/mot/serving/app.py", line 11, in index return handle_post_request() File "/src/mot/serving/inference.py", line 60, in handle_post_request return handle_file(request.files['file'], upload_folder, **request.form) File "/src/mot/serving/inference.py", line 196, in handle_file total=len(image_paths), File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1178, in iter for obj in iterable: File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next raise value AttributeError: 'NoneType' object has no attribute 'astype' 13.74.59.243 - - [26/Apr/2021 09:05:41] "POST / HTTP/1.1" 500 -