We are currently using Flask in single-threaded mode. This is warned against, since it is not production ready. Also, we are doing so much IO that even when we are under 100% use our CPU usage is only ~25%.
We should try using gunicorn, but make sure it is not in a multi-threaded mode (tensorflow is not happy with multi-threading). So either one of the asyinc workers, or the sync one.
epa095
commented
4 years ago
We are currently using Flask in single-threaded mode. This is warned against, since it is not production ready. Also, we are doing so much IO that even when we are under 100% use our CPU usage is only ~25%.
We should try using gunicorn, but make sure it is not in a multi-threaded mode (tensorflow is not happy with multi-threading). So either one of the asyinc workers, or the sync one.