search

unicef / kindly

GNU Affero General Public License v3.0
24 stars 17 forks source link

Monitor / Improve API request latencies #29

Open nathanfletcher opened 3 years ago

nathanfletcher commented 3 years ago

This issue is to help keep track of API latency times and to put in the effort to document and improve on it. Currently, response times are between 5.185s - 7s

Test response times in Terminal

time curl \
  -X POST \
  -H "Content-Type: application/json" \
  https://kindly-api.azurewebsites.net/detect \
  -d '{"text":"this movie is great"}'
nathanfletcher commented 3 years ago

As a strategy to increase latency, we could convert the python model to a Tensorflow.js model and use it in a NodeJS endpoint to see if that would be faster This can be a good resource to achieve that.

TensorFlow.js: Convert a Python SavedModel to TensorFlow.js format

@AyiteyDjaba May have other suggestions too

AyiteyDjaba commented 3 years ago

From what I have seen so far, you can only convert a model to Tensorflow.js if it was originally built with Tensorflow. I have no idea if the cardiffnlp model is based on Tensorflow. However, in the grand scheme of things, moving the backend completely to NodeJS would not be a problem if we are training our own model because we can train and serve the model with tensorflowjs