Speed of the simplification

[igormis]

I am using this code to simplify sentence by sentence:

def paraphrase():
    source_filepath = get_temp_filepath()
    pred_filepath = get_temp_filepath()
    data = request.get_data().decode()

    write_lines([word_tokenize(data)], source_filepath)
    start=time.time()
    simplifier(source_filepath, pred_filepath)
    print(time.time()-start)
    for line in yield_lines(pred_filepath):
      output = line
    os.remove(source_filepath)
    os.remove(pred_filepath)
    return output

And the transformation time is from 0.8 to 2.0 seconds is it possible to speed this? I need the response time to be bellow 0.3 seconds?

[igormis]

I forgot the part where I load the models:

# Load best model
best_model_dir = prepare_models()
recommended_preprocessors_kwargs = {
'LengthRatioPreprocessor': {'target_ratio': 0.95},
'LevenshteinPreprocessor': {'target_ratio': 0.75},
'WordRankRatioPreprocessor': {'target_ratio': 0.75},
'SentencePiecePreprocessor': {'vocab_size': 10000},
}
preprocessors = get_preprocessors(recommended_preprocessors_kwargs)
simplifier = get_fairseq_simplifier(best_model_dir, beam=8)
simplifier = get_preprocessed_simplifier(simplifier, preprocessors=preprocessors)

[louismartin]

Two checks: 1) Make sure that you only load the model only once (and not at every request). 2) Make sure the model is loaded on GPU for faster inference

If you are already doing these two things, there is not many easy solutions to speed up the inference. You could batch requests together to process them all at once but it might not fit your use case.