pytorch result is unpredictable

[andysharma1997]

I tried running roberta for checking sentence similarity score and used torchvision my code look like this

from flask import request
from flask import Flask
from fairseq.models.roberta import RobertaModel

app = Flask(__name__)
roberta = None

def init_roberta():
    roberta = RobertaModel.from_pretrained(
    '/home/vv/',checkpoint_file='checkpoint_best.pt',
    data_name_or_path='/home/vv/STS-B-bin')
    roberta.cuda()
    return roberta

from flask import jsonify

@app.route("/sentence_similarity_roberta",methods=["POST","GET"])
def check_smilarity_reboerta():
    global roberta
    if roberta == None:
        print('Setting up roberta')
        roberta = init_roberta()
    list1=request.form.getlist('sentence1')
    list2=request.form.getlist('sentence2')
    print(list1)
    arg=request.args.get('model')
    listx=list1[0].split("[")
    listy=listx[1].split("]")
    listz=listy[0].split(",")
    input_sentences1=[]
    for item in listz:
        input_sentences1.append(item.replace('"',""))
    listx1=list2[0].split("[")
    listy1=listx1[1].split("]")
    listz1=listy1[0].split(",")
    input_sentences2=[]
    for item in listz1:
        input_sentences2.append(item.replace('"',""))
    tokens = roberta.encode(input_sentences1[0], input_sentences2[0])
    features = roberta.extract_features(tokens)
    predictions = roberta.model.classification_heads['sentence_classification_head'](features)
    return jsonify(input_sentences1[0]+"  "+input_sentences2[0]+"  "+ "similarity_score:"+str(predictions.item()))

if __name__=='__main__':
    app.run(host="0.0.0.0",port='8080')

when i run apache bench to test this the score value that i get is totally random this is the command that i used for apache bench marking ab -c 10 -n 10 -v 3 -T 'multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' -p /home/andy/benchmark/ignore1.txt http://192.168.0.199:8080/sentence_similarity_roberta

I used gunicorn for running the service and the command that i used is : gunicorn --threads=150 -b:8080 hello:app

The values that i got from apache are :

Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.0.199 (be patient)...INFO: POST header == 
---
POST /sentence_similarity_roberta HTTP/1.0
Content-length: 236
Content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
Host: 192.168.0.199:8080
User-Agent: ApacheBench/2.3
Accept: */*

---
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 46

"hi  hi  similarity_score:0.6313190460205078"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 46

"hi  hi  similarity_score:0.7136876583099365"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 47

"hi  hi  similarity_score:0.18845096230506897"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 46

"hi  hi  similarity_score:0.7439256310462952"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 46

"hi  hi  similarity_score:0.7837516069412231"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 46

"hi  hi  similarity_score:0.7196391224861145"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 46

"hi  hi  similarity_score:0.7833828926086426"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 46

"hi  hi  similarity_score:0.7694887518882751"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 46

"hi  hi  similarity_score:0.7049023509025574"

LOG: Response code = 200
LOG: header received:
HTTP/1.0 200 OK
Server: gunicorn/19.9.0
Date: Wed, 28 Aug 2019 13:09:53 GMT
Connection: close
Content-Type: application/json
Content-Length: 45

"hi  hi  similarity_score:0.760443925857544"

LOG: Response code = 200
..done

Server Software:        gunicorn/19.9.0
Server Hostname:        192.168.0.199
Server Port:            8080

Document Path:          /sentence_similarity_roberta
Document Length:        46 bytes

Concurrency Level:      10
Time taken for tests:   0.183 seconds
Complete requests:      10
Failed requests:        2
   (Connect: 0, Receive: 0, Length: 2, Exceptions: 0)
Total transferred:      1980 bytes
Total body sent:        4540
HTML transferred:       460 bytes
Requests per second:    54.53 [#/sec] (mean)
Time per request:       183.385 [ms] (mean)
Time per request:       18.338 [ms] (mean, across all concurrent requests)
Transfer rate:          10.54 [Kbytes/sec] received
                        24.18 kb/s sent
                        34.72 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    1   0.0      1       1
Processing:   162  174   7.3    177     183
Waiting:      162  174   7.3    177     183
Total:        163  175   7.3    178     183

Percentage of the requests served within a certain time (ms)
  50%    178
  66%    179
  75%    180
  80%    182
  90%    183
  95%    183
  98%    183
  99%    183
 100%    183 (longest request)

Please let me know why is the value of similarity score not coming same for every request Any help would be appriciated Thank you

[ngoyal2707]

can you try putting model in eval mode by calling roberta.eval() to disable dropout while inferencing?

[andysharma1997]

It worked now i am getting consistent result thankyou sir for the support.Can you please let me know why this issue was there or any links to read upon it

[ngoyal2707]

eval() disables some features like Dropout and BatchNorm for inferencing. Dropout is only used for training as regularization and should be turned off for inference time. more details: https://discuss.pytorch.org/t/trying-to-understand-the-meaning-of-model-train-and-model-eval/20158/2

[andysharma1997]

it was really helpful sir thanks a lot once again .