search

facebookresearch / metaseq

Repo for external large-scale work
MIT License
6.45k stars 723 forks source link

Using the Metaseq API #57

Closed Mrs-Hudson closed 2 years ago

Mrs-Hudson commented 2 years ago

Hi I've managed to get the API flask service up with the 125M OPT model by running python -m metaseq_cli.interactive_hosted on an Azure compute instance with 8 GPUs. I followed the suggestions in issue #23 and #26. I don't have browser access to the Azure compute instance. What's the best way to query the API? Should run the flask service in the background and then query it?

stephenroller commented 2 years ago

That's what I do, but you'll need to open the port or forward the port somehow.

Mrs-Hudson commented 2 years ago

Thanks @stephenroller . I tried querying the server this way:

`import requests import json from flask import jsonify

url = 'http://127.0.0.1:6010/completions' headers = {'Content-Type': 'application/json'}

filters = {'prompt':'LinkedIn is a great company2'}

print(jsonify(filters))

params = dict(json=json.dumps(filters))

print(filters) print(json.dumps(filters)) response = requests.post(url, json=filters,headers=headers)

print(response.text)

assert response.status_code == 200 print(response.json()) `

But I see the following error in the server logs: In completion b'{"prompt": "LinkedIn is a great company2"}' {'environ': {'wsgi.version': (1, 0), 'wsgi.url_scheme': 'http', 'wsgi.input': <_io.BufferedReader name=47>, 'wsgi.errors': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, 'wsgi.multithread': True, 'wsgi.multiprocess': False, 'wsgi.run_once': False, 'werkzeug.server.shutdown': <function WSGIRequestHandler.make_environ.<locals>.shutdown_server at 0x7f9ef59cf820>, 'SERVER_SOFTWARE': 'Werkzeug/1.0.1', 'REQUEST_METHOD': 'POST', 'SCRIPT_NAME': '', 'PATH_INFO': '/completions', 'QUERY_STRING': '', 'REQUEST_URI': '/completions', 'RAW_URI': '/completions', 'REMOTE_ADDR': '127.0.0.1', 'REMOTE_PORT': 36506, 'SERVER_NAME': '0.0.0.0', 'SERVER_PORT': '6010', 'SERVER_PROTOCOL': 'HTTP/1.1', 'HTTP_HOST': '127.0.0.1:6010', 'HTTP_USER_AGENT': 'python-requests/2.27.1', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', 'HTTP_ACCEPT': '*/*', 'HTTP_CONNECTION': 'keep-alive', 'CONTENT_TYPE': 'application/json', 'CONTENT_LENGTH': '42', 'werkzeug.request': <Request 'http://127.0.0.1:6010/completions' [POST]>}, 'shallow': False, 'url_rule': <Rule '/completions' (OPTIONS, POST) -> completions>, 'view_args': {}, 'stream': <werkzeug.wsgi.LimitedStream object at 0x7f9f0e918a30>, 'form': ImmutableMultiDict([]), 'files': ImmutableMultiDict([]), '_cached_data': b'{"prompt": "LinkedIn is a great company2"}', 'data': b'{"prompt": "LinkedIn is a great company2"}', 'url': 'http://127.0.0.1:6010/completions'} In completion b'{"prompt": "LinkedIn is a great company2"}' {'environ': {'wsgi.version': (1, 0), 'wsgi.url_scheme': 'http', 'wsgi.input': <_io.BufferedReader name=111>, 'wsgi.errors': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>, 'wsgi.multithread': True, 'wsgi.multiprocess': False, 'wsgi.run_once': False, 'werkzeug.server.shutdown': <function WSGIRequestHandler.make_environ.<locals>.shutdown_server at 0x7f9ef59cfb80>, 'SERVER_SOFTWARE': 'Werkzeug/1.0.1', 'REQUEST_METHOD': 'POST', 'SCRIPT_NAME': '', 'PATH_INFO': '/completions', 'QUERY_STRING': '', 'REQUEST_URI': '/completions', 'RAW_URI': '/completions', 'REMOTE_ADDR': '127.0.0.1', 'REMOTE_PORT': 37038, 'SERVER_NAME': '0.0.0.0', 'SERVER_PORT': '6010', 'SERVER_PROTOCOL': 'HTTP/1.1', 'HTTP_HOST': '127.0.0.1:6010', 'HTTP_USER_AGENT': 'python-requests/2.27.1', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', 'HTTP_ACCEPT': '*/*', 'HTTP_CONNECTION': 'keep-alive', 'CONTENT_TYPE': 'application/json', 'CONTENT_LENGTH': '42', 'werkzeug.request': <Request 'http://127.0.0.1:6010/completions' [POST]>}, 'shallow': False, 'url_rule': <Rule '/completions' (OPTIONS, POST) -> completions>, 'view_args': {}, 'stream': <werkzeug.wsgi.LimitedStream object at 0x7f9ef59e48b0>, 'form': ImmutableMultiDict([]), 'files': ImmutableMultiDict([]), '_cached_data': b'{"prompt": "LinkedIn is a great company2"}', 'data': b'{"prompt": "LinkedIn is a great company2"}', 'url': 'http://127.0.0.1:6010/completions'} 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad request version ('\x00') 2022-05-08 22:38:17 | INFO | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] "âÞô+±½ÞKxÅ4ðYæâË?È#ÈÃIr¤ÌÌ] 'ÄcÁ{w¿–-{V‘ýhdñ– ½ƒÏzÿ " HTTPStatus.BAD_REQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad request version ('\x00') 2022-05-08 22:38:17 | INFO | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] "âÞ­[f¥ô=:© ë"€²GÈ$¿¥_ç8ê7·­·QÃ[›… ‹ëúWJ²ÅIf9úÝ9Þ}«ý2*ÊQ]SÚыƒªÿ " HTTPStatus.BAD_REQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad request version ("\x00\x00|À,À0\x00£\x00\x9f̨̩̪À¯À\xadÀ£À\x9fÀ]ÀaÀWÀSÀ+À/\x00¢\x00\x9eÀ®À¬À¢À\x9eÀ\\ÀÀVÀRÀ$À(\x00k\x00jÀ#À'\x00g\x00@À") -ZcWJþ¿Rßõg‰t#øPxw¡’Ðñ°7h­| wer|À,À0£ŸÌ©Ì¨ÌªÀ¯À­À£ÀŸÀ]ÀaÀWÀSÀ+À/¢žÀ®À¬À¢ÀžÀ\ÀÀVÀRÀ$À(kjÀ#À'g@À" HTTPStatus.BAD_REQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad request version ("Wág¾\x9dèL¦\x9d\x18\x96\x00\x00¦À,À0\x00£\x00\x9f̨̩̪À¯À\xadÀ£À\x9fÀ]ÀaÀWÀSÀ+À/\x00¢\x00\x9eÀ®À¬À¢À\x9eÀ\ÀÀVÀRÀ$À(\x00k\x00jÀsÀw\x00Ä\x00ÃÀ#À'\x00g\x00@ÀrÀv\x00¾\x00½À") =€A˜¾x†ùìÝNÏÎEßy Wág¾èL¦–¦À,À0£ŸÌ©Ì¨ÌªÀ¯À­À£ÀŸÀ]ÀaÀWÀSÀ+À/¢žÀ®À¬À¢ÀžÀ\ÀÀVÀRÀ$À(kjÀsÀwÄÃÀ#À'g@ÀrÀv¾½À" HTTPStatus.BAD_REQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x00\xad\x01\x00\x00©\x03\x02') 2022-05-08 22:38:17 | INFO | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] "­©á×(‹Ék4üâA~0&54:ã¥jnðWŸÎFEûf>ÀÀ" HTTPStatus.BAD_REQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad request version ('î\x8ed¿Ra\x05ÏØFþ®1\x9cùDÝ=¨ø\x00½É.\x8b´ëh\x00\x00¢À\x14À') «H îŽd¿RaÏØFþ®1œùDÝ=¨ø½É.‹´ëh¢ÀÀ" HTTPStatus.BAD_REQUEST -May/2022 22:38:17] " 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x00\xad\x01\x00\x00©\x03\x01\x87BÐ\x1bé»ìü\x96\x7fx\x04\x1b\x91\x8f\x8aT\x86Êå\x80õc') 2022-05-08 22:38:17 | INFO | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] "­©‡BÐ�»ìü–x�ŠT†Êå€õc d{Û0¢ïàë>ÀÀ" HTTPStatus.BADREQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x01\x11\x01\x00\x01') Œ¸Üܕ¶ÿVùN$³Qڦˌ=bò–Ñ5¦¢ÀÀ" HTTPStatus.BAD_REQUEST -May/2022 22:38:17] " 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x00\x00j\x01\x00\x00f\x03\x00µ<\x82.\x15U¶\x87ù\x9cÀ') 2022-05-08 22:38:17 | INFO | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] "jfµ<‚.U¶‡ùœÀM(q–YûÈïÑ­¿Z–ðõg¥>ÀÀ" HTTPStatus.BAD_REQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad request version ('¾\x92¹\x00\x00¢À\x14À') 2022-05-08 22:38:17 | INFO | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] "ÎÊÆöÿ•@np^™”²ÕcF¸–Îc?OÖ¾’¹¢ÀÀ" HTTPStatus.BAD_REQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x01\x19\x01\x00\x01\x15\x03\x03O\x9b4Å°føuBÔ\x00\x9f-\x7f&i\x98,') 2022-05-08 22:38:17 | INFO | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] "O›4Å°føuBԟ-&i˜,…QXüžÑLݙÎèt‘†À0À,À(À$ÀÀ" HTTPStatus.BAD_REQUEST - 2022-05-08 22:38:17 | ERROR | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] code 400, message Bad HTTP/0.9 request type ('\x16\x03\x01\x01\x87\x01\x00\x01\x83\x03\x03\x1bæ\x8f?>äÜ\x0ehWtåëf\x9eµ@') 2022-05-08 22:38:17 | INFO | werkzeug | 127.0.0.1 - - [08/May/2022 22:38:17] "‡ƒ�?>äÜhWtåëfžµ@H¯M_þ;¾—J÷ÿOôÀ0À,À(À$ÀÀ" HTTPStatus.BAD_REQUEST -

Mrs-Hudson commented 2 years ago

If I replace http with https in the API URL I see the below: return request('post', url, data=data, json=json, **kwargs) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/anaconda/envs/azureml_py38/lib/python3.8/site-packages/requests/adapters.py", line 517, in send raise SSLError(e, request=request) requests.exceptions.SSLError: HTTPSConnectionPool(host='127.0.0.1', port=6010): Max retries exceeded with url: /completions (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1123)')))

Mrs-Hudson commented 2 years ago

This issue was solved by adding verfiy=False to the post request

stephenroller commented 2 years ago

Here's an example of hitting it:

curl -k http://localhost:6010/completions \
  -H "Content-Type: application/json" \
  -d '{
  "prompt": "A chat between a teacher and student who wants to learn about tacos.\n\nTeacher: Hi there. What would you like to learn about today?\nStudent:",
  "temperature": 1.0,
  "max_tokens": 32, "min_tokens": 4,
  "top_p": 0.9, "n": 1,
  "echo": false, "stop": "\n"
}'| jq .
nickums commented 2 years ago

I cannot find metaseq-api-local anywhere in OPT