LucienShui
commented
1 year ago Sorry, there was an error in my previous documentation. The max_new_tokens parameter should be an integer. Apologies for any confusion caused by this mistake.
The error is arising from the JSON payload that you are sending with your POST request. The max_new_tokens parameter expects an integer, but you are sending a string.
Here is the corrected curl command:
curl -X POST http://localhost:8000/api/generate/ -d '{"inputs": "import numpy as np", "parameters": {"max_new_tokens": 512}}'In this corrected command, 512 is provided as an integer, not a string, by removing the quotes around it.
when i run the command : curl -X POST http://localhost:8000/api/generate/ -d '{"inputs": "import numpy as np", "parameters": {"max_new_tokens": "512"}}'
The error is as follows:
INFO: Started server process [36622] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) 2023-05-26 16:27:34,726 INFO 127.0.0.1:36548 inputs = "import numpy as np" INFO: 127.0.0.1:36548 - "POST /api/generate/ HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/usr/local/anaconda3/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 428, in run_asgi result = await app( # type: ignore[func-returns-value] File "/usr/local/anaconda3/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in call return await self.app(scope, receive, send) File "/usr/local/anaconda3/lib/python3.10/site-packages/fastapi/applications.py", line 276, in call await super().call(scope, receive, send) File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/applications.py", line 122, in call await self.middleware_stack(scope, receive, send) File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in call raise exc File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in call await self.app(scope, receive, _send) File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in call await self.app(scope, receive, send) File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in call raise exc File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in call await self.app(scope, receive, sender) File "/usr/local/anaconda3/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call raise e File "/usr/local/anaconda3/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call await self.app(scope, receive, send) File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/routing.py", line 718, in call await route.handle(scope, receive, send) File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle await self.app(scope, receive, send) File "/usr/local/anaconda3/lib/python3.10/site-packages/starlette/routing.py", line 66, in app response = await func(request) File "/usr/local/anaconda3/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app raw_response = await run_endpoint_function( File "/usr/local/anaconda3/lib/python3.10/site-packages/fastapi/routing.py", line 163, in run_endpoint_function return await dependant.call(values) File "/home/hbshao/Workspace/models/huggingface-vscode-endpoint-server/main.py", line 22, in api generated_text: str = generator.generate(inputs, parameters) File "/home/hbshao/Workspace/models/huggingface-vscode-endpoint-server/generators.py", line 27, in generate json_response: dict = self.pipe(query, generation_config=config)[0] File "/usr/local/anaconda3/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 209, in call return super().call(text_inputs, kwargs) File "/usr/local/anaconda3/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1109, in call return self.run_single(inputs, preprocess_params, forward_params, postprocess_params) File "/usr/local/anaconda3/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1116, in run_single model_outputs = self.forward(model_inputs, forward_params) File "/usr/local/anaconda3/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1015, in forward model_outputs = self._forward(model_inputs, forward_params) File "/usr/local/anaconda3/lib/python3.10/site-packages/transformers/pipelines/text_generation.py", line 251, in _forward generated_sequence = self.model.generate(input_ids=input_ids, attention_mask=attention_mask, *generate_kwargs) File "/usr/local/anaconda3/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(args, **kwargs) File "/usr/local/anaconda3/lib/python3.10/site-packages/transformers/generation/utils.py", line 1320, in generate generation_config.max_length = generation_config.max_new_tokens + input_ids_seq_length TypeError: can only concatenate str (not "int") to str