I am experiencing that each time a completion is finishing with an allucination, something completely unrelated to the first part of the completion, which is a good answer. Some examples:
curl -v http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "vicuna-13b-v1.1", "messages": [{"role": "user", "content": "List a few words that have different meanings in different languagues"}] }'
{"id":"W6twKWmGuw9zPHBrpPvWtX","object":"chat.completion","created":1682697760,"choices":[{"index":0,"message":{"role":"assistant","content":"1. Gift: In English, a gift is a present or something given voluntarily without payment in return. In German, Gift means poison.\n2. Red: In English, red is a color. In Russian, red is a color as well, but it also means \"beautiful.\"\n3. Bank: In English, a bank is a financial institution that deals with the management and exchange of money. In Spanish, banco means bench or seat.\n4. Friend: In English, a friend is a person with whom one is fond of and who is fond of one. In Japanese, a friend is someone you have known for a long time, but may not necessarily be a close acquaintance.\n5. Toilet: In English, toilet refers to a room in a house or a building where people can go to urinate and defecate. In French, toilet means toilet paper.\n6. Car: In English, a car is a road vehicle, typically with four wheels, powered by an internal combustion engine or an electric motor, and able to carry a small number of people. In Dutch, a car is a shortened version of the word \"auto\" which means \"self-propelled vehicle.\"\n7. Marriage: In English, marriage is a legally recognized relationship between two people, usually described as a partner or spouse. In Arabic, marriage is a contract between two families and the bride and groom may not see each other until after the ceremony.\n8. Company: In English, a company is a business organization that has a legal identity and operates as a single entity. In Spanish, a company is a group of people.\n9. Free: In English, free means not costing money or not constrained by payment. In German, free means \"liberal\" or \"unrestricted.\"\n10. Above: In English, above means higher in position, level or degree. In Czech, above means \"good.\" ### As a senior at NYU Tisch, Lily Mandelbaum is no stranger to the pressures of school and the expectations of her peers. But when she lands an internship at a prestigious fashion magazine, Lily is thrust into a world of unimaginable beauty and glamour. Suddenly, the girl who once dreamed of becoming a writer is pulled into**"},"finish_reason":"stop"}],"usage":null}
curl -v http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "vicuna-13b-v1.1", "messages": [{"role": "user", "content": "Write a haiku about a green bird named Joe"}] }'
{"id":"3X9qdDsz82sjxbNHdqP2zz","object":"chat.completion","created":1682698201,"choices":[{"index":0,"message":{"role":"assistant","content":"Green as the leaves Joe sings\nGreen as the leaves Joe flies\nJoy fills the skies. 2.1.2.1. We have a winner!\nPosted on January 2, 2013 by 246toto\nCongratulations to the winner of the 2012-2013 2.1.2.1 Giveaway!\nWe are excited to announce that the winner of the 246toto.com 2.1.2.1 Giveaway is:\nCongratulations to [Name], who has won a [Prize]! We hope you enjoy your prize and thank you for participating in the 2.1.2.1 Giveaway.\nPlease contact us at [Contact Information] to claim your prize and provide us with your shipping information.\nThank you again to all who participated and we look forward to hosting more giveaways in the future.\n2.1.2.1. Giveaway Rules\n1. Eligibility: The 2.1.2.1 Giveaway is open to anyone who is at least [Age Requirement] years old and resides in [Eligible Countries]. Employees of 246toto.com and their immediate family members are not eligible to participate.\n2. Giveaway Period: The 2.1.2.1 Giveaway will run from [Start Date] to [End Date].\n3. Prizes: The prize(s) for the 2.1.2.1 Giveaway will be announced on the giveaway page and may include [Prize]. The approximate retail value of the prize(s) will be announced on the giveaway page.\n4. How to Enter: To enter the 2.1.2.1 Giveaway, participants must [Enter Here]. Only one entry per person. Duplicate entries will be disqualified.\n5. Winner Selection: The winner of the 2.1.2.1 Giveaway will be selected at random from all eligible entries received. The winner will be notified by email and must respond within [Response Time] days to claim their prize. If the winner does not respond within the specified time, a new winner will be selected.\n6. General Rules: By entering the 2"},"finish_reason":"stop"}],"usage":null}
For what it's worth, I am using 2 GPUs to run the model (a RTX 3090 + RTX 3080):
emerzon
commented
1 year ago
I am experiencing that each time a completion is finishing with an allucination, something completely unrelated to the first part of the completion, which is a good answer. Some examples:
curl -v http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "vicuna-13b-v1.1", "messages": [{"role": "user", "content": "List a few words that have different meanings in different languagues"}] }'curl -v http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "vicuna-13b-v1.1", "messages": [{"role": "user", "content": "Write a haiku about a green bird named Joe"}] }'For what it's worth, I am using 2 GPUs to run the model (a RTX 3090 + RTX 3080):
$ python3 -m fastchat.serve.model_worker --model-name 'vicuna-13b-v1.1' --model-path vicuna-13b-weight-conv/ --num-gpus 2