search

sgl-project / sglang

SGLang is a fast serving framework for large language models and vision language models.
https://sgl-project.github.io/
Apache License 2.0
6.21k stars 529 forks source link

[BUG] Problems with jump forward decoding #2045

Open merrymercy opened 1 week ago

merrymercy commented 1 week ago

There are still some issues with jump forward decoding for both backends (outlines and xgrammar). The outputs w/ jump forward are different from the outputs w/o jump forward. I tested the first 10 examples in https://github.com/sgl-project/sglang/tree/main/benchmark/json_schema and found the following issues.

Issues with Outlines

There is an extra space " " before the colon ":" for each key in the json. You can compare the outputs below.

The outputs of outlines w/ jumpforward

{"ssid" : "OfficeNetSecure", "securityProtocol" : "WPA2-Enterprise", "bandwidth" : "1300 Mbps on the 5 GHz band"}
{"/" : {"device" : "string", "mount_point" : "string", "file_system_type" : "string", "options" : "string", "dump" : "0", "pass" : "1"}}
{"campaignID" : "CAMP123456", "productID" : "PROD7891011", "startDate" : "2023-06-01", "endDate" : "2023-06-30", "discountDetails" : "15% off on all purchases"}
{"reservationID" : "AH-158394", "guestName" : "Alexander Hamilton", "reservationTime" : "2023-04-15T19:30:00", "specialRequests" : ["table by the window", "surprise dessert for special occasion"]}
{"HomeImprovement" : {"room_interest" : "living room", "budget" : 500, "preferred_style" : "minimalist", "project_ideas" : ["installing floating shelves", "creating a gallery wall", "repainting the walls", "upcycling old furniture", "adding greenery with low-maintenance indoor plants"]}}
{"deviceID" : "MON123456", "patientID" : "PAT654321", "metrics" : {"heartRate" : 78, "bloodPressure" : "120/80 mmHg", "oxygenSaturation" : 98}, "timestamp" : "2023-04-05T14:30:00Z"}
{"FitnessTracking" : {"current_health_status" : {"weight" : 70, "height" : 175, "heart_rate" : 62}, "health_goals" : ["increase muscle mass", "improve cardiovascular endurance", "enhance flexibility"], "recommended_routines" : ["Strength training sessions three times a week focusing on major muscle groups", "Cardiovascular exercises such as running or cycling for at least 30 minutes, five days a week", "Daily stretching exercises to improve flexibility, including yoga or pilates"]}}
{"seatNumber" : "12A", "serviceType" : "vegetarian meal", "specialInstructions" : "gluten-free"}
{"SKU" : "TOB-1928", "quantity" : 150, "restockDate" : "2023-04-15", "supplier" : "Global Tobacco Ltd."}
{"patentId" : "US98765432A", "applicationDate" : "2021-07-15", "inventorNames" : ["Dr. Alice Smith", "Dr. Bob Jones"], "currentStatus" : "Pending Examination"}

The outputs of outlines w/o jumpforward

{"ssid": "OfficeNetSecure", "securityProtocol": "WPA2-Enterprise", "bandwidth": "1300 Mbps on the 5 GHz band"}
{"/": {"device": "string", "mount_point": "string", "file_system_type": "string", "options": "string", "dump": "0", "pass": "1"}}
{"campaignID": "CAMP123456", "productID": "PROD7891011", "startDate": "2023-06-01", "endDate": "2023-06-30", "discountDetails": "15% off on all purchases"}
{"reservationID": "AH-158394", "guestName": "Alexander Hamilton", "reservationTime": "2023-04-15T19:30:00", "specialRequests": ["table by the window", "surprise dessert for special occasion"]}
{"HomeImprovement": {"room_interest": "living room", "budget": 500, "preferred_style": "minimalist", "project_ideas": ["installing floating shelves", "creating a gallery wall", "repainting the walls", "upcycling old furniture", "adding greenery with low-maintenance indoor plants"]}}
{"deviceID": "MON123456", "patientID": "PAT654321", "metrics": {"heartRate": 78, "bloodPressure": "120/80 mmHg", "oxygenSaturation": 98}, "timestamp": "2023-04-05T14:30:00Z"}
{"FitnessTracking": {"current_health_status": {"weight": 70, "height": 175, "heart_rate": 62}, "health_goals": ["increase muscle mass", "improve cardiovascular endurance", "enhance flexibility"], "recommended_routines": ["Strength training sessions three times a week focusing on major muscle groups", "Cardiovascular exercises such as running or cycling for at least 30 minutes, five days a week", "Daily stretching exercises to improve flexibility, including yoga or pilates"]}}
{"seatNumber": "12A", "serviceType": "vegetarian meal", "specialInstructions": "gluten-free"}
{"SKU": "TOB-1928", "quantity": 150, "restockDate": "2023-04-15", "supplier": "Global Tobacco Ltd."}
{"patentId": "US98765432A", "applicationDate": "2021-07-15", "inventorNames": ["Dr. Alice Smith", "Dr. Bob Jones"], "currentStatus": "Pending Examination"}

Issues with xgrammar

Sometimes, the output will include an additional "/" before the string value. For example, in the last line in the following outputs, the output w/ jumpforward for the key "inventorNames" is "/Dr. Alice Smith", but the correct output should be "Dr. Alice Smith". It also happens for the key "health_goals" in the following examples. This is a very critical bug.

The outputs of xgrammar w/ jumpforward

{"ssid": "OfficeNetSecure", "securityProtocol": "WPA2-Enterprise", "bandwidth": "1300 Mbps (5 GHz)"}
{"/": {"device": "string", "mount_point": "string", "file_system_type": "string", "options": "string", "dump": "string", "pass": "string"}}
{"campaignID": "CAMP123456", "productID": "PROD7891011", "startDate": "2023-06-01", "endDate": "2023-06-30", "discountDetails": "15% off on all purchases"}
{"reservationID": "AH-158394", "guestName": "Alexander Hamilton", "reservationTime": "2023-04-15T19:30:00", "specialRequests": ["> A table by the window", "Note to the chef: Celebrating a special occasion, surprise dessert would be wonderful"]}
{"HomeImprovement": {"room_interest": "living room", "budget": 500, "preferred_style": "minimalist", "project_ideas": [">installing floating shelves for a clean look and extra storage", "creating a gallery wall with my favorite prints and photographs", "repainting the walls with a neutral color palette for a fresh feel", "upcycling old furniture with a new coat of paint or new upholstery", "adding some greenery with low-maintenance indoor plants"]}}
{"deviceID": "MON123456", "patientID": "PAT654321", "metrics": {"heartRate": 78, "bloodPressure": "120/80 mmHg", "oxygenSaturation": 98}, "timestamp": "2023-04-05T14:30:00Z"}
{"FitnessTracking": {"current_health_status": {"weight": 70, "height": 175, "heart_rate": 62}, "health_goals": ["/increase muscle mass", "improve cardiovascular endurance", "enhance flexibility"], "recommended_routines": ["/Strength training sessions three times a week focusing on major muscle groups", "Cardiovascular exercises such as running or cycling for at least 30 minutes, five days a week", "Daily stretching exercises to improve flexibility, including yoga or pilates"]}}
{"seatNumber": "12A", "serviceType": "vegetarian meal", "specialInstructions": "gluten-free"}
{"SKU": "TOB-1928", "quantity": 150, "restockDate": "2023-04-15", "supplier": "Global Tobacco Ltd."}
{"patentId": "US98765432A", "applicationDate": "2021-07-15", "inventorNames": ["/Dr. Alice Smith", "Dr. Bob Jones"], "currentStatus": "Pending Examination"}

The outputs of xgrammar w/o jumpforward

{"ssid": "OfficeNetSecure", "securityProtocol": "WPA2-Enterprise", "bandwidth": "1300 Mbps on the 5 GHz band"}
{"/": {"device": "string", "mount_point": "string", "file_system_type": "string", "options": "string", "dump": "string", "pass": "string"}}
{"campaignID": "CAMP123456", "productID": "PROD7891011", "startDate": "2023-06-01", "endDate": "2023-06-30", "discountDetails": "15% off on all purchases"}
{"reservationID": "AH-158394", "guestName": "Alexander Hamilton", "reservationTime": "2023-04-15T19:30:00", "specialRequests": ["table by the window", "surprise dessert for special occasion"]}
{"HomeImprovement": {"room_interest": "living room", "budget": 500, "preferred_style": "minimalist", "project_ideas": ["installing floating shelves", "creating a gallery wall", "repainting the walls", "upcycling old furniture", "adding greenery with low-maintenance indoor plants"]}}
{"deviceID": "MON123456", "patientID": "PAT654321", "metrics": {"heartRate": 78, "bloodPressure": "120/80 mmHg", "oxygenSaturation": 98}, "timestamp": "2023-04-05T14:30:00Z"}
{"FitnessTracking": {"current_health_status": {"weight": 70, "height": 175, "heart_rate": 62}, "health_goals": ["increase muscle mass", "improve cardiovascular endurance", "enhance flexibility"], "recommended_routines": ["Strength training sessions three times a week focusing on major muscle groups", "Cardiovascular exercises such as running or cycling for at least 30 minutes, five days a week", "Daily stretching exercises to improve flexibility, including yoga or pilates"]}}
{"seatNumber": "12A", "serviceType": "vegetarian meal", "specialInstructions": "gluten-free"}
{"SKU": "TOB-1928", "quantity": 150, "restockDate": "2023-04-15", "supplier": "Global Tobacco Ltd."}
{"patentId": "US98765432A", "applicationDate": "2021-07-15", "inventorNames": ["Dr. Alice Smith", "Dr. Bob Jones"], "currentStatus": "Pending Examination"}

Generate outputs

python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --port 30000
~/sglang/benchmark/json_schema# python3 bench_sglang.py --num-json 10 --parallel 1 --port 30000
merrymercy commented 19 hours ago

Reason for the additional space in "outlines w/ jumpforward": The jump forward will jump "ssid" once. However, the last " should be tokenized together with :. This leads to the difference. The solution is to only jump "ssid.