When writing with JSON Schema Parser, model can output empty object or invalid JSON.

[kmccleary3301]

I've been toying around with this library, and I'm planning on using it in production for some upcoming projects. One issue I've ran into is that the JSON Schema parser, at least with vLLM, does not strictly enforce required fields. My model sometimes outputs empty bracket enclosures, or even nothing at all.

I'm still familiarizing myself with this library, and I'd like to make a PR to help fix this, as this being a possibility defeats the purpose of JSON/regex enforcement to begin with.

Here's the result of the colab_vllm_integration.ipynb example, with my modified code on one of the cells. Also, I used my own modification which I've PRed that allowed me to ban the newline character.

from lmformatenforcer import JsonSchemaParser
from pydantic import BaseModel

class AnswerFormat(BaseModel):
    name: str = ''
    comment: Optional[Union[Literal["hello", "how are \|\|\"\"you?"]]] = None
    age: float = 0.0
    isActive: bool = False
    nameList: List[str] = None

question = 'Please give me information about Michael Jordan.'
# question_with_schema = f'{question}{AnswerFormat.schema_json()}'
prompt = get_prompt(question)

# display_header("Prompt:")
# display_content(prompt)

display_header("Answer, With json schema enforcing:")

print("RESULT")
for i in range(50):
    result = vllm_with_character_level_parser(prompt, JsonSchemaParser(AnswerFormat.schema()))
    # display_content(result)

    print(result)

RESULT
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.28it/s]
{ "name": "Michael Jordan", "nameList": [ "basketball", "professional basketball", "hoops", "player title" ], "age": 59, "isActive": false, "comment": "hello" }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.22it/s]
 {"name":"Michael Jordan","nameList":["Michael Jordan"],"age":58,"isActive":false,"comment": "hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.07it/s]
{ "name": "Michael Jordan", "age": 56, "nameList": ["NBA Rajon Rondo", "NBA Doncic Luka", "NBA LeBron James", "NBA Patrick Ewing", "NBA JaVale McGee"], }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.59it/s]
"comment": "hello"}ember" ],"retired roundballer", "Chicago Bulls legend",
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.97it/s]
 { "name": "Michael Jordan", "nameList": ["Michael Jordan", "Michael B. Jordan", "Michael T. Jordan", "Michael J. Jordan"] } 
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 21.65it/s]
{ }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.70it/s]
 {"name":"Michael Jordan","age":57,"nameList":["player","entrepreneur","speaker"], "isActive":true} 
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.18it/s]
    {   "name": "Michael Jordan",   "nameList": ["Michael Jordan", "Basketball legend", "Professional basketball player", "2021 NBA Hall of Famer", "Olympic gold medalist", "5-time NBA MVP", "6-time NBA Finals champion", "14-time NBA All-Star", "12,923 career points"],   "age": 59,  "isActive": false,  "comment": "hello"  }   
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.47it/s]
 {"name": "Michael Jordan", "age": 59, "nameList": ["Michael Jordan"]}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.70it/s]
{ "name": "Michael Jordan", "age": 57, "nameList": [" basketball player", " motorsport racer", " film actor", " film producer", " husband", " father", " best friends", " speakers", " intellectual property billionaire", " business executive" ], "comment": "hello", "isActive": false}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.84it/s]
{ "name": "Michael Jordan", "nameList": [ "Basketball player", "Athlete", "NBA veteran", "Former Team player", "Family man", "ESPN executives" ], "age": 59, "isActive": false, "comment": "hello" }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 21.64it/s]
{}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 18.92it/s]
 { }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.65it/s]
 {"name":"Michael Jordan","age":56,"nameList":["Charlotte Hornets","Detroit Pistons","Portland Trail Blazers","Milwaukee Bucks","Los Angeles Clippers"]} 
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.91it/s]
"Big Bang Theory"], "comment": "hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 16.88it/s]
 { }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 19.00it/s]
 { }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 16.88it/s]
 { }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.89it/s]
 {"name" : "Michael Jordan", "age" : 56, "nameList" : ["10x basketball champion", "Legendary NBA player", "Also known as 'ThesixGod' and 'Fluke'"], "isActive" : false, "comment" : "hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.40it/s]
{ "name": "Michael Jordan", "age": 56, "nameList": ["Michael Jordan", " Michael Jordan", "mike jordan"], "isActive": false, "comment": "hello" }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.69it/s]
 {"name":"Colin","age":33,"nameList":[ ", brig"],"isActive":true,"comment": "hello"} 
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 16.87it/s]
 { }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.03it/s]
{ "name": "Michael Jordan", "nameList": [], "isActive": true, "age": 59, "comment": "hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.60it/s]
    {   "name": "Michael Jordan",   "nameList": [   "Michael Candace Jordan",   "His Airness"   ]}  
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.71it/s]
 {"name": "Michael Jordan", "age": 57, "nameList": [ "Michael", "Jordan" ], "isActive": false, "comment": "hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.45it/s]
    {   "name": "Michael Jordan",   "age": 56,  "nameList": [   "Michael Jordan"    ],  "isActive": true,   "comment": "hello"  }   
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.23it/s]
 {"name":"Jordan Michael Craig","age":50,"nameList":[ "Michael Craig","Arthur George","Michael Barry","Michael Dwayne","William Jordan"],"isActive":true,"comment": "hello"} 
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.27it/s]
{ "name": "Michael Jordan", "age": 56, "nameList": ["basketball player", "investor", "legend", "speaker"], "isActive": true, "comment": "hello" } 
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.76it/s]
["name","age","bio"], "isActive": false ,"comment":"hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.53it/s]
    { "name": "Michael Jordan", "age": 52, "nameList": [ "Michael Jordan", "NBA record" ]} 
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.18it/s]
    {   "name":"Michael Jordan", "age": 58, "nameList":["Michael Dell", "Michael J. Fox", "Michael Douglas", "Michael Flander", "Michael G Colombini", "Michael T. Hetherington", "Michael Kazemi", "Michael Lake", "Michael Maika", "Michael McDougal", "Michael Mifsud", "Michael Oxley", "Michael Purdigal", "Michael Sabel"]    }   
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 16.86it/s]
}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.86it/s]
 { "name" :"Michael Jordan ","age": 59 ,"nameList" : ["football"] ,"isActive": false,"comment":"hello"} 
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.26it/s]
    {   "name": "Michael Jordan",   "nameList": [   "Michael",  "Jordan"    ],  "age": 59,  "isActive": false,  "comment": "hello" }    
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.59it/s]
    {   "name" : "Michael Jordan",  "age" : 58, "nameList" : [  "Basketball player",    "Skilled player",   "Professional basketball player",   " Small forward",   "Shooting guard"],  "isActive" : false, "comment" : "hello" }   
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 16.91it/s]
}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.37it/s]
{ "name": "Michael Jordan", "nameList": [ "Mickey", "Mike", "Jord" ], "age": 57.29, "isActive": false, "comment": "hello" }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.37it/s]
 { "name": "Michael Jordan" ,"nameList": ["jimmy", "Michael Jordan", "Michael", "Jordan"], }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.33it/s]
{ "name": "Michael Jordan", "age": 59, "nameList": ["Michael Jordan", "Michael Scott", "Michael Augustus Westphal"], "isActive": true, "comment": "hello" }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  1.89it/s]
{ "name": "Michael Jordan", "age": 59, "nameList": [ "Basketball player", "Soccer player", "Baseball player", "Ice hockey player", "Hockey goaltender" ], "isActive": false, "comment": "hello" }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  3.23it/s]
 {"name":"Michael Jordan","age":59,"nameList":["Basketball player"],"isActive":false,"comment": "hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  4.33it/s]
"comment": "hello" }dan",
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.81it/s]
 {"name": "Michael Jordan", "nameList": ["Michael", "Jordan"], "age": 57, "isActive": false, "comment": "hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.91it/s]
"age": 57,"nameList": ["Michael K. Jordan", "Michael Jordan"], "comment": "hello"}
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 16.89it/s]
 { }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00, 16.90it/s]
 { }
Processed prompts: 100%|██████████| 1/1 [00:00<00:00,  2.48it/s]
 { "name" :"Michael Jordan", "nameList" : [ "],  "] ,"age" : 57,"isActive" : false ,"comment" :"hello" }
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.42s/it]
 { "name": "Michael Jordan", "age": 58, "nameList": [ " NBA Basketball", " NBA MVP", " NBA Hall of Fame", " Chicago Bulls", " 15 NBA Finals", " 6 NBA Titles", " 5 MVP Seasons", " 14 All Star Games", " 10 consecutive All Star appearances", " 14 All Defensive Selections", " 14 First Team All Defense", " 14 All Star Weekend Participations", " 154 Points A Game", " 12235 Career points", " 5993 Rebounds", " 11035 Assists", "  Clearwater Aquarium", "  University of North Carolina", " 1984 NBA Draft", " 1990 NBA All Rookie" ], "isActive": false, "comment": "hello" }

I can make a PR to fix this, but I've looked at the code and I might need a little guidance from the author on how JSONSchemaParser works.

[noamgat]

Hi! Can you please highlight exactly where the problem is? There are many examples here and its hard to understand exactly.

[kmccleary3301]

Hi! Can you please highlight exactly where the problem is? There are many examples here and its hard to understand exactly.

The issue is that the model is allowed to output empty JSONs, such as { }, or just white space, while a scheme is specified. This defeats the purpose of schema enforcement, which is supposed to enforce a JSON scheme at the sampler level. I am willing to help fix it, but I'm a bit confused by the JSONSchemaParser code.

[noamgat]

Can you check if it reproduces without the line comment: Optional[Union[Literal["hello", "how are \|\|\"\"you?"]]] = None I don't think we support the "Literal" field yet, and it might be causing an exception which causes the ForceStopParser to activate, making the sequence end.

[kmccleary3301]

Can you check if it reproduces without the line comment: Optional[Union[Literal["hello", "how are \|\|\"\"you?"]]] = None I don't think we support the "Literal" field yet, and it might be causing an exception which causes the ForceStopParser to activate, making the sequence end.

I'll check and post the results manually later today, but I've tried the literal field and it does work at forcing the output to match when it doesn't output empty JSON, if that's any help.