Open HSLUCKY opened 5 months ago
've identified the root cause. It's due to Outliens creating a massive cache object during FSM initialization. Despite our server utilizing 77G VIRT MEMORY, the process didn't terminate, ultimately leading to memory exhaustion and process termination.
After removing the "maxLength": 2048 parameter, apart from the initial slow loading, subsequent executions are able to locate the cache and infer results immediately. {"type": "string", "maxLength": 2048}
def init(self, regex_string: str, tokenizer): @cache() def create_states_mapping( regex_string: str, cacheable_vocabulary: Tuple[Tuple[str, int], ...] ) -> Tuple[dict, set]: """Create the variables related to the mapping between states and tokens The parameters of the function are used for caching purpose """ regex_pattern = interegular.parse_pattern(regex_string) regexfsm, = make_deterministic_fsm(regex_pattern.to_fsm().reduce()) states_to_token_maps, empty_token_ids = create_fsm_index_tokenizer( regex_fsm, tokenizer )
# We make sure that it is possible to generate strings in the language
# of the regular expression with the tokens present in the model's
# vocabulary.
if not any(
regex_fsm.finals.intersection(v.values())
for v in states_to_token_maps.values()
):
raise ValueError(
"The vocabulary does not allow us to build a sequence that matches the input regex"
)
return states_to_token_maps, empty_token_ids
self.states_to_token_maps, self.empty_token_ids = create_states_mapping(
regex_string, tuple(sorted(tokenizer.vocabulary.items()))
)
self.vocabulary = list(tokenizer.vocabulary.values())
self.eos_token_id = tokenizer.eos_token_id
The problematic code resides within the following lines in fsm.py. regexfsm, = make_deterministic_fsm(regex_pattern.to_fsm().reduce()) states_to_token_maps, empty_token_ids = create_fsm_index_tokenizer( regex_fsm, tokenizer )
Can you leverage the GPU to generate the cache for improved speed and address the bug causing the generation of excessively large cache indices?
Yes, maxLength
generates a DFA with many repeated states, which leads to large memory usage. We are considering ways to dramatically reduce the memory usage, but they require low-level work (replacing the DFA with something else), which takes some time.
I've been using SGLang library, which depends on this outlines library for regex output constraint. I've observed a problem where if regex is used, then throughput performance is 37x lower than if it's not used, and GPU utilization stays < 5% during most of the work as well.
I wanted to ask, is there a better performance in outlines if Context-Free Grammar was used instead of regex?
Link to my issue on SGLang: https://github.com/sgl-project/sglang/issues/450
Describe the issue as clearly as possible:
When utilizing the Outliens VLLM mode for JSON and regex inference, the GPU utilization remains at 0%, while the CPU utilization peaks at 100%. What could be the reason behind this phenomenon? How can we leverage the GPU for inference formatting? Furthermore, when inferring with JSON mode on problems of length 2048, CPU utilization reaches 100% while GPU utilization remains at 0%. Consequently, the process appears to hang indefinitely without termination, indicating a potential deadlock.![image](https://github.com/outlines-dev/outlines/assets/27761608/d89496f4-10ea-4171-aa08-c43cf25a2257)
We've integrated Outliens into FastChat.
Steps/code to reproduce the bug:
Expected result:
Error message:
No response
Outlines/Python version information:
Version information
Context for the issue:
No response