Batched generation does not work

maximilianmordig commented 2 months ago

Describe the issue as clearly as possible:

Perform batched generation. The problem occurs because CFGGuide.copy() which does not copy the attribute regex_fsm. https://github.com/outlines-dev/outlines/blob/25b6bcd498d0f335683f0026e9baeb591061d001/outlines/fsm/guide.py#L432

Steps/code to reproduce the bug:

from outlines import models, generate

grammar = """?start: "a".."z" """
model = models.transformers("gpt2")
generator = generate.cfg(model, grammar)
sequence = generator("hello")
sequence = generator(["hello", "world"])
print(sequence)

Expected result:

Batched generations

Error message:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File /somepath/scratch.py:2
      1 #%%
----> 2 sequence = generator(["hello", "world"])
      3 print(sequence)

File /someenv//lib/python3.10/site-packages/outlines/generate/api.py:207, in SequenceGenerator.__call__(self, prompts, max_tokens, stop_at, rng)
    205 while True:
    206     try:
--> 207         last_state = next(states)
    208         if max_tokens or stop_sequences:
    209             token_ids = last_state.token_ids

File /someenv//lib/python3.10/site-packages/outlines/generate/generator.py:93, in sequence_generator(model, sampler, fsms, token_ids, sequence_weights, attention_masks, fsm_states, rng)
     90     fsms = reorder_fsms(fsms, ancestors)
     91     fsm_states = reorder_fsm_states(fsm_states, ancestors)
---> 93 fsm_states = get_next_fsm_states(fsms, fsm_states, next_token_ids)
     94 is_finished = is_generation_finished(fsms, fsm_states)
     96 if is_finished:

File /someenv//lib/python3.10/site-packages/outlines/generate/generator.py:132, in get_next_fsm_states(fsms, fsm_states, next_token_ids)
    115 def get_next_fsm_states(
    116     fsms: List["Guide"], fsm_states: List[int], next_token_ids: "torch.Tensor"
    117 ) -> List[int]:
    118     """
    119 
    120     Parameters
   (...)
    130 
    131     """
--> 132     return [
    133         fsm.get_next_state(fsm_state, int(token_id[0]))
    134         for fsm, fsm_state, token_id in zip(fsms, fsm_states, next_token_ids)
    135     ]

File /someenv//lib/python3.10/site-packages/outlines/generate/generator.py:133, in <listcomp>(.0)
    115 def get_next_fsm_states(
    116     fsms: List["Guide"], fsm_states: List[int], next_token_ids: "torch.Tensor"
    117 ) -> List[int]:
    118     """
    119 
    120     Parameters
   (...)
    130 
    131     """
    132     return [
--> 133         fsm.get_next_state(fsm_state, int(token_id[0]))
    134         for fsm, fsm_state, token_id in zip(fsms, fsm_states, next_token_ids)
    135     ]

File /someenv//lib/python3.10/site-packages/outlines/fsm/guide.py:419, in CFGGuide.get_next_state(self, state, token_id)
    416     self.reset_state = False
    417     state = self.start_state
--> 419 return self.regex_fsm.get_next_state(state, token_id)

AttributeError: 'CFGGuide' object has no attribute 'regex_fsm'

Outlines/Python version information:

0.0.46 Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] accelerate==0.31.0 aiohttp==3.9.5 aiosignal==1.3.1 annotated-types==0.7.0 asttokens==2.4.1 async-timeout==4.0.3 attrs==23.2.0 bitsandbytes==0.43.1 certifi==2024.6.2 cfgv==3.4.0 charset-normalizer==3.3.2 click==8.1.7 cloudpickle==3.0.0 comm==0.2.2 contourpy==1.2.1 cycler==0.12.1 datasets==2.20.0 debugpy==1.8.2 decorator==5.1.1 dill==0.3.8 diskcache==5.6.3 distlib==0.3.8 docker-pycreds==0.4.0 docstring_parser==0.16 einops==0.8.0 exceptiongroup==1.2.1 executing==2.0.1 filelock==3.15.4 flake8==7.1.0 flash-attn==2.5.9.post1 fonttools==4.53.0 frozenlist==1.4.1 fsspec==2024.5.0 gitdb==4.0.11 GitPython==3.1.43 huggingface-hub==0.23.4 identify==2.5.36 idna==3.7 iniconfig==2.0.0 interegular==0.3.3 ipykernel==6.29.5 ipython==8.26.0 jedi==0.19.1 Jinja2==3.1.4 jsonschema==4.22.0 jsonschema-specifications==2023.12.1 jupyter_client==8.6.2 jupyter_core==5.7.2 kiwisolver==1.4.5 lark==1.1.9 llvmlite==0.43.0 markdown-it-py==3.0.0 MarkupSafe==2.1.5 matplotlib==3.9.0 matplotlib-inline==0.1.7 mccabe==0.7.0 mdurl==0.1.2 mpmath==1.3.0 multidict==6.0.5 multiprocess==0.70.16 nest-asyncio==1.6.0 networkx==3.3 nodeenv==1.9.1 numba==0.60.0 numpy==1.26.4 nvidia-cublas-cu12==12.1.3.1 nvidia-cuda-cupti-cu12==12.1.105 nvidia-cuda-nvrtc-cu12==12.1.105 nvidia-cuda-runtime-cu12==12.1.105 nvidia-cudnn-cu12==8.9.2.26 nvidia-cufft-cu12==11.0.2.54 nvidia-curand-cu12==10.3.2.106 nvidia-cusolver-cu12==11.4.5.107 nvidia-cusparse-cu12==12.1.0.106 nvidia-nccl-cu12==2.20.5 nvidia-nvjitlink-cu12==12.5.82 nvidia-nvtx-cu12==12.1.105 outlines==0.0.46 packaging==24.1 pandas==2.2.2 parso==0.8.4 peft==0.11.1 pexpect==4.9.0 pillow==10.4.0 platformdirs==4.2.2 pluggy==1.5.0 pre-commit==3.7.1 prompt_toolkit==3.0.47 protobuf==5.27.2 psutil==6.0.0 ptyprocess==0.7.0 pure-eval==0.2.2 pyairports==2.1.1 pyarrow==16.1.0 pyarrow-hotfix==0.6 pycodestyle==2.12.0 pycountry==24.6.1 pydantic==2.8.2 pydantic_core==2.20.1 pyflakes==3.2.0 Pygments==2.18.0 pyparsing==3.1.2 pytest==8.2.2 python-dateutil==2.9.0.post0 pytz==2024.1 PyYAML==6.0.1 pyzmq==26.0.3 referencing==0.35.1 regex==2024.5.15 requests==2.32.3 rich==13.7.1 rpds-py==0.18.1 safetensors==0.4.3 sentry-sdk==2.7.1 setproctitle==1.3.3 shtab==1.7.1 six==1.16.0 smmap==5.0.1 stack-data==0.6.3 sympy==1.12.1 tokenizers==0.19.1 tomli==2.0.1 torch==2.3.1 torchaudio==2.3.1 torchvision==0.18.1 tornado==6.4.1 tqdm==4.66.4 traitlets==5.14.3 transformers==4.42.3 triton==2.3.1 trl==0.9.4 typing_extensions==4.12.2 tyro==0.8.5 tzdata==2024.1 urllib3==2.2.2 virtualenv==20.26.3 wandb==0.17.3 wcwidth==0.2.13 xxhash==3.4.1 yarl==1.9.4

Context for the issue:

Cannot use batching with cfg

maximilianmordig commented 2 months ago

Temporary fix:

from outlines.fsm.guide import CFGGuide
def new_copy(self):
    """Create a copy of the FSM."""
    res = CFGGuide(self.cfg_string, self.tokenizer)
    if hasattr(self, "regex_fsm"):
        res.regex_fsm = self.regex_fsm.copy()
    return res
CFGGuide.copy = new_copy

However, it does not seem to work since the output is much shorter when generating the batch.

lapp0 commented 1 week ago

@maximilianmordig can you please try again main since the following PR has merged?

https://github.com/dottxt-ai/outlines/pull/1067

dottxt-ai / outlines