Open cbjrobertson opened 1 year ago
I'm having trouble reproducing the exact error, but I suspect it's related to the 512 token limit of T5, which is shared with the input text and the command to T5. In general, it's not a good idea to put more than a single sentence at a time into the sentence
param, since during training the model is only ever trained on single sentences. I don't know how it will perform if it's given multiple sentences like that since it's out of domain of the training regimen. This is also a problem with the documentation, since I don't think this is highlighted properly
Your point about distribution shift aside, I don't think it can be text length.
Firstly because:
len(text_2.split())
>>> 76
This is way under the T5 input limit.
Secondly because:
text_3 = "Well, I came out and put on these and run around. And uh I like to run. I see the car. And I don't know, I uh. Yeah they're over there is going to put up and have a party uh. It's a child getting water. And she's she's going to making something nice. That's my father. He goes everywhere on that one. And that's people a lot of people like to go out on those and other stuff like that."
fst_small.detect_frames(text_3)
>>> DetectFramesResult(sentence="Well, I came out and put on these and run around. And uh I like to run. I see the car. And I don't know, I uh. Yeah they're over there is going to put up and have a party uh. It's a child getting water. And she's she's going to making something nice. That's my father. He goes everywhere on that one. And that's people a lot of people like to go out on those and other stuff like that", trigger_locations=[8, 38, 59, 67, 99, 182, 261, 335, 363], frames=[FrameResult(name='Arriving', trigger_location=8, frame_elements=[FrameElementResult(name='Theme', text='I'), FrameElementResult(name='Goal', text='out')]), FrameResult(name='Self_motion', trigger_location=38, frame_elements=[FrameElementResult(name='Self_mover', text='I'), FrameElementResult(name='Goal', text='around')]), FrameResult(name='Likelihood', trigger_location=59, frame_elements=[FrameElementResult(name='Hypothetical_event', text='I'), FrameElementResult(name='Hypothetical_event', text='to run')]), FrameResult(name='Self_motion', trigger_location=67, frame_elements=[FrameElementResult(name='Self_mover', text='I')]), FrameResult(name='Awareness', trigger_location=99, frame_elements=[FrameElementResult(name='Cognizer', text='I'), FrameElementResult(name='Content', text='I')]), FrameResult(name='People_by_age', trigger_location=182, frame_elements=[FrameElementResult(name='Person', text='child')]), FrameResult(name='Kinship', trigger_location=261, frame_elements=[FrameElementResult(name='Ego', text='my'), FrameElementResult(name='Alter', text='father')]), FrameResult(name='Likelihood', trigger_location=335, frame_elements=[FrameElementResult(name='Hypothetical_event', text='people a lot of people'), FrameElementResult(name='Hypothetical_event', text='to go out on those and other stuff like that')]), FrameResult(name='Increment', trigger_location=363, frame_elements=[FrameElementResult(name='Class', text='stuff')])])
In other words, text_3
does not return an error, even though it is longer than text_2
(i.e. it's the same but "and other stuff like that."
is added to the end).
It may be helpful to know that I get the same error in the same line of code when I pass an empty string to detect_frames:
fst = FrameSemanticTransformer("small")
fst.detect_frames("")
>>>---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
/tmp/ipykernel_2486399/937327281.py in <module>
1 fst = FrameSemanticTransformer("small")
----> 2 fst.detect_frames("")
3
~/anaconda3/envs/ktrain_loo/lib/python3.7/site-packages/frame_semantic_transformer/FrameSemanticTransformer.py in detect_frames(self, sentence)
167 base_sentence, trigger_locs = self._identify_triggers(sentence)
168 # next detect frames for each trigger
--> 169 frames = self._classify_frames(base_sentence, trigger_locs)
170
171 frame_and_locs = [
~/anaconda3/envs/ktrain_loo/lib/python3.7/site-packages/frame_semantic_transformer/FrameSemanticTransformer.py in _classify_frames(self, sentence, trigger_locs)
126 frame_classification_tasks, chunk_size=self.max_batch_size
127 ):
--> 128 batch_results = self._batch_predict([task.get_input() for task in batch])
129 for preds, frame_task in zip(
130 chunk_list(batch_results, self.predictions_per_sample),
~/anaconda3/envs/ktrain_loo/lib/python3.7/site-packages/frame_semantic_transformer/FrameSemanticTransformer.py in <listcomp>(.0)
126 frame_classification_tasks, chunk_size=self.max_batch_size
127 ):
--> 128 batch_results = self._batch_predict([task.get_input() for task in batch])
129 for preds, frame_task in zip(
130 chunk_list(batch_results, self.predictions_per_sample),
~/anaconda3/envs/ktrain_loo/lib/python3.7/site-packages/frame_semantic_transformer/data/tasks/FrameClassificationTask.py in get_input(self)
25
26 def get_input(self) -> str:
---> 27 potential_frames = get_possible_frames_for_trigger_bigrams(self.trigger_bigrams)
28 return f"FRAME {' '.join(potential_frames)} : {self.trigger_labeled_text}"
29
~/anaconda3/envs/ktrain_loo/lib/python3.7/site-packages/frame_semantic_transformer/data/tasks/FrameClassificationTask.py in trigger_bigrams(self)
44 pre_trigger_tokens = self.text[: self.trigger_loc].split()
45 trigger_and_after_tokens = self.text[self.trigger_loc :].split()
---> 46 trigger = trigger_and_after_tokens[0]
47 post_trigger_tokens = trigger_and_after_tokens[1:]
48 bigrams: list[list[str]] = []
IndexError: list index out of range
It works as expected with the "base"
model:
fst = FrameSemanticTransformer("base")
fst.detect_frames("")
>>> DetectFramesResult(sentence='TRIGGER:', trigger_locations=[0], frames=[FrameResult(name='Triggering', trigger_location=0, frame_elements=[FrameElementResult(name='', text='N/A')])])
That is strange that it's happening on an empty string as well. I still can't reproduce this error for some reason. It seems like the small model must be outputting some strange string that's breaking processing of subtasks, I wish I could see what string it is. I'm not sure if it's related, but the library only supports Python 3.8+ currently, so I'm surprised it was possible to install it on Python 3.7.
For the length of the input, the reason it could spill over the 512 token limit is that the subtask prompts can also be pretty long. The library tries to include any info it thinks might be helpful for T5 to make the classification as part of the task definition. So, for example, one of the subtask inputs for text_2 looks like the following when I try it locally:
ARGS Perception_experience | Perceiver_passive Phenomenon Body_part Location_of_protagonist Direction Depictive State Manner Degree Means Time Duration Contrastive_context Concessive Circumstances Frequency Place Ground Obscuring_medium : Well, I came out and put on these and run around. And uh I like to run. I * see the car. And I don't know, I uh. Yeah they're over there is going to put up and have a party uh. It's a child getting water. And she's she's going to making something nice. That's my father. He goes everywhere on that one. And that's people a lot of people like to go out on those.
Here, it's trying to include all the possible argument names that the Perception_experience
frame might have as part of the text it sends to T5 to make the task easier for T5. The length of this prompt prefix depends on the frame that's predicted, so it's possible that it might spill over the 512 limit seemingly randomly depending on which frame it's predicting if the input text is long enough.
That being said, if this is happening on an empty string input then something is definitely wrong. Are you able to see what strings are being passed to the model and what it's returning? It might require hacking print statements into the library for debugging. Or maybe we can add a debug
param to the library to make this sort of thing easier 🤔
Actually, it's possible that what's going on is that you're on an old version of the library, since old versions did support Python 3.7. Can you confirm what version of the library you have installed?
For some texts, using t5-small, detect_frames returns a cryptic index error.
MRE:
Seems like a bug to me!