Question: Is there a limit on the document size?

jackboyla / GLiREL

Generalist and Lightweight Model for Relation Extraction (Extract any relationship types from text)

90 stars 6 forks source link

Question: Is there a limit on the document size? #8

Closed nickknyc closed 3 months ago

nickknyc commented 3 months ago

I quickly tried Gliner and found that it only processed text that was about as long as a sentence. I may be missing something, but am wondering if there is any example of processing say a 30 page document. If I need to chunk the data, please let me know what the max chunk size is.

urchade commented 3 months ago

Hi @nickknyc, gliner can process more than 300+ words (paragraph length), but you can chunk to process longer text (https://github.com/urchade/GLiNER/discussions/113#discussioncomment-9898480)

jackboyla commented 3 months ago

hi @nickknyc, just to add that the max sequence length can be found in the config: https://github.com/jackboyla/GLiREL/blob/main/configs/config_few_rel.yaml#L45

thanks for catching this too @urchade 😃