search

fhamborg / NewsMTSC

Target-dependent sentiment classification in news articles reporting on political events. Includes a high-quality data set of over 11k sentences and a state-of-the-art classification model.
Other
140 stars 21 forks source link

Introduce truncation parameter for long sentences #24

Closed miriamventurini closed 1 year ago

miriamventurini commented 1 year ago

I am working on historical newspapers, and sometimes the text data is not perfect. For this reason, I have sentences with too many tokens occasionally.

Would it be possible to introduce a truncation parameter for the first and third inputs of the function?

This would save me a lot of time checking exceptions (and truncating the sentences) one by one when it happens.

Many thanks for considering my request!

fhamborg commented 1 year ago

Hi! I just checked the code (again, and in contrast to what I said in my previous mail) the three inputs will be first concatenated and then truncated (if necessary). Thus, it is not conceptually senseful to automatically truncate the input sequences adequately, e.g., truncate the left part to a certain length and the right part to a certain length, since this would turn the entire input sequence (consisting of left, target, and right phrases) into something not reflecting a sentence. While the model may still be able to interpret the sentiment correctly, NewsMTSC and the python library NewsSentiment was not designed to be able to run on sequences that are not correct sentences.

In sum, since such sequence-parts-truncation would result in input sequences that the model was not designed for, we will not implement this feature. I recommend that you check out whether you can truncate the input yourself before using NewsSentiment.

miriamventurini commented 1 year ago

Hi, thanks for your answer!

You wrote "the inputs will be concatenated and then truncated if necessary". So, is there a truncation for long sentences or not?

Sorry, I understand what you wrote about truncating inputs, but you seemed to imply that the code does in fact truncate output if necessary.

Best,

Miriam

From: Felix Hamborg @.> Sent: Friday, March 17, 2023 3:00:54 PM To: fhamborg/NewsMTSC @.> Cc: Miriam Venturini @.>; Author @.> Subject: Re: [fhamborg/NewsMTSC] Introduce truncation parameter for long sentences (Issue #24)

Hi! I just checked the code (again, and in contrast to what I said in my previous mail) the three inputs will be first concatenated and then truncated (if necessary). Thus, it is not conceptually senseful to automatically truncate the input sequences adequately, e.g., truncate the left part to a certain length and the right part to a certain length, since this would turn the entire input sequence (consisting of left, target, and right phrases) into something not reflecting a sentence. While the model may still be able to interpret the sentiment correctly, NewsMTSC and the python library NewsSentiment was not designed to be able to run on sequences that are not correct sentences.

In sum, since such sequence-parts-truncation would result in input sequences that the model was not designed for, we will not implement this feature. I recommend that you check out whether you can truncate the input yourself before using NewsSentiment.

— Reply to this email directly, view it on GitHubhttps://github.com/fhamborg/NewsMTSC/issues/24#issuecomment-1473889209, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALK5227GR3N7DXVLQXDZPELW4RVBNANCNFSM6AAAAAAVP3RC5A. You are receiving this because you authored the thread.Message ID: @.***>

fhamborg commented 1 year ago

What error exactly are you getting? And can you provide an exemplary input causing such error?