DocumentTokens: incorrect limit when stopList is used

sgsinclair / trombone

GNU General Public License v3.0

14 stars 6 forks source link

DocumentTokens: incorrect limit when stopList is used #22

Open ajmacdonald opened 3 years ago

ajmacdonald commented 3 years ago

Number of tokens returned will be likely less than the limit when a stopList is used. The tool uses int maxPos = documentStart+limit to determine when to stop searching, however this needs to be updated when one of the tokens in the stopList is encountered.