node-red-node-multilang-sentiment: allow tokenization

[chuank]

Which node are you reporting an issue on?

node-red-node-multilang-sentiment

What are the steps to reproduce?

The current build only accepts an optional words object to override sentiment scores. It does not support the provision of tokenizers using a tokens object that the underlying multilang-sentiment Node.js package allows.

(am also happy to submit a PR for this; please let me know)

What happens?

Sentiment analysis of languages that require tokenization fails.

What do you expect to happen?

tokens should be passable to multilang-sentiment

Please tell us about your environment:

• [x] Node-RED version: 3.0.2
• [x] node.js version: 18.17.1
• [x] npm version: 8.19.4
• [x] Platform/OS: Ubuntu 22.04.3 LTS
• [x] Browser: Chrome

[dceejay]

Hi yes a PR would be much appreciated. Thanks