[X] I searched the issues and found no similar issues.
Component
Transforms/Other
Feature
Implement a new feature to detect and eliminate toxic words or sentences from a given text. This functionality should work on every row of a parquet file, where every row contains one document. The output should be True or False, and this should be added as an output column along with span of the text where toxic words are detected.
This can be added as a new transform for text/NLP data. One can refer to code quality module as a reference for how filters have been applied for code data.
Search before asking
Component
Transforms/Other
Feature
Implement a new feature to detect and eliminate toxic words or sentences from a given text. This functionality should work on every row of a parquet file, where every row contains one document. The output should be True or False, and this should be added as an output column along with span of the text where toxic words are detected.
This can be added as a new transform for text/NLP data. One can refer to code quality module as a reference for how filters have been applied for code data.
Are you willing to submit a PR?