We need to update transformers to >= v4.36.0 due to security vulnerabilities, but
the latest detoxify v0.5.1 requires transformers v4.22.1. Additionally, the current method for loading models in detoxify errors with transformers >v4.30.0, see issue.
Description of changes:
To resolve the dependency conflict and model loading issue, this PR:
Loads the unbiased detoxify model directly from the state dict file to remove dependency on detoxify package. Model loading method is based on detoxify's load_checkpoint with modifications to address the issue above.
Upgrades transformers version to ^4.36.0 to resolve security vulnerabilities. (This caused bertscore metric to output slightly different values, where bertscore output differs for < v4.24.0 and >= v4.24.0, similar to in this issue).
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
Issue #, if available: https://tiny.amazon.com/f6f228ty/issuamazissuRAI7
We need to update transformers to >=
v4.36.0due to security vulnerabilities, but the latest detoxifyv0.5.1requires transformersv4.22.1. Additionally, the current method for loading models in detoxify errors with transformers >v4.30.0, see issue.Description of changes:
To resolve the dependency conflict and model loading issue, this PR:
load_checkpointwith modifications to address the issue above.^4.36.0to resolve security vulnerabilities. (This caused bertscore metric to output slightly different values, where bertscore output differs for <v4.24.0and >=v4.24.0, similar to in this issue).By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.