Closed shishirb126 closed 3 years ago
This PR is currently blocked by an unrelated CI issue. I opened https://github.com/dmlc/gluon-nlp/pull/1383/ to attempt fixing it.
Job PR-1382/2 is complete. Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1382/2/index.html
Job PR-1382/3 is complete. Docs are uploaded to http://gluon-nlp-staging.s3-accelerate.dualstack.amazonaws.com/PR-1382/3/index.html
Description
The v0.x vocab implementation uses a custom dict implementation to handle case where unknown tokens are specified. When looking up multiple tokens, it is faster to work with a Python dict directly. A similar change has already been made in the master branch. This change pulls it to the v0.x branch.
Results from a micro benchmark below. Current implementation:
With the patch:
Checklist
Essentials
Changes
Comments
cc @dmlc/gluon-nlp-team