search

asyml / texar-pytorch

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
https://asyml.io
Apache License 2.0
745 stars 117 forks source link

Add an extra function tokenize_with_span to WordpieceTokenizer #332

Closed jieralice13 closed 3 years ago

codecov[bot] commented 3 years ago

Codecov Report

Merging #332 (4f86bb0) into master (b95e810) will increase coverage by 0.01%. The diff coverage is 94.44%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #332      +/-   ##
==========================================
+ Coverage   80.17%   80.18%   +0.01%     
==========================================
  Files         136      136              
  Lines       11242    11248       +6     
==========================================
+ Hits         9013     9019       +6     
  Misses       2229     2229
Impacted Files Coverage Δ
...exar/torch/data/tokenizers/bert_tokenizer_utils.py 96.85% <92.85%> (+0.12%) :arrow_up:
texar/torch/data/tokenizers/bert_tokenizer.py 87.73% <100.00%> (ø)
texar/torch/data/tokenizers/tokenizer_base.py 89.95% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update b95e810...4f86bb0. Read the comment docs.

gpengzhi commented 3 years ago

Could you add the corresponding unit test for the added function here? BTW, the new function looks almost the same as the existing tokenize function. Is it possible to refactor the existing function to achieve your expected functionality?

gpengzhi commented 3 years ago

Thanks for the modifications! Could you also fix the CI error?

jieralice13 commented 3 years ago

Thanks for the modifications! Could you also fix the CI error?

Thanks, I've fixed the CI error.

hunterhector commented 3 years ago

There is a migration process from travis org to travis com, for the PRs at this moment, please check the build status manually https://travis-ci.org/github/asyml/texar-pytorch/pull_requests

gpengzhi commented 3 years ago

@jieralice13 Instead of waiting for the response from CI, you can actually run all CI tests (mypy, unittest, ...) locally. You could check the correctness of your implementation before pushing codes to GitHub. This might save you time when you contribute to the repo in the future.