JayYip / m3tl

BERT for Multitask Learning
https://jayyip.github.io/m3tl/
Apache License 2.0
545 stars 125 forks source link

疑惑:没有看到哪里执行了 text 2 tokens 的过程 #119

Open EdwardChan5000 opened 1 year ago

EdwardChan5000 commented 1 year ago
  1. 看到 _create_multimodal_bert_features 是被 m3tl_preproc_fn.py 通过 @preprocessing_fn 调用 decorator
  2. 应该是 input_fn.py 中 write_tfrecord 调用 chain_processed_data 的时候触发?
  3. 还是哪里?可以 q 一下相应调用
JayYip commented 1 year ago

这里: https://github.com/JayYip/m3tl/blob/63c44c37d4b2d695d5074dd76e28a0273d00064f/m3tl/bert_preprocessing/create_bert_features.py#LL290C28-L290C28