search

rapidsai / crossfit

Metric calculation library
Apache License 2.0
2 stars 6 forks source link

[BUG] clipping logic fails when padding token is not 0. #59

Closed uahmed93 closed 3 months ago

uahmed93 commented 3 months ago

in crossfit/backend/torch/loader.py when padded tokens are other than 0 , seq_length will always be max value. 

 seq_length = (frame[sort_key] != 0).sum(axis=1)

because of which clipping logic is failing.

VibhuJawa commented 3 months ago

Fixed by: https://github.com/rapidsai/crossfit/pull/66