Closed mayank31398 closed 2 years ago
@mayank31398, I think it'd be much better to file this issue with transformers, since this code isn't Mega-DS-related.
I know several folks have been tweaking the bloom modeling code a lot recently, so you may want to tag them on that (peek into the history of https://github.com/huggingface/transformers/blame/main/src/transformers/models/bloom/modeling_bloom.py)
Thanks, I will do that. Closing this issue here.
Filed this issue https://github.com/huggingface/transformers/issues/18809
FYI: I have re-tagged that new issue to those who have been actively tweaking the model, so they are the best to talk to.
@stas00 , I wrote this script to do get the conditional NLL for the labels given the context. Tried different batches with only the first example changing and rest of the examples fixed in the batch. However, after a certain point, the changing of first examples, affects the NLL for other examples.
This is not supposed to happen.
Output:
Value drops from 3.29 to 3.28 in column 2 when only example for column 0 is changed. Even column 3 changes in last case. Only column 0 is supposed to change here.