After 10h training on 2 GPUs, the only logs I have are a bunch of WARNING:root:NaN or Inf found in input tensor..
From the tensorboard tfevents file, I can see that the valid loss is NaN...
Before trying to find out what's wrong, could you please confirm that it's actually conceptually feasible to fine-tune a BioBERT model using fast-bert ?
Hi.
I'd like to use fast-bert to fine-tune a BioBERT model on a NER corpus.
Here is the code I use to create a learner from a pretrained BioBERT model:
After 10h training on 2 GPUs, the only logs I have are a bunch of
WARNING:root:NaN or Inf found in input tensor.
. From the tensorboard tfevents file, I can see that the valid loss is NaN...Before trying to find out what's wrong, could you please confirm that it's actually conceptually feasible to fine-tune a BioBERT model using fast-bert ?