search

keras-team / keras-nlp

Modular Natural Language Processing workflows with Keras
Apache License 2.0
734 stars 216 forks source link

Cannot reproduce results from notebook on Colab #1592

Open jespernwulff opened 2 months ago

jespernwulff commented 2 months ago

Describe the bug Running the KerasNLP guide in Colab, I get more or less 50% classification accuracy, i.e. no better than a random classifier. It get the same when fine-tuning the classifier. I am making no changes to any code in the notebook. Just running the code.

To Reproduce Link to the Colab Notebook, which reproduces the bug.

Expected behavior I expected to get the same accuracy results as reported in the text, i.e., around 78% accuracy for the first example.

SuryanarayanaY commented 2 months ago

Hi @jespernwulff ,

I have replicated the issue and attached gist here.

In the generated documentation guide the sparse_categorical_accuracy touced to 0.8443 in 2nd epoch but when trained now it is staying at 0.5 only even in epoch 3.

SamanehSaadat commented 2 months ago

Hi @jespernwulff!

Thanks for reporting this issue! I'll look into it!

SamanehSaadat commented 2 months ago

I think the issue is related to the dataset. I also got around 50% accuracy when I ran the fine-tuning with the IMDB dataset downloaded in the guide but when I switched to loading the IMDB dataset from tensorflow_datasets, I got 87% accuracy.

I'll update the guide to get the IMDB dataset from tensorflow_datasets for now.