The PILE dataset is full of racist content and thus GPT-J produces racist thinking.

[azeemh]

Any way to train on a different dataset?

[Joseffb]

I too have a question on how to train the dataset, specifically can we set GPT-J to learn from it's interactions?

[havietisov]

Model card mentions this : https://huggingface.co/EleutherAI/gpt-j-6b#out-of-scope-use