Using self trained model

[ShivamSharma1997]

Hi, I have a GPT-2 model trained on a different dataset and was trying to run the model for generation using GeDi. The method I followed is as follows

I changed the code in run_generation.bash to follows

python ../generate_GeDi.py \ --gen_length 200 \ --model_type gpt2 \ --gen_model_name_or_path 'path-to-my-model' \ --disc_weight 30 \ --rep_penalty_scale 10 \ --filter_p 0.8 \ --target_p 0.8 \ --gen_type "cclm" \ --repetition_penalty 1.2 \ --mode "topic" \ --penalize_cond

when I run the bash script, I get AssertionError on the tokenizer line in generate_GeDi.py file.

Please guide on where I am going wrong.

[yugaljain1999]

@ShivamSharma1997 Can you share code where you are getting error? I have faced similar issue when trying GeDi on custom data, I can solve your error if you share code snippet or colab notebook.

[akhileshgotmare]

@ShivamSharma1997 It seems you want to generate with GeDi guiding your GPT-2 (fine-tuned on a different dataset). If this is correct, you should change the --gen_type argument to gedi. With cclm the generation will be directly from the class conditional language model (GeDi topic in this case). This way the original topic GeDi will be guiding your GPT-2 model. I am not sure what purpose this will serve, but could be interesting depending on what data you fine-tune the GPT-2 model on (I am assuming this is standard LM finetuning and NOT GeDi training).

Re the assertion error: Can you paste the error message here? I can find only two assert statements in generate_GeDi.py, none of which involve tokenizers.

[ShivamSharma1997]

@akhileshgotmare Thanks, I had figured it out, there was a difference in the versions of transformers I think. That was the reason for assertion error.

Can you also guide me on how to self train the GeDi on a custom dataset. My understanding is I have to add the topics to the text and mark half as 0 and half as 1 labels randomly and pass the dataset through the run_training.sh bash script.

Just wanted to confirm whether that is the way you processed the data?

[akhileshgotmare]

Ben's comment here describes the process to train GeDi on custom datasets - https://github.com/salesforce/GeDi/issues/1#issuecomment-699655593

[akhileshgotmare]

Closing for now, feel free to re-open or comment on this thread for more clarifications.