yakt00 / IRGen

12 stars 1 forks source link

Question Regarding the Use of IDs in ISC Dataset Processing #4

Open Jinyunrising opened 4 months ago

Jinyunrising commented 4 months ago

Dear Author,

I hope this message finds you well.

I noticed that when processing the ISC dataset, you did not include IDs. However, in your training code, both the tokenizer training and the IRGen training use IDs. Could you please advise me on how to handle this issue?

Thank you very much for your assistance!