Closed Gare-Ng closed 2 years ago
Hi @Gare-Ng ,
I organize the data as I'm told in readme. Although in readme it says that it should be jsonl file, I find that jsonl file won't work and just rename it as json works
The file format is expected to be jsonl
but the scripts expect the file extension to be .json
.
I think the reason for this error is that, there seems to be an example which has an empty sting as the translation/target. The example scripts are kept simple and easy to adapt so they don't do any extra pre-processing to detect and remove empty examples. You should inspect the dataset to remove such problematic examples and run the scripts.
Thank you @patil-suraj, and I do write a small script to check whether there is such empty things that I guess it is string there.
which has an empty sting as the translation/target.
But it turns out that there isn't such problematic examples in my test dev and test file. I train this model without modifying json file today morning and it works every well. Vaguely I remenber it pops this errors when I decide to change a model and train. At first I think it is related to the new model, but errors remains when change model back. Till now I still get this error. I'm certain I didn't change json file. My little script is here just in case.
import jsonlines
jsonl_name=r'C:\Users\Gare\PycharmProjects\Gare\transformers\examples\pytorch\translation\test.json'
f=jsonlines.open(jsonl_name, "r")
lines=1
errors=[]
for i in f:
if len((i['translation']['Mdn']))==0 or len((i['translation']['Cls']))==0:
errors.append(lines)
lines += 1
if len(errors)==0:
print('No Error')
else:
print(errors)
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Environment info
transformers
version: 4.18.0.dev0Who can help
@patrickvonplaten , @patil-suraj
Information
Model I am using (Bert, XLNet ...): Marian
The problem arises when using:
The tasks I am working on is:
And here is a example of my dataset
I organize the data as I'm told in readme. Although in readme it says that it should be jsonl file, I find that jsonl file won't work and just rename it as json works. It works well for several thousands steps.
To reproduce
Steps to reproduce the behavior:
python examples/pytorch/translation/run_translation.py --model_name_or_path "Helsinki-NLP/opus-mt-zh-en" --do_train --do_eval --source_lang Cls --target_lang Mdn --source_prefix "translate Classical to Modern: " --train_file examples\pytorch\translation\train.json --validation_file examples\pytorch\translation\dev.json --test_file examples\pytorch\translation\test.json --output_dir D:/Gare-translation/Helsinki-NLP --per_device_train_batch_size=32 --per_device_eval_batch_size=16 --overwrite_output_dir --predict_with_generate --num_train_epochs=200 --save_total_limit=200 --save_steps=10000 --load_best_model_at_end True --evaluation_strategy "steps"
Expected behavior
Any advice that helps with it would be appreciated.