Open xinghuang2050 opened 1 year ago
To solve the unicode problem in output jsonl file, the following update in write_output_to_jsonl.py is recommended:
Line 27: temp_file.write(json.dumps(one_json) + '\n') modified to: temp_file.write(json.dumps(one_json, ensure_ascii=False)+'\n')
temp_file.write(json.dumps(one_json) + '\n')
temp_file.write(json.dumps(one_json, ensure_ascii=False)+'\n')
ok，请提个pr吧
aplmikex
commented
1 year ago aplmikex
commented
1 year ago
To solve the unicode problem in output jsonl file, the following update in write_output_to_jsonl.py is recommended:
Line 27:
temp_file.write(json.dumps(one_json) + '\n')modified to:temp_file.write(json.dumps(one_json, ensure_ascii=False)+'\n')