data preprocess suggestion

[kargaranamir]

Hi Laurie,

The code was super easy to use, thanks for that.

just some suggesstions.

I had troubles with this line of code: reformat-itu-tureng.py#L38

The error is: f-string expression part cannot include a backslash I solve it this way (not very clean 😄):

with open(args.out_file, 'w') as f:
    backslash_char = "\t"
    for s, t in zip(sents, tags):
        f.write(f"{s}{backslash_char}{'🍔'.join(t)}\n".replace("🍔", backslash_char))

• For turkish dataset there is no need to refer to the form ... you can access it here (TR-EN CS Corpus - with language tags.txt):

  https://github.com/zeynepyirmibes/code-switching-tr-en

• For indonasian also there is need to refer to email you can access it here (825_Indonesian_English_CodeMixed.csv)

  https://github.com/ir-nlp-csui/id-en-code-mixed

[laurieburchell]

Hi Amir, Thanks for pointing out the bug! I put the join on a separate line, problem solved. I'll keep the emails in the script for now since this is how the authors ask people to obtain the data in the original papers.