Closed zfallahnejad closed 5 years ago
The issue is caused here: https://github.com/zfallahnejad/sentiment-discovery/blob/master/data_utils/loaders.py#L65
The tokenization isn't a list of text tokens. It's actually a list of IDs. So your normalization_map logic is failing every single check and never appending. This causes the lstm to process a [B,0] shaped tensor and error out.
If you would like to process your text with a normalization map I would recommend supplying a preprocess function to the csv dataset https://github.com/zfallahnejad/sentiment-discovery/blob/master/data_utils/datasets.py#L187.
You can do this by creating def my_preprocess_fn
in https://github.com/zfallahnejad/sentiment-discovery/blob/master/data_utils/preprocess.py and, importing it in https://github.com/zfallahnejad/sentiment-discovery/blob/master/data_utils/__init__.py . You can then specify this in all training/transfer code from the command line by using --preprocess_fn my_preprocess_fn
.
Thank you very much. I change process_str
function and I will use it for text normalization. I think I had another mistake which was related to the conversion between character and id. Your conversion cannot apply for my language. So, I define a dictionary which converts each character to an id. How can I use this dictionary? Should I change batch_tokens
function or CharacterLevelTokenizer
class?
batch_tokens just converts a batch of token lists to a padded tensor.
I think for your use case I would modify the tokenizer class itself as you've suggested.
Thanks.
I forked your project and changed it in order to test it for my language. I faced the following error during the run of transfer:
Do you have any solution for this problem? I would appreciate it if you could help me figure this out.
This is the forked project and This is a colab notebook which contains my test.