Please add a tutorial

[StanSilas]

Hi, I'm quite new to this field. Yours was the only implementation of NER that I could find wrt Bi-LSTM. It would be very helpful if you could add tutorials for creating a minimally reproducible results. Specifically,

• How to train model using own data (i.e how to run code)
• How to load data
• How to test the model on data , ex : If I pass a sentence "I was eating an apple while sitting inside the apple headquarters" . It should output the entity tags apple: fruit , apple headquarters : Organisation, I : Person Thanks,

[dilipbobby]

yeah ! it will help us .please add tutorial

[0xnurl]

As mentioned in the Readme file, you just need to implement these methods:

• get_texts() - returns a list of strings representing the texts (x's) to train on, without labels.
• get_x_y() - returns a tuple (x, y) where: x: Array of shape (batch_size, sentence_maxlen). Entries in dimension 1 are alphabet indices, index 0 is the padding symbol y: Array of shape (batch_size, sentence_maxlen, self.num_labels). Entries in dimension 1 are label indices, index 0 is the null label.
• get_labels() - returns a list of all possible labels (entities) in your dataset.

After implementing these methods, just run train.py.

To test the model prediction, use the method predict_str().

[fatal-exception]

Many thanks for your work @0xnurl , I'm greatly enjoying trying your model.

I am struggling with one thing - construction of the y tensor. The first 2 dimensions are fine, but it confuses me that the 3rd dimension has to be the same length as self.num_labels. Is the third dimension of the y tensor a one-hot?

Thanks!

Matt

[0xnurl]

@fatal-exception Yes, it is one-hot, representing the target label.

[sonali856]

Hi , I am trying to implement your model , but having difficulties with implementing the dataset methods, It would be great if you could give a simple example of the implementation to these methods :D

[sonali856]

@fatal-exception
Can you explain me how you implemented dataset.py . I am trying to implement it for CoNLL 2003 english dataset but cannot understand the x, y tensors and dimensions. Can you please elaborate on how to use it and possibly show with an example?

Thanks, Sonali

[fatal-exception]

Sure, sorry not to reply sooner I am quite busy, but here is a start:

https://github.com/fatal-exception/project/blob/master/keras_character_based_ner/src/dataset.py

get_texts is implemented here https://github.com/fatal-exception/project/blob/master/keras_character_based_ner/src/matt/file_management.py#L155. It needs to return a list of strings. This is only needed for the keras model to build an alphabet from

get_x_y is implemented here: https://github.com/fatal-exception/project/blob/master/keras_character_based_ner/src/matt/model_integration.py#L145 Returns a Python tuple x and y, where x and y are Numpy arrays! x is a list-of-list-of-ints created here https://github.com/fatal-exception/project/blob/master/keras_character_based_ner/src/matt/model_integration.py#L30 y is a list-of-list-of-ints created here https://github.com/fatal-exception/project/blob/master/keras_character_based_ner/src/matt/model_integration.py#L87. In both cases, I start with list manipulation, and convert to an np.ndarray later

get_labels is a simple list of strings, defined here https://github.com/fatal-exception/project/blob/master/keras_character_based_ner/src/matt/model_integration.py#L21

Hope this is of some use to you, sorry not to have time to explain more

[fatal-exception]

get_x_y is the hardest, and this is the key:

        :return: Tuple (x, y)
                x: Array of shape (batch_size, sentence_maxlen).
                Entries in dimension 1 are alphabet indices, index 0 is the padding symbol
                y: Array of shape (batch_size, sentence_maxlen, self.num_labels).
                Entries in dimension 2 are label indices, index 0 is the null label

x is a tensor (as an np.ndarray) which has a dimension of batch_size (each element is an item of data in the batch) and a 2nd dimension of sentence_maxlen (each element is a number representing the letter in the alphabet system. The first element is a number representing the first letter-of-the-alphabet in that text item, and so on)

y is a tensor where the first 2 dimensions are the same as in x. But it has a 3rd dimension, which is a onehot dimension (all 0s except one dimension is a 1), where the 1 shows which label should be applied to that letter in that text item

[sonali856]

Hey, Thank you so much. That was a very nice explanation. I was having difficulty with the get_x_y() function only. I understood it with your help and was able to implement it successfully. Thanks:D Sonali

[sarwar187]

@fatal-exception I am not able to get your implementation. Those links are not available.

[fatal-exception]

my apologies @sarwar187 I have now updated

[sarwar187]

my apologies @sarwar187 I have now updated

Thanks a lot