How to get the embedding for each word in the sentance?

[ghost]

Hi,

I am struggling to get the embedding for individual words. I used this command:

python -m elmoformanylangs test --input_format conll --input input.conllu --model ar.model --output_prefix ./output/ --output_format hdf5 --output_layer -1

And it dumbs hdf5 encoded onto the disk, as said. However, as far as I understand, the file encoded a dict where the key is tab speerated sentence, and the value is its representation.

But when I print the key:

f = h5py.File(filename, 'r')

for key in list(f.keys()):
    print(key)

I can see that f.keys() contain only a one string key of all sentences in the input file. 1) Why? And how to get individual sentence representation? 2) How to get individual word representation?

This is example of my input with 2 sentences :

1   ik  ik  PRON    VNW|pers|pron|nomin|vol|1|ev    Case=Nom|Person=1|PronType=Prs  2   nsubj   2:nsubj _
2   zie zien    VERB    WW|pv|tgw|ev    Number=Sing|Tense=Pres|VerbForm=Fin 0   root    0:root  _
3   hem hem PRON    VNW|pers|pron|obl|vol|3|ev|masc Case=Acc|Person=3|PronType=Prs  2   obj 2:obj|4:nsubj:xsubj _
4   fietsen fietsen VERB    WW|inf|vrij|zonder  VerbForm=Inf    2   xcomp   2:xcomp _
1   Jan Jan PROPN   N|eigen|ev|basis|zijd|stan  Gender=Com|Number=Sing  2   nsubj   2:nsubj _
2   komt    komen   VERB    WW|pv|tgw|met-t Number=Sing|Tense=Pres|VerbForm=Fin 0   root    0:root  _
3   vandaag vandaag ADV BW  _   2   advmod  2:advmod    _
4   en  en  CCONJ   VG|neven    _   5   cc  5.1:cc  _
5   Piet    Piet    PROPN   N|eigen|ev|basis|zijd|stan  Gender=Com|Number=Sing  2   conj    5.1:nsubj   _ 

[Oneplus]

Hi, the value of f[key] is a numpy array of (seq_len, dim) (If you use the recently patch and output all the layers, it will be (n_layer, seq_len, dim)). You can get embeddings for each word by numpy.split along the seq_len dimension.

[ghost]

So the sentence embedding is not averaged, I understand now. However, in f[Key] , the key should be the sentence itself, right?

Another problem that I mentioned in my issue is regarding the input format, I suspect that I am doing something wrong because when I print the length of f.keys(), it returns 1 even that my input contains more than one sentence. So this loop is executed only once and treat all my sentences as a single one.

for key in list(f.keys()):
    print(key)

Am I doing something wrong?

The

[Oneplus]

So the sentence embedding is not averaged, I understand now. However, in f[Key] , the key should be the sentence itself, right?

Yes, the key should be the sentence itself.

Am I doing something wrong?

Please check if your input file follows the conll format (https://github.com/HIT-SCIR/ELMoForManyLangs#use-elmoformanylangs-in-command-line) and specify the input format as conll