search

jonathanbratt / RBERT

Implementation of BERT in R
Apache License 2.0
158 stars 19 forks source link

include zeroth-level embeddings in extract_features #6

Closed jonathanbratt closed 5 years ago

jonathanbratt commented 5 years ago

For completeness, would be good to return the bare token embeddings before any transformer layers along with the layer outputs.

leungi commented 5 years ago

If I understand this post correctly, in most BERT-related articles, when 12 layers are mentioned for the uncased model, this corresponds to layer_output_1 to layer_output_12 in output from RBERT::extract_features(), ya?

jonathanbratt commented 5 years ago

That is correct. When you see layer_output_0 in RBERT output, that corresponds to the vectors input to the first transformer layer.