search

microsoft / unilm

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
https://aka.ms/GeneralAI
MIT License
19.62k stars 2.5k forks source link

should we remove "position embedding" #254

Open phucnsp opened 3 years ago

phucnsp commented 3 years ago

Describe Model I am using LayoutLM.

"Position embedding" used in layoutlm is borrowed from standard transformer architecture. However, in the case of layoutlm when bounding boxes' embeddings are introduced into, should we remove the position embedding?

I have removed it out and the performance even better. But I am not sure whether there is any hidden trade-off here.

elnazsn1988 commented 3 years ago

@phucnsp can you elaborate how you 'removed' the position embedding?

phucnsp commented 3 years ago

I just remove this line below (see. layoutlm/modeling/layoutlm.py) image

wolfshow commented 3 years ago

@phucnsp Removing the 1d positional embeddings will perform better on some tasks while worse on other tasks. It depends on what task you are working on. Please feel free to remove it if you find the results are better.