search

NormXU / ERNIE-Layout-Pytorch

An unofficial Pytorch implementation of ERNIE-Layout which is originally released through PaddleNLP.
http://arxiv.org/abs/2210.06155
MIT License
96 stars 11 forks source link

diff between paddle and torch #3

Closed WallE-Chang closed 1 year ago

WallE-Chang commented 1 year ago

Hi, NormXU Have you compared the difference of ernie-layout output between paddle version and pytorch version ?

NormXU commented 1 year ago

Hi Chang,

I have compared the ernie-layout output between the Paddle version and Pytorch version with the same input, and the difference between them can be ignored. Actually, I barely translated paddle verion line by line and converted all the weights Paddle released into Pytorch-style.

WallE-Chang commented 1 year ago

nice work !! Can you share the python file which you use to compared the ernie-layout output between the Paddle version and Pytorch version with the same input ? It is important to prove your work

WallE-Chang commented 1 year ago

By the way, the convert2torch file may have some bugs. model.ernie_layout.init_config can not be dumped. I fix it by following code: 

with open(os.path.join(pretrain_torch_model_or_path, "config.json"), "w") as outfile:
    config = model.ernie_layout.init_config
    config['init_args'] = (config['init_args'][0].__dict__)
    outfile.write(json.dumps(config))
NormXU commented 1 year ago

Sure, I can share the codes to compare the output between the two versions. I will update it sooner or later. Thank for your advice

NormXU commented 1 year ago

I add this script to compare the output between the two versions. Hope this can be helpful.

The MSE between two versions: eps of pooled output: 0.004253871738910675; eps of sequence output: 3.5654803762219522e-12.

WallE-Chang commented 1 year ago

Your work is very excellent, and can I add your wechat for further discussion?

NormXU commented 1 year ago

Thank you. Please feel free to post any questions you may have here. I hope others will be able to review our discussion.