search

oalieno / asm2vec-pytorch

Unofficial implementation of asm2vec using pytorch ( with GPU acceleration )
MIT License
74 stars 21 forks source link

I tried to recapture the experiments in the Asm2Vec paper using this implementation yet my results are really bad - what is the problem? #13

Open anandramakris opened 2 years ago

anandramakris commented 2 years ago

I attempted to use this library to figure out the cosine similarity between the O0 and O3 optimized functions in coreutils version 8.30 (taken from https://github.com/yueduan/DeepBinDiff). In order to try to replicate the results, I used the same options for training as in the paper - embedding dimension 200, learning rate 0.025 - and changed the number of random walks to be 10 as in the paper.

I found the average cosine similarity to be 0.128. Given that the results in the paper show that Asm2Vec should correctly match around 80% of O0 and O3 functions, the score is very poor. Do you have an explanation?

anandramakris commented 2 years ago

My original comparison script was faulty, but after fixing it the average cosine similarity was still only 0.151.

true-zk commented 1 year ago

Hi, have u solved ur problem?

I find its strange that whatever dataset I use to train the model, it shows a quite bad performance, about only 30% accuracy after 100 epochs.

So I think there is something wrong with this implement.