Issue in training

[estiver-alvarez]

Hi sir, I want to take this occasion to express my admiration for your great job extremely useful for my Bachelor's tesis. Currently I am trying train the model under my own embedding of C++ sources code.

The sizes of my training set is 72727 validation set is 13637, test set is 13637. when I try train the model I receive the issue of the image. may you indicate me How can I fix this issue. Thanks very much and greetings from colombia.

[urialon]

Hi @estiver-alvarez , Thank you for your interest in our work!

Are you using the keras backend? Can you use the TensorFlow backend instead?

Best, Uri

P.S - https://code2seq.org/ is a stronger version of code2vec. See also: https://anycodegen.org/

[Sohaib90]

Hello,

I am also facing the same problem. As you said to switch to tensorflow, I tried that too but when I use TF backend I get the error

Traceback (most recent call last): File "code2vec.py", line 23, in <module> model.train() File "/local/home/aru7rng/masterthesis/code2vec/tensorflow_model.py", line 97, in train evaluation_results = self.evaluate() File "/local/home/aru7rng/masterthesis/code2vec/tensorflow_model.py", line 174, in evaluate subtokens_evaluation_metric.update_batch(zip(original_names, top_words)) File "/local/home/aru7rng/masterthesis/code2vec/tensorflow_model.py", line 463, in update_batch prediction = self.filter_impossible_names_fn(top_words)[0] IndexError: list index out of range

Any idea why this would be happening? This only happened when I tried to remap my function names in the parser to some simplified function names rather than using the names I already had (hypothesis was that the function names were really complicated and had to be fixed) So I fixed this indexing issue by the solution of another thread https://github.com/tech-srl/code2vec/issues/117#issue-898122479

but this error did not occur when I was using the old pipeline without remapping the function names in the dataset

[urialon]

Hi @Sohaib90 , It seems that you are describing a problem that is unrelated to the original problem described in this issue.

Can you please create a separate issue and explain your problem and the changes that you have made?

At a glance, it seems that self.filter_impossible_names_fn(top_words) returns an empty list after your modification. Is it possible that it filters all the predicted words, so no prediction is left?

Best, Uri

[Avv22]

Hello,

I am also facing the same problem. As you said to switch to tensorflow, I tried that too but when I use TF backend I get the error

Traceback (most recent call last): File "code2vec.py", line 23, in <module> model.train() File "/local/home/aru7rng/masterthesis/code2vec/tensorflow_model.py", line 97, in train evaluation_results = self.evaluate() File "/local/home/aru7rng/masterthesis/code2vec/tensorflow_model.py", line 174, in evaluate subtokens_evaluation_metric.update_batch(zip(original_names, top_words)) File "/local/home/aru7rng/masterthesis/code2vec/tensorflow_model.py", line 463, in update_batch prediction = self.filter_impossible_names_fn(top_words)[0] IndexError: list index out of range

Any idea why this would be happening? This only happened when I tried to remap my function names in the parser to some simplified function names rather than using the names I already had (hypothesis was that the function names were really complicated and had to be fixed) So I fixed this indexing issue by the solution of another thread #117 (comment)

but this error did not occur when I was using the old pipeline without remapping the function names in the dataset

What is the command you run to train the model pls? I have data in place and run train.sh but it does not work. Here is my issue