search

tech-srl / code2seq

Code for the model presented in the paper: "code2seq: Generating Sequences from Structured Representations of Code"
http://code2seq.org
MIT License
556 stars 164 forks source link

Empty hypothesis when periods are included in dataset #62

Closed vijayantajain closed 4 years ago

vijayantajain commented 4 years ago

Hello Uri,

I am trying to train the Code2Seq model on the Funcom dataset. I tokenized the dataset by removing all special characters except for periods and commas. When I train a Code2Seq model on this dataset, I get the following error :

Saved after 1 epochs in: models/funcom-test/model_iter1
Finished 1 epochs
Done testing, epoch reached
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
     [[{{node IteratorGetNext}}]]
  (1) Out of range: End of sequence
     [[{{node IteratorGetNext}}]]
     [[IteratorGetNext/_27]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vijayantajain/code/experiments/code2seq/model.py", line 96, in train
    _, batch_loss = self.sess.run([optimizer, train_loss])
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflowI have tried this couple of times by changing model configurations, batch-size but still get this error when the comments have periods.
_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
     [[node IteratorGetNext (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
  (1) Out of range: End of sequence
     [[node IteratorGetNext (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
     [[IteratorGetNext/_27]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'IteratorGetNext':
  File "code2seq.py", line 39, in <module>
    model.train()
  File "/home/vijayantajain/code/experiments/code2seq/model.py", line 77, in train
    config=self.config)
  File "/home/vijayantajain/code/experiments/code2seq/reader.py", line 43, in __init__
    self.output_tensors = self.compute_output()
  File "/home/vijayantajain/code/experiments/code2seq/reader.py", line 192, in compute_output
    return self.iterator.get_next()
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
    name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "code2seq.py", line 39, in <module>
    model.train()
  File "/home/vijayantajain/code/experiments/code2seq/model.py", line 108, in train
    results, precision, recall, f1, rouge = self.evaluate()
  File "/home/vijayantajain/code/experiments/code2seq/model.py", line 230, in evaluate
    hyp_path=predicted_file_name, ref_path=ref_file_name, avg=True, ignore_empty=True)
  File "/opt/conda/lib/python3.7/site-packages/rouge/rouge.py", line 47, in get_scores
    ignore_empty=ignore_empty)
  File "/opt/conda/lib/python3.7/site-packages/rouge/rouge.py", line 105, in get_scores
    return self._get_avg_scores(hyps, refs)
  File "/opt/conda/lib/python3.7/site-packages/rouge/rouge.py", line 145, in _get_avg_scores
    sc = fn(hyp, ref, exclusive=self.exclusive)
  File "/opt/conda/lib/python3.7/site-packages/rouge/rouge.py", line 53, in <lambda>
    "rouge-1": lambda hyp, ref, **k: rouge_score.rouge_n(hyp, ref, 1, **k),
  File "/opt/conda/lib/python3.7/site-packages/rouge/rouge_score.py", line 253, in rouge_n
    raise ValueError("Hypothesis is empty.")
ValueError: Hypothesis is empty.

When I check pred.txt in the models directory I see that some lines are empty which is most likely causing the error.

When I remove all special characters in the Funcom dataset, including periods and comma, and train again I do not get this error.

Any idea on why the model would not predict anything for some examples if there are periods and commas in the dataset?

Thanks! VJ

urialon commented 4 years ago

Hi @vijayantajain , Thank you for your interest in code2seq!

I hypothesize that it's just an issue of the first iteration - the model is not trained enough to produce good results, so sometimes the empty hypothesis is the best. We can manually prevent the model from predicting end-of-sequence at the first step. But before that - can you please try to disable Rouge (by returning always a dummy score, without checking rouge), so your model can continue training for additional epochs, and see if the problem is solved by itself in the next training iterations?

vijayantajain commented 4 years ago

Hi @urialon,

I was going through the logs from previous runs I tried and I came across a very similar problem. Here's the log for that.

Finished 1 epochs
Done testing, epoch reached
Evaluation time: 0h0m3s
Accuracy after 1 epochs: 0.00000
After 1 epochs: Precision: 0.99527, recall: 0.68522, F1: 0.81165
Rouge:  {'rouge-1': {'f': 0.012882575672040321, 'p': 0.010102335345053785, 'r': 0.02283571109591501}, 'rouge-2': {'f': 0.0002206531098050254, 'p': 0.00026968716289104636, 'r': 0.00018670649738610905}, 'rouge-l': {'f': 0.03772484042499432, 'p': 0.1407766990291262, 'r': 0.022670375460831883}}
Saved after 1 epochs in: models/DEBUG/funcom-modified-test-3-code2seq-large-test-2-ignore-empty-true/model_iter1
Finished 1 epochs
Done testing, epoch reached
Evaluation time: 0h0m1s
Accuracy after 2 epochs: 0.00000
After 2 epochs: Precision: 0.86817, recall: 0.67134, F1: 0.75717
Rouge:  {'rouge-1': {'f': 0.08981957268154325, 'p': 0.10148185998978035, 'r': 0.10669313639812034}, 'rouge-2': {'f': 0.00018318371095680936, 'p': 0.00025549310168625444, 'r': 0.00014277555682467162}, 'rouge-l': {'f': 0.12804680996155146, 'p': 0.7378640776699029, 'r': 0.07233086263065558}}
Finished 1 epochs
Done testing, epoch reached
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
    target_list, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
     [[{{node IteratorGetNext}}]]
  (1) Out of range: End of sequence
     [[{{node IteratorGetNext}}]]
     [[model/gradients/model/embedding_lookup_grad/Size/_33]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vijayantajain/code/experiments/code2seq/model.py", line 96, in train
    _, batch_loss = self.sess.run([optimizer, train_loss])
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
    run_metadata_ptr)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
    feed_dict_tensor, options, run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
    run_metadata)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
     [[node IteratorGetNext (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
  (1) Out of range: End of sequence
     [[node IteratorGetNext (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
     [[model/gradients/model/embedding_lookup_grad/Size/_33]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'IteratorGetNext':
  File "code2seq.py", line 39, in <module>
    model.train()
  File "/home/vijayantajain/code/experiments/code2seq/model.py", line 77, in train
    config=self.config)
  File "/home/vijayantajain/code/experiments/code2seq/reader.py", line 43, in __init__
    self.output_tensors = self.compute_output()
  File "/home/vijayantajain/code/experiments/code2seq/reader.py", line 192, in compute_output
    return self.iterator.get_next()
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next
    name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
    op_def=op_def)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
    attrs, op_def, compute_device)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
    op_def=op_def)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
    self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "code2seq.py", line 39, in <module>
    model.train()
  File "/home/vijayantajain/code/experiments/code2seq/model.py", line 108, in train
    results, precision, recall, f1, rouge = self.evaluate()
  File "/home/vijayantajain/code/experiments/code2seq/model.py", line 230, in evaluate
    hyp_path=predicted_file_name, ref_path=ref_file_name, avg=True, ignore_empty=True)
  File "/opt/conda/lib/python3.7/site-packages/rouge/rouge.py", line 47, in get_scores
    ignore_empty=ignore_empty)
  File "/opt/conda/lib/python3.7/site-packages/rouge/rouge.py", line 98, in get_scores
    hyps, refs = zip(*hyps_and_refs)
ValueError: not enough values to unpack (expected 2, got 0)

In these iterations, pred.txt is completely empty.

I tried by commenting rouge as you suggested and still the same result. On batches, I actually get loss as nan.

urialon commented 4 years ago

Hi, According to the stack trace, it looks like "rouge.py" is still called. I should make the code more robust, but in the meantime - You can either catch the exception, or comment the call to rouge.

On Sat, 18 Jul 2020 at 20:07 Vijayanta Jain notifications@github.com wrote:

Hi @urialon https://github.com/urialon,

I was going through the logs from previous runs I tried and I came across a very similar problem. Here's the log for that.

Finished 1 epochsDone testing, epoch reachedEvaluation time: 0h0m3sAccuracy after 1 epochs: 0.00000After 1 epochs: Precision: 0.99527, recall: 0.68522, F1: 0.81165Rouge: {'rouge-1': {'f': 0.012882575672040321, 'p': 0.010102335345053785, 'r': 0.02283571109591501}, 'rouge-2': {'f': 0.0002206531098050254, 'p': 0.00026968716289104636, 'r': 0.00018670649738610905}, 'rouge-l': {'f': 0.03772484042499432, 'p': 0.1407766990291262, 'r': 0.022670375460831883}}Saved after 1 epochs in: models/DEBUG/funcom-modified-test-3-code2seq-large-test-2-ignore-empty-true/model_iter1Finished 1 epochsDone testing, epoch reachedEvaluation time: 0h0m1sAccuracy after 2 epochs: 0.00000After 2 epochs: Precision: 0.86817, recall: 0.67134, F1: 0.75717Rouge: {'rouge-1': {'f': 0.08981957268154325, 'p': 0.10148185998978035, 'r': 0.10669313639812034}, 'rouge-2': {'f': 0.00018318371095680936, 'p': 0.00025549310168625444, 'r': 0.00014277555682467162}, 'rouge-l': {'f': 0.12804680996155146, 'p': 0.7378640776699029, 'r': 0.07233086263065558}}Finished 1 epochsDone testing, epoch reachedTraceback (most recent call last): File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata)tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence [[{{node IteratorGetNext}}]] (1) Out of range: End of sequence [[{{node IteratorGetNext}}]] [[model/gradients/model/embedding_lookup_grad/Size/33]]0 successful operations.0 derived errors ignored. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/vijayantajain/code/experiments/code2seq/model.py", line 96, in train , batch_loss = self.sess.run([optimizer, train_loss]) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message)tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found. (0) Out of range: End of sequence [[node IteratorGetNext (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] (1) Out of range: End of sequence [[node IteratorGetNext (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[model/gradients/model/embedding_lookup_grad/Size/_33]]

0 successful operations.0 derived errors ignored. Original stack trace for 'IteratorGetNext': File "code2seq.py", line 39, in model.train() File "/home/vijayantajain/code/experiments/code2seq/model.py", line 77, in train config=self.config) File "/home/vijayantajain/code/experiments/code2seq/reader.py", line 43, in init self.output_tensors = self.compute_output() File "/home/vijayantajain/code/experiments/code2seq/reader.py", line 192, in compute_output return self.iterator.get_next() File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py", line 426, in get_next name=name) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_dataset_ops.py", line 2518, in iterator_get_next output_shapes=output_shapes, name=name) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred: Traceback (most recent call last): File "code2seq.py", line 39, in model.train() File "/home/vijayantajain/code/experiments/code2seq/model.py", line 108, in train results, precision, recall, f1, rouge = self.evaluate() File "/home/vijayantajain/code/experiments/code2seq/model.py", line 230, in evaluate hyp_path=predicted_file_name, ref_path=ref_file_name, avg=True, ignore_empty=True) File "/opt/conda/lib/python3.7/site-packages/rouge/rouge.py", line 47, in get_scores ignore_empty=ignore_empty

) File "/opt/conda/lib/python3.7/site-packages/rouge/rouge.py", line 98, in get_scores hyps, refs = zip(*hyps_and_refs)ValueError: not enough values to unpack (expected 2, got 0)

In these iterations, pred.txt is completely empty.

I tried by commenting rouge as you suggested and still the same result. On batches, I actually get loss as nan.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/tech-srl/code2seq/issues/62#issuecomment-660511995, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSOXMGZSNCDQMWEEF4KP7TR4HJE3ANCNFSM4O46BJFA .

vijayantajain commented 4 years ago

Hi,

Sorry for the confusion. The previous stack trace is not after I commented out rouge. It was one of the experiments I tried and had a slightly different error than the first stack trace.

Following is the stack trace after I commented out the rouge part. 

Average loss at batch 100: nan,     throughput: 175 samples/sec
Average loss at batch 200: nan,     throughput: 206 samples/sec
Average loss at batch 300: nan,     throughput: 206 samples/sec
Average loss at batch 400: nan,     throughput: 206 samples/sec
Average loss at batch 500: nan,     throughput: 206 samples/sec
Average loss at batch 600: nan,     throughput: 206 samples/sec
Average loss at batch 700: nan,     throughput: 206 samples/sec
Average loss at batch 800: nan,     throughput: 206 samples/sec
Average loss at batch 900: nan,     throughput: 206 samples/sec
Average loss at batch 1000: nan,    throughput: 205 samples/sec
Average loss at batch 1100: nan,    throughput: 206 samples/sec
...
Prediction throughput: 525
Prediction throughput: 540
Prediction throughput: 545
Prediction throughput: 547
Prediction throughput: 548
Prediction throughput: 548
Prediction throughput: 549
Prediction throughput: 549
Prediction throughput: 549
Prediction throughput: 550
Prediction throughput: 547
Prediction throughput: 548
Prediction throughput: 548
Prediction throughput: 548
Prediction throughput: 549
Prediction throughput: 549
Prediction throughput: 549
Prediction throughput: 549
Prediction throughput: 549
Prediction throughput: 549
Prediction throughput: 550
Prediction throughput: 549
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Prediction throughput: 550
Done testing, epoch reached
Evaluation time: 0h6m35s
Accuracy after 1 epochs: 0.00000
After 1 epochs: Precision: 0.00000, recall: 0.00000, F1: 0.00000
Rouge:  1.0
Average loss at batch 13600: nan,   throughput: 27 samples/sec
Average loss at batch 13700: nan,   throughput: 207 samples/sec
Average loss at batch 13800: nan,   throughput: 207 samples/sec
Average loss at batch 13900: nan,   throughput: 206 samples/sec
Average loss at batch 14000: nan,   throughput: 206 samples/sec
Average loss at batch 14100: nan,   throughput: 206 samples/sec
Average loss at batch 14200: nan,   throughput: 206 samples/sec
Average loss at batch 14300: nan,   throughput: 205 samples/sec
Average loss at batch 14400: nan,   throughput: 205 samples/sec
Average loss at batch 14500: nan,   throughput: 206 samples/sec
Average loss at batch 14600: nan,   throughput: 205 samples/sec
Average loss at batch 14700: nan,   throughput: 205 samples/sec
Average loss at batch 14800: nan,   throughput: 205 samples/sec
Average loss at batch 14900: nan,   throughput: 205 samples/sec
Average loss at batch 15000: nan,   throughput: 205 samples/sec
Average loss at batch 15100: nan,   throughput: 205 samples/sec
...

This goes for 3 epochs and then it stops (I set config.PATIENCE as 3). I checked pred.txt file in the models directory and it is empty.

urialon commented 4 years ago

Where did you comment rouge? Because rouge should be run after the first training iteration, at test time. So the training loss should not be affected by this (because rouge happens later).

vijayantajain commented 4 years ago

I commented them in model.py as follows - 

        elapsed = int(time.time() - eval_start_time)
        precision, recall, f1 = self.calculate_results(true_positive, false_positive, false_negative)
#       files_rouge = FilesRouge()
#       rouge = files_rouge.get_scores(
#           hyp_path=predicted_file_name, ref_path=ref_file_name, avg=True, ignore_empty=True)
        print("Evaluation time: %sh%sm%ss" % ((elapsed // 60 // 60), (elapsed // 60) % 60, elapsed % 60))
        return num_correct_predictions / total_predictions, \
               precision, recall, f1, 1.0
urialon commented 4 years ago

Are you sure that:

  1. This is the only change and
  2. Without this change, the loss is not nan?

Because this doesn't have anything to do with the loss, it is reached only afterward.

vijayantajain commented 4 years ago

You are right. It's not the rouge, it's the dataset. Originally, I had punctuations in the dataset, but after I removed it, I don't get the error. Any ideas why that might be happening?

urialon commented 4 years ago

Oh, I suspect that it's the comma (",") symbol. Since we use a comma to separate between the path and the tokens of each "context" (see the format here) - if you have a token that contains commas - it breaks the input.

Can you keep the punctuations that you need - but remove all commas?

vijayantajain commented 4 years ago

Makes sense! I forgot about the format of the input. I will need to try this.

vijayantajain commented 4 years ago

I tried with only periods and no other punctuation and it didn't work. Getting nan as loss. Only when there is no punctuation, the training runs fine.

urialon commented 4 years ago

Can you try to find a single example that makes the nan loss? It shouldn't be too hard, because it seems that you are getting a nan loss starting from the first batch.

If you create a dataset that is created of only the first line in your dataset (like: cat my_data.train.c2s | head -1 > my_data_one_line.train.c2s) - are you still getting a nan?

vijayantajain commented 4 years ago

No, with one example it was not nan. It worked well. I did not even have any trouble with rouge. Although, the model only predicted a sequence of periods but it sort of worked.

urialon commented 4 years ago

I meant - maybe there is a single (or a few) examples that cause the nan and we can find them using a binary search:

  1. Create a dataset of the first 1000 examples - check if it produces nans
  2. If not - double its size. If it does produce nan loss - cut the dataset in half, and find the half that does create the nan loss.
  3. repeat...

But before you do that - maybe you can remove the "pipe" symbol (|) as well as commas and see if it fixes that nans?

On Sat, Jul 25, 2020 at 12:37 AM Vijayanta Jain notifications@github.com wrote:

No, with one example it was not nan. It worked well. I did not even have any trouble with rouge. Although, the model only predicted a sequence of periods but it sort of worked.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tech-srl/code2seq/issues/62#issuecomment-663744561, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSOXMCKVENLHSEXR3N6S53R5H5JDANCNFSM4O46BJFA .

vijayantajain commented 4 years ago

Ok, so we are finding individual sample(s) that might be causing the error. Sure. Regarding removing the pip symbols, you mean to remove them from the target part of the data, right? If so, should I then separate them with spaces?

urialon commented 4 years ago

Hi, Regarding the pipe symbol - I meant to remove them from the "tokens", if they exist. But I guess that it's very rare, only if they appear in strings, such as:

String s = "hello | world";

If you find a single example that causes nan and paste it here, I will probably be able to see what's wrong with it.

urialon commented 4 years ago

Closing due to inactivity, feel free to re-open.

vijayantajain commented 4 years ago

Hi @urialon,

Sorry for the delayed response, I got caught up with other things.

I tried finding out individual samples but it turns out that this happens in larger quantities.

Here's what I tried:

  1. I checked with first lines 0-1000 - no error.
  2. Then I doubled the number of lines and checked with 0-2048 lines. This caused an error and pred.txt was empty. So I halved the lines to find the sample giving the error.
  3. I tried with lines 1000-2048, which gave no error. This was confusing because lines 0-1000 did not give any error but 1000-2048 did.
  4. Just to check, I then tried with lines 1000-1501 - no error.
  5. Finally, with lines 1501-2048 - no error.

My guess is it is not an individual sample that is causing this error but the dataset itself that is causing this error.

urialon commented 4 years ago

Let's try this:

  1. Keep a copy of the original training data of lines 0-2048. Verify again that it causes nan loss.
  2. Use the following commands to remove hidden characters from the data (the data right before running the script preprocess.py): 
    sed -i 's/\x0//g' my_file.txt
sed -i 's/\x0b//g' my_file.txt
sed -i 's/\x1d//g' my_file.txt
sed -i 's/\x1db//g' my_file.txt
tr -cd "[:print:]\n" < my_file.txt > my_file_new.txt

Then check if the new file my_file_new.txt is different than the old file my_file.txt using diff or md5sum.

vijayantajain commented 4 years ago

So I did that on the training data with 2k lines and found some lines with non-alphanumeric characters. These were also present in the original data. Here are some examples -

Original train.c2s file: trueをセットするとjs|engine.receive||がsent-byのポートで待ち受け、|falseにするとjs|engine.receive||がlocalポートで待ち受ける動作モードとなる。|.|true|sent-by|false|local|. void,Void0|Mth|Nm1,METHOD_NAME void,Void0|Mth|Prm|VDID0,b ...

Modified: truejs|engine.receive||sent-by|falsejs|engine.receive||local|.|true|sent-by|false|local|. void,Void0|Mth|Nm1,METHOD_NAME void,Void0|Mth|Prm|VDID0,b void,Void0|Mth|Prm|Prim1,boolean METHOD_NAME,Nm1|Mth|Prm|VDID0,b ...

Original: scroll��|off|image��|�׷�|���ƾ�|�ѵ�|. void,Void0|Mth|Nm1,METHOD_NAME void,Void0|Mth|Prm|VDID0,e ...

Modified: scroll|off|image||||. void,Void0|Mth|Nm1,METHOD_NAME void,Void0|Mth|Prm|VDID0,e ...

Original: este|m�todo|inicializa|o|txt|modelos|disponiveis|.|return|javax.swing.jtext|field|. j|text|field,Cls0|Mth|Nm1,METHOD_NAME j|text|field,Cls0|Mth|Bk|If|Eq|Nm0 ...

Modified: este|mtodo|inicializa|o|txt|modelos|disponiveis|.|return|javax.swing.jtext|field|. j|text|field,Cls0|Mth|Nm1,METHOD_NAME j|text|field,Cls0|Mth|Bk|If|Eq|Nm0 ...

However it did not work. I tried again with the modified data and still getting nan loss.

Maybe I can send you the data and you can get a better idea about it?

urialon commented 4 years ago

It indeed seems like there were many unicode characters.

It also seems that this cleaning created some empty targets, for example scroll|off|image||||. Try to remove these "empty" targets by converting the target sequence into scroll|off|image .

On Thu, Sep 3, 2020 at 1:50 AM Vijayanta Jain notifications@github.com wrote:

So I did that on the training data with 2k lines and found some lines with non-alphanumeric characters. These were also present in the original data. Here are some examples -

Original train.c2s file: trueをセットするとjs|engine.receive||がsent-byのポートで待ち受け、|falseにするとjs|engine.receive||がlocalポートで待ち受ける動作モードとなる。|.|true|sent-by|false|local|. void,Void0|Mth|Nm1,METHOD_NAME void,Void0|Mth|Prm|VDID0,b ...

Modified: truejs|engine.receive||sent-by|falsejs|engine.receive||local|.|true|sent-by|false|local|. void,Void0|Mth|Nm1,METHOD_NAME void,Void0|Mth|Prm|VDID0,b void,Void0|Mth|Prm|Prim1,boolean METHOD_NAME,Nm1|Mth|Prm|VDID0,b ...

Original: scroll��|off|image��|�׷�|���ƾ�|�ѵ�|. void,Void0|Mth|Nm1,METHOD_NAME void,Void0|Mth|Prm|VDID0,e ...

Modified: scroll|off|image||||. void,Void0|Mth|Nm1,METHOD_NAME void,Void0|Mth|Prm|VDID0,e ...

Original: este|m�todo|inicializa|o|txt|modelos|disponiveis|.|return|javax.swing.jtext|field|. j|text|field,Cls0|Mth|Nm1,METHOD_NAME j|text|field,Cls0|Mth|Bk|If|Eq|Nm0 ...

Modified: este|mtodo|inicializa|o|txt|modelos|disponiveis|.|return|javax.swing.jtext|field|. j|text|field,Cls0|Mth|Nm1,METHOD_NAME j|text|field,Cls0|Mth|Bk|If|Eq|Nm0 ...

However it did not work. I tried again with the modified data and still getting nan loss.

Maybe I can send you the data and you can get a better idea about it?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tech-srl/code2seq/issues/62#issuecomment-686068646, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSOXMCYZPGKWZEEP7W37FDSD3D4TANCNFSM4O46BJFA .

vijayantajain commented 4 years ago

Hi @urialon,

I removed all the redundant pipes (|| or |||) from the dataset but still getting nan when training with the refined dataset.