Closed gargisri68 closed 7 years ago
@gargisri68 Hi, it seems to be the problem of METEOR evaluator.
At each time to evaluate METEOR, a new thread will be setup to call the java interface of the original METEOR evaluator (implemented in "pycoco", provided by the COCO team). If you constantly monitors METOER during training, too many threads may lead to broken pipe problem.
Commenting out METEOR in evaluate may be a solution. However, you can still recover it at evaluation stage, in which the evaluator is called only once.
after running the train.py getting the following error:
Bleu_3: 0.227 Bleu_4: 0.130 computing METEOR score... METEOR: 0.160 computing Rouge score... ROUGE_L: 0.417 computing CIDEr score... CIDEr: 0.346 ---- this Bleu-4 = [0.130], best Bleu-4 = [0.130], endding -> 42 snapshot=33, iter=3300, loss=3.572622, time=2415.5 sec computing Bleu score... Bleu_1: 0.578 Bleu_2: 0.385 Bleu_3: 0.237 Bleu_4: 0.141 computing METEOR score... METEOR: 0.164 computing Rouge score... ROUGE_L: 0.426 computing CIDEr score... CIDEr: 0.377 ---- this Bleu-4 = [0.141], best Bleu-4 = [0.141], endding -> 43 snapshot=34, iter=3400, loss=3.424325, time=2681.8 sec computing Bleu score... Bleu_1: 0.580 Bleu_2: 0.390 Bleu_3: 0.246 Bleu_4: 0.149 computing METEOR score... Traceback (most recent call last): File "train.py", line 242, in
display=100, starting=m, endding=10, validation=100, patience=10, logger=logger)
File "train.py", line 43, in train
scores = validate(beam_searcher, valid_set, logger)
File "train.py", line 70, in validate
scores = evaluate(gt_file, res_file, logger)
File "/home/cse/phd/16071009/nic/neural-image-captioning-master/codes/tools.py", line 77, in evaluate
score, scores = scorer.compute_score(gts, res)
File "/home/cse/phd/16071009/nic/neural-image-captioning-master/codes/pycoco/meteor/meteor.py", line 40, in compute_score
stat = self._stat(res[i][0], gts[i])
File "/home/cse/phd/16071009/nic/neural-image-captioning-master/codes/pycoco/meteor/meteor.py", line 62, in _stat
self.meteor_p.stdin.write('{}\n'.format(score_line))
IOError: [Errno 32] Broken pipe
Unhandled exception in thread started by <bound method Reader.bootstrap of <Reader(Thread-2, stopped 140361061521152)>>
Traceback (most recent call last):
File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/threading.py", line 774, in bootstrap
self.__bootstrap_inner()
File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/threading.py", line 814, in __bootstrap_inner
(self.name, _format_exc()))
File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/traceback.py", line 242, in format_exc
return ''.join(format_exception(etype, value, tb, limit))
File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/traceback.py", line 141, in format_exception
list = list + format_tb(tb, limit)
File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/traceback.py", line 76, in format_tb
return format_list(extract_tb(tb, limit))
File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/traceback.py", line 100, in extract_tb
linecache.checkcache(filename)
TypeError: 'NoneType' object is not callable
(theano) [16071009@compute-01 codes]$