fukun07 / neural-image-captioning

Using scene-specific contexts and region-based attention in neural image captioning
MIT License
44 stars 18 forks source link

IOError: [Errno 32]Broken pipe #3

Closed gargisri68 closed 7 years ago

gargisri68 commented 7 years ago

after running the train.py getting the following error:

Bleu_3: 0.227 Bleu_4: 0.130 computing METEOR score... METEOR: 0.160 computing Rouge score... ROUGE_L: 0.417 computing CIDEr score... CIDEr: 0.346 ---- this Bleu-4 = [0.130], best Bleu-4 = [0.130], endding -> 42 snapshot=33, iter=3300, loss=3.572622, time=2415.5 sec computing Bleu score... Bleu_1: 0.578 Bleu_2: 0.385 Bleu_3: 0.237 Bleu_4: 0.141 computing METEOR score... METEOR: 0.164 computing Rouge score... ROUGE_L: 0.426 computing CIDEr score... CIDEr: 0.377 ---- this Bleu-4 = [0.141], best Bleu-4 = [0.141], endding -> 43 snapshot=34, iter=3400, loss=3.424325, time=2681.8 sec computing Bleu score... Bleu_1: 0.580 Bleu_2: 0.390 Bleu_3: 0.246 Bleu_4: 0.149 computing METEOR score... Traceback (most recent call last): File "train.py", line 242, in display=100, starting=m, endding=10, validation=100, patience=10, logger=logger) File "train.py", line 43, in train scores = validate(beam_searcher, valid_set, logger) File "train.py", line 70, in validate scores = evaluate(gt_file, res_file, logger) File "/home/cse/phd/16071009/nic/neural-image-captioning-master/codes/tools.py", line 77, in evaluate score, scores = scorer.compute_score(gts, res) File "/home/cse/phd/16071009/nic/neural-image-captioning-master/codes/pycoco/meteor/meteor.py", line 40, in compute_score stat = self._stat(res[i][0], gts[i]) File "/home/cse/phd/16071009/nic/neural-image-captioning-master/codes/pycoco/meteor/meteor.py", line 62, in _stat self.meteor_p.stdin.write('{}\n'.format(score_line)) IOError: [Errno 32] Broken pipe Unhandled exception in thread started by <bound method Reader.bootstrap of <Reader(Thread-2, stopped 140361061521152)>> Traceback (most recent call last): File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/threading.py", line 774, in bootstrap self.__bootstrap_inner() File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/threading.py", line 814, in __bootstrap_inner (self.name, _format_exc())) File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/traceback.py", line 242, in format_exc return ''.join(format_exception(etype, value, tb, limit)) File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/traceback.py", line 141, in format_exception list = list + format_tb(tb, limit) File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/traceback.py", line 76, in format_tb return format_list(extract_tb(tb, limit)) File "/home/cse/phd/16071009/anaconda3/envs/theano/lib/python2.7/traceback.py", line 100, in extract_tb linecache.checkcache(filename) TypeError: 'NoneType' object is not callable (theano) [16071009@compute-01 codes]$

fukun07 commented 7 years ago

@gargisri68 Hi, it seems to be the problem of METEOR evaluator.

At each time to evaluate METEOR, a new thread will be setup to call the java interface of the original METEOR evaluator (implemented in "pycoco", provided by the COCO team). If you constantly monitors METOER during training, too many threads may lead to broken pipe problem.

Commenting out METEOR in evaluate may be a solution. However, you can still recover it at evaluation stage, in which the evaluator is called only once.