same train config but different performance

[jetyingjia]

Thank you for this perfect projects! I have made some experiments: with same seettings(code) but get different results.

SiamFC-3s-color-scratch: train_config | TB50 | TB100 | OTB2013 momentum、momentum0.9 | 51.14 | 58.13 | 60.41 momentum、momentum0.9_2nd | 50.52 | 57.08 | 59.12 momentum、momentum0.9_3rd | 51.76 | 57.43 | 59.12

In this project, many "fix seed for reproducing experiments", but I cannot reproducing my results. I only rename the RUN_NAME = 'SiamFC-3s-color-scratch' with different name. Is there something I missing?

I also have some questions about config: a. 'num_examples_per_epoch': 5.32e4, I don't understanding how to get this num(5.32e4) b. when I debug this code, using (ipdb), but something wrong python -m ipdb ./experiments/SiamFC-3s-color-scratch.py train_siamese_model.py:90

[[14;1RERROR - SiamFC-3s-color-scratch-train_val_15_17_3rd - Failed after 0:00:00! Traceback (most recent call last): File "/home/users/buyingjia/.local/virenv/tf/lib/python2.7/site-packages/ipdb/main.py", line 169, in main pdb._runscript(mainpyfile) File "/usr/lib/python2.7/pdb.py", line 1233, in _runscript self.run(statement) File "/usr/lib/python2.7/bdb.py", line 400, in run exec cmd in globals, locals File "", line 1, in File "./experiments/SiamFC-3s-color-scratch.py", line 29, in '--enforce_clean': False, File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/sacred/experiment.py", line 209, in run run() File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/sacred/run.py", line 221, in call self.result = self.main_function(args) File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/sacred/config/captured_function.py", line 46, in captured_function result = wrapped(args, **kwargs) File "./experiments/../train_siamese_model.py", line 90, in main train_dir = train_config['train_dir'] File "./experiments/../train_siamese_model.py", line 90, in main train_dir = train_config['train_dir'] File "/usr/lib/python2.7/bdb.py", line 49, in trace_dispatch return self.dispatch_line(frame) File "/usr/lib/python2.7/bdb.py", line 67, in dispatch_line self.user_line(frame) File "/usr/lib/python2.7/pdb.py", line 158, in user_line self.interaction(frame, None) File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/IPython/core/debugger.py", line 288, in interaction OldPdb.interaction(self, frame, traceback) File "/usr/lib/python2.7/pdb.py", line 210, in interaction self.cmdloop() File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/IPython/terminal/debugger.py", line 96, in cmdloop line = self.pt_cli.run(reset_current_buffer=True).text File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/prompt_toolkit/interface.py", line 425, in run self._redraw() File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/prompt_toolkit/interface.py", line 358, in _redraw self.renderer.render(self, self.layout, is_done=self.is_done) File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/prompt_toolkit/renderer.py", line 405, in render size = output.get_size() File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/prompt_toolkit/terminal/vt100_output.py", line 426, in get_size rows, columns = _get_size(stdout.fileno()) File "/home/users/buyingjia/.local/virenv/tf/local/lib/python2.7/site-packages/prompt_toolkit/terminal/vt100_output.py", line 365, in _get_size fcntl.ioctl(fileno, termios.TIOCGWINSZ, buf) IOError: [Errno 25] Inappropriate ioctl for device Uncaught exception. Entering post mortem debugging Running 'cont' or 'step' will restart the program

Thank you in advance!

[huanglianghua]

@jetyingjia The random seed for tensorflow not really works. See https://github.com/tensorflow/tensorflow/issues/9171.

[huanglianghua]

The `num_samples_per_epoch' is calculated by using the number of videos in ILSVRC dataset to time the number of pairs you'd like to sample for each video.

[bilylee]

Hi,

Totally Deterministic training is challenging as small differences during training will be amplified after hunderds of thousands of training iterations. However, the performance should be stable in the sense that you can get results slightly better or worse than reported for each run.