fazega
commented
11 months ago First, note that the average score for the transformer on this task is 67.7% ± 15.2 (standard deviation over 10 seeds). The number we report is the maximum score we got over 10 seeds. Unfortunately it's impossible to get perfect reproducibility (from the given best seed) just because we don't use the exact same hardware. And OOD performance is very brittle, especially for Transformers.
Second, I think Anian leaked some parameters we used internally for experiments that we removed in the final paper. You can ignore those (such as act_tau, used for Adaptive Computation Time, that we also tried).
In your case, the accuracy is very low even within the range of training lengths, so there is a problem. Can you try a lower learning rate (like 1e-4)? What do the losses look like? I will launch the exact set of parameters to compare on my own machine ASAP.
kirk86
anianruoss
Hi, I've tried to reproduce the results on the task of even pairs similar to what another user was asking on issue #2. I've noticed you've posted the hyperparameters to reproduce the results but some arguments are not callable directly from the command line, e.g.
act_tauwhere/how do we set this argument? Even after setting most hyperparameters according to the values you've provided I still can't seem to get the results reported in the paper. Any ideas what I'm doing wrong/missing?For instance, to reproduce I call the example.py script with the following params: --architecture transformer_encoder \ --task even_pairs \ --ckpt_frequency 10000 \ --computation_steps_mult 0 \ --compute_full_range_test True \ --curriculum uniform \ --curriculum_kwargs.training_range 40 \ --eval_log_frequency 500 \ --is_autoregressive False \ --learning_rate 0.0005 \ --max_range_test_length 500 \ --model_init_seed 3 \ --range_test_sub_batch_size 32 \ --range_test_total_batch_size 512 \ --save_frequency 100000 \ --task even_pairs \ --task_level regular \ --training_data_seed 1 \ --training_log_frequency 500 \ --training_steps 1000000 \ --validation_batch_size 128 \ --validation_data_seed 1 \ --validation_lengths [60, 200, 500] \ --batch_size 128
Then, I get the following results, which to my interpretation is quite bad, assuming the
Network score: 0.494corresponds to network's accuracy on the even pairs task. Any suggestions?