search

sgugger / Adam-experiments

Experiments with Adam/AdamW/amsgrad
194 stars 36 forks source link

With latest version of Fastai. I am getting issue unexpected keyword amsgrad, not recognized #3

Closed adsmgth closed 6 years ago

adsmgth commented 6 years ago

ran 'train_cifar10.py' 3e-3 --wd=0.1 --wd_loss=False

Output lr: 0.003; moms: (0.95, 0.85); wd: 0.1; wd_loss: False; opt_fn: Adam; bs: 128; cyc_len: 30;beta2: 0.99; amsgrad: False; div: 10; pct: 0.075; lin_end: True; tta: False

Experiment 1

TypeError Traceback (most recent call last) /usr/local/lib/python3.6/dist-packages/Adam-experiments/train_cifar10.py in () 68 main_train(lr, moms, wd, wd_loss, opt_fn, bs, cyc_len, beta2, amsgrad, div, pct, lin_end, tta, fname) 69 ---> 70 if name == 'main': fire.Fire(train_lm)

/usr/local/lib/python3.6/dist-packages/fire/core.py in Fire(component, command, name) 125 context.update(caller_locals) 126 --> 127 component_trace = _Fire(component, args, context, name) 128 129 if component_trace.HasError():

/usr/local/lib/python3.6/dist-packages/fire/core.py in _Fire(component, args, context, name) 364 365 component, consumed_args, remaining_args, capacity = _CallCallable( --> 366 component, remaining_args) 367 368 # Update the trace.

/usr/local/lib/python3.6/dist-packages/fire/core.py in _CallCallable(fn, args) 540 (varargs, kwargs), consumed_args, remaining_args, capacity = parse(args) 541 --> 542 result = fn(*varargs, **kwargs) 543 return result, consumed_args, remaining_args, capacity 544

/usr/local/lib/python3.6/dist-packages/Adam-experiments/train_cifar10.py in train_lm(lr, moms, wd, wd_loss, opt_fn, bs, cyc_len, beta2, amsgrad, div, pct, lin_end, tta, name, cuda_id, nb_exp) 66 for i in range(nb_exp): 67 log_msg(open(fname, 'a'), '\n' + f'Experiment {i+1}') ---> 68 main_train(lr, moms, wd, wd_loss, opt_fn, bs, cyc_len, beta2, amsgrad, div, pct, lin_end, tta, fname) 69 70 if name == 'main': fire.Fire(train_lm)

/usr/local/lib/python3.6/dist-packages/Adam-experiments/train_cifar10.py in main_train(lr, moms, wd, wd_loss, opt_fn, bs, cyc_len, beta2, amsgrad, div, pct, lin_end, tta, fname) 39 nbs = [cyc_len (1-pct) / 2, cyc_len (1-pct) / 2, cyc_len * pct] 40 phases = get_phases(lr, moms, opt_fn, div, list(nbs), wd, lin_end, wd_loss) ---> 41 learn.fit_opt_sched(phases, callbacks=[LogResults(learn, fname)]) 42 if tta: 43 preds, targs = learn.TTA()

/usr/local/lib/python3.6/dist-packages/fastai/learner.py in fit_opt_sched(self, phases, cycle_save_name, best_save_name, stop_div, data_list, callbacks, cut, use_swa, swa_start, swa_eval_freq, *kwargs) 436 if data_list is None: data_list=[] 437 if callbacks is None: callbacks=[] --> 438 layer_opt = LayerOptimizer(phases[0].opt_fn, self.get_layer_groups(), 1e-2, phases[0].wds) 439 if len(data_list) == 0: nb_batches = [len(self.data.trn_dl)] len(phases) 440 else: nb_batches = [len(data.trn_dl) for data in data_list]

/usr/local/lib/python3.6/dist-packages/fastai/layer_optimizer.py in init(self, opt_fn, layer_groups, lrs, wds) 13 wds = listify(wds, layer_groups) 14 self.layer_groups,self.lrs,self.wds = layer_groups,lrs,wds ---> 15 self.opt = opt_fn(self.opt_params()) 16 17 def opt_params(self):

TypeError: init() got an unexpected keyword argument 'amsgrad'

sgugger commented 6 years ago

You should upgrade pytorch to 0.4.0 to get amsgrad, it's not implemented in the earliest versions.