Torch probably needs to be less than 2.0

sgbaird commented 1 year ago

https://github.com/sparks-baird/matsci-opt-benchmarks/actions/runs/5193086890/jobs/9363184385

def test_matbench_metric_calculator():
        actual_param = userparam_to_crabnetparam(user_param, seed=50)
>       matbench_metric_calculator(actual_param, dummy=True)

tests/crabnet_hyperparameter_test.py:41: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.tox/default/lib/python3.10/site-packages/matsci_opt_benchmarks/crabnet_hyperparameter/utils/parameters.py:361: in matbench_metric_calculator
    cb.fit(train_df=train_df)
.tox/default/lib/python3.10/site-packages/crabnet/crabnet_.py:409: in fit
    self._train()
.tox/default/lib/python3.10/site-packages/crabnet/crabnet_.py:628: in _train
    self.optimizer.step()
.tox/default/lib/python3.10/site-packages/torch/optim/lr_scheduler.py:69: in wrapper
    return wrapped(*args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

args = (SWA (
Parameter Group 0
    betas: (0.89992, 0.7[49](https://github.com/sparks-baird/matsci-opt-benchmarks/actions/runs/5193086890/jobs/9363184385#step:5:50)95)
    eps: 1e-06
    initial_lr: 0.0001
    lookahead_alpha: 0.5
    lookahead_k: 6
    lookahead_step: 0
    lr: 0.0001
    n_avg: 0
    step_counter: 0
    weight_decay: 0.0
),)
kwargs = {}
self = SWA (
Parameter Group 0
    betas: (0.89992, 0.74995)
    eps: 1e-06
    initial_lr: 0.0001
    lookahead_alpha: 0.5
    lookahead_k: 6
    lookahead_step: 0
    lr: 0.0001
    n_avg: 0
    step_counter: 0
    weight_decay: 0.0
)
_ = [], profile_name = 'Optimizer.step#SWA.step'

    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        self, *_ = args
        profile_name = "Optimizer.step#{}.step".format(self.__class__.__name__)
        with torch.autograd.profiler.record_function(profile_name):
            # call optimizer step pre hooks
>           for pre_hook in chain(_global_optimizer_pre_hooks.values(), self._optimizer_step_pre_hooks.values()):
E           AttributeError: 'SWA' object has no attribute '_optimizer_step_pre_hooks'. Did you mean: '_optimizer_step_code'?

.tox/default/lib/python3.10/site-packages/torch/optim/optimizer.py:2[71](https://github.com/sparks-baird/matsci-opt-benchmarks/actions/runs/5193086890/jobs/9363184385#step:5:72): AttributeError

sgbaird commented 1 year ago

For me to address this error without forcing torch < 2.0, I will need to adjust CrabNet to use SWA from the official PyTorch implementation rather than the code that Anthony or Kaai copied (probably from a blog) when developing CrabNet.

Aside: I'm not sure how this will affect hyperparameters in the context of matsci-opt-benchmarks and crabnet-hyperparameter.

Official PyTorch SWA

PyTorch documentation for SWA
Official source code for SWA (optim/swa_utils.py)

Uncited source code within CrabNet

Maybe Lamb is from https://github.com/cybertronai/pytorch-lamb/blob/master/pytorch_lamb/lamb.py?

Maybe Lookahead is from https://github.com/lonePatient/lookahead_pytorch/blob/master/optimizer.py?

Places that require refactoring

... or at least a double check. This might not be comprehensive, but it's a rough idea.

cc @anthony-wang @Kaaiian

cseeg commented 1 year ago

What version of pytorch was usually used?

sgbaird commented 1 year ago

Probably just the latest version before 2.0. You shouldn't have to do anything fancy when you're installing locally on your computer, since it's already written into the latest requirements.

sgbaird commented 10 months ago

From https://github.com/sparks-baird/CrabNet/discussions/73#discussioncomment-7437336 by @DavidSiretMarques

I'm using pytorch for other things too, so that isn't an option...

That will make things tough. While not ideal, have you tried forcing installation of PyTorch < 2.0 and seeing if that breaks things in your other code?

Do you know the cause of the problem? I've dedicated some time to this, but only enough to identify the places that require refactoring and some potential solutions.

It was weird that in my old mac computer it worked and in my newer windows computer didn't..., they should have the exact same env with the same libraries... That's pretty interesting. I'm not sure why that would happen, though.

sparks-baird / CrabNet