search

ContinualAI / avalanche

Avalanche: an End-to-End Library for Continual Learning based on PyTorch.
http://avalanche.continualai.org
MIT License
1.78k stars 291 forks source link

Checkpointing issues in 0.4.0 #1549

Open TuKo opened 10 months ago

TuKo commented 10 months ago

When trying to save a checkpoint the, torch.save(checkpoint_data, fname, pickle_module=dill) call in save_checkpoint() function fails. It complains about a recursive definition when calling torch.save() inside the respective function.

Additional context: I've followed the checkpoint example with the few changes below. The model is a CNN model (inheriting from DynamicModule) with an IncrementalClassifier as the output layer. This is in order to do class-incremental learning. Starts with 2 output classes and adapts in the train-eval loop. The dataset is MNIST loaded from torchvision (not avalanche) and the benchmark is created with nc_benchmark with 10 experiences and task_labels=False. Only ToTensor() transform is used for the data. The strategy is Naive. I'm using a ReplayBuffer, which I try removing it but still fails. The evaluation plugin has the accuracy_metrics, the forgetting_metrics, and an interactive logger. The main loop calls adaptation(), train(), eval() in that order.

System: Mac 2023 with M2

Versions: 

avalanche-lib 0.4.0
torch 2.1.1 
torchvision 0.16.1
python 3.10.13
/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py:412: PicklingWarning: Cannot locate reference to <class 'typing.ParamSpec'>.
  StockPickler.save(self, obj, save_persistent_id)
/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py:412: PicklingWarning: Cannot pickle <class 'typing.ParamSpec'>: typing.ParamSpec has recursive self-references that trigger a RecursionError.
  StockPickler.save(self, obj, save_persistent_id)
Traceback (most recent call last):
  File "/Users/me/projects/cl/train.py", line 172, in <module>
    main(args)
  File "/Users/me/projects/cl/train.py", line 144, in main
    save_checkpoint(strategy, filename)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/avalanche/training/checkpoint.py", line 88, in save_checkpoint
    torch.save(checkpoint_data, fname, pickle_module=dill)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/torch/serialization.py", line 619, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol, _disable_byteorder_record)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/torch/serialization.py", line 831, in _save
    pickler.dump(obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 418, in dump
    StockPickler.dump(self, obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 487, in dump
    self.save(obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 717, in save_reduce
    save(state)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 687, in save_reduce
    save(cls)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1093, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 887, in save_tuple
    save(element)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1107, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1812, in save_type
    _save_with_postproc(pickler, (_create_type, (
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1093, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 902, in save_tuple
    save(element)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1212, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1990, in save_function
    StockPickler.save_global(pickler, obj, name=name)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 1100, in save_global
    self.save_reduce(getattr, (parent, lastname))
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 692, in save_reduce
    save(args)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 887, in save_tuple
    save(element)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 412, in save
    StockPickler.save(self, obj, save_persistent_id)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/site-packages/dill/_dill.py", line 1849, in save_type
    StockPickler.save_global(pickler, obj, name=obj_name)
  File "/Users/me/anaconda3/envs/avalanche/lib/python3.10/pickle.py", line 1076, in save_global
    raise PicklingError(
_pickle.PicklingError: Can't pickle <class 'typing.ParamSpec'>: it's not the same object as typing.ParamSpec
ghost commented 10 months ago

2(1d1ts1)/1^-b^^-2/9 + ddx(Ass.V).dmb R2 = rm^2 + 2/1 (oh^-eipi^n2-2mf.rsc n^3x.u'/ r^2, 64pi rnt?d3/4.int -L^2 = 0rc . g3/2pi / Am^-pu.u.lk z = r^2 v^2 - 2.zip 3+1/2Nmf = -Aq' . 1/2hco.dnc Trc . dt^2 = oH OH = 5Tu' . 3/5q^2 dryHe = Frv^vu^2 10KT = / . 2/1 Api/-pu.ucc DM = ds^5/4 - pu^-2mû'. -hu! arcsen^2 = Apu'-mf'/1 5/0- . Mo - ddt^2E = GEo . Q^nk . pEc - Ec2 dt = 3/2j - dc2 . t^h + KPR+-4/27 mu^2 = -t^2 + arcsen r^216pirsc^-2/1 s = -2/1 rsc^2 - TKqu' +DEc o = -pmiEc't^3/2 a = d-pmu . QPRRct q = +- -2/1dp (-R) cos159 = ^8pi^KPR 24/7 + 2/3 pidrt^2ms / r^2arsen3/0 FRmo = Amu + KP445 -r^2h gcr =dt^3 . -1/2dt

AntonioCarta commented 9 months ago

@lrzpellegrini do you know if there are any workaround for this issue?