CentML / centml-python-client

Apache License 2.0
1 stars 0 forks source link

Upgrade pytorch's minimum version #66

Closed destefy closed 4 months ago

destefy commented 4 months ago

When I try to compile with pytorch 2.0.0, I am unable to torch.save the model's inputs. I get the following error

Traceback (most recent call last):
  File "/home/ubuntu/local_compile/centml-python-client/venv-torch/lib/python3.10/site-packages/centml/compiler/backend.py", line 63, in _serialize_model_and_inputs
    torch.save(self.inputs, self.serialized_input_path, pickle_protocol=config_instance.PICKLE_PROTOCOL)
  File "/home/ubuntu/local_compile/centml-python-client/venv-torch/lib/python3.10/site-packages/torch/serialization.py", line 441, in save
    _save(obj, opened_zipfile, pickle_module, pickle_protocol)
  File "/home/ubuntu/local_compile/centml-python-client/venv-torch/lib/python3.10/site-packages/torch/serialization.py", line 653, in _save
    pickler.dump(obj)
AttributeError: Can't pickle local object 'WeakValueDictionary.__init__.<locals>.remove'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/home/ubuntu/local_compile/centml-python-client/venv-torch/lib/python3.10/site-packages/centml/compiler/backend.py", line 147, in remote_compilation
    self._serialize_model_and_inputs()
  File "/home/ubuntu/local_compile/centml-python-client/venv-torch/lib/python3.10/site-packages/centml/compiler/backend.py", line 65, in _serialize_model_and_inputs
    raise Exception(f"Failed to save module or inputs with torch.save: {e}") from e
Exception: Failed to save module or inputs with torch.save: Can't pickle local object 'WeakValueDictionary.__init__.<locals>.remove'

Without having looked into it too deeply, this seems to be because in 2.0.0, the tensor gets turned into an object of type torch._subclasses.fake_tensor.FakeTensor.FakeTensor when passed to the backend, while 2.1.0 keeps it as just a torch.Tensor

anandj91 commented 4 months ago

@yaxan was getting similar issue in the nightly version as well.

yaxan commented 4 months ago

This issue exists in the latest nightly release as well when running torch.save on inputs containing SymInts