search

Lightning-AI / lightning-thunder

Make PyTorch models up to 40% faster! Thunder is a source to source compiler for PyTorch. It enables using different hardware executors at once; across one or thousands of GPUs.
Apache License 2.0
1.12k stars 69 forks source link

CI failure in test_core.py::test_traceback #844

Open t-vi opened 1 month ago

t-vi commented 1 month ago 
FAILED thunder/tests/test_core.py::test_traceback - AssertionError: assert 'torch.neg' in '        return func(*args, **kwargs)'
 +  where '        return func(*args, **kwargs)' = str(<_pytest._code.source.Source object at 0x7f5b48703d90>)
 +    where <_pytest._code.source.Source object at 0x7f5b48703d90> = <TracebackEntry /usr/local/lib/python3.10/dist-packages/torch/utils/_device.py:78>.statement

This could actually be a generator-context-manager of an earlier test not cleaning up properly due to #815 , but this is speculation at this point.

cc @apaz-cli @borda

t-vi commented 1 month ago

@kshitij12345 so in #849 I'm introducing a "hard cleanup" that will make the CI failure go away in two of the tests of #820 . The gist of the problem seems to be how PyTorch implements get_default_device. Maybe we need to reach into the internals of PyTorch ourselves there rather than relying on it or something. I'm not sure that this happens on all Python versions.

Would you have a clue what's going on / could you take a look, please?

kshitij12345 commented 1 month ago

Thanks for the ping. I am not sure what is happening here, will have a look :)