search

CentML / centml-python-client

Apache License 2.0
1 stars 0 forks source link

Tests failing on local #73

Open yaxan opened 1 month ago

yaxan commented 1 month ago

Two tests fail on local pytest run, but not in github actions.

Logs:

================================================================================================ FAILURES =================================================================================================
______________________________________________________________________________ TestRemoteCompilation_1.test_compiled_cached _______________________________________________________________________________

self = <tests.test_backend.TestRemoteCompilation_1 testMethod=test_compiled_cached>, mock_load = <MagicMock name='load' id='139762224172864'>

    @patch("os.path.isfile", new=lambda x: True)
    @patch("centml.compiler.backend.Runner._get_model_id", new=lambda x: "1234")
    @patch("centml.compiler.backend.torch.load")
    def test_compiled_cached(self, mock_load):
        mock_load.return_value = MagicMock()
>       self.call_remote_compilation()

tests/test_backend.py:245: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_backend.py:236: in call_remote_compilation
    compiled_model(self.inputs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:451: in _fn
    return fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py:1245: in forward
    outputs = self.bert(
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:921: in catch_errors
    return callback(frame, cache_entry, hooks, frame_state, skip=1)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:786: in _convert_frame
    result = inner_convert(
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:400: in _convert_frame_assert
    return _compile(
/usr/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:676: in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:262: in time_wrapper
    r = func(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:535: in compile_inner
    out_code = transform_code_object(code, transform)
venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py:1036: in transform_code_object
    transformations(instructions, code_options)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:165: in _fn
    return fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:500: in transform
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2149: in run
    super().run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1219: in CALL_FUNCTION
    self.call_function(fn, args, {})
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py:336: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1260: in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:335: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:289: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:90: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1219: in CALL_FUNCTION
    self.call_function(fn, args, {})
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py:336: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1260: in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:335: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:289: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:90: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1219: in CALL_FUNCTION
    self.call_function(fn, args, {})
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py:336: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1260: in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:335: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:289: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:90: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1219: in CALL_FUNCTION
    self.call_function(fn, args, {})
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py:309: in call_function
    return wrap_fx_proxy(
venv/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py:1330: in wrap_fx_proxy
    return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py:1415: in wrap_fx_proxy_cls
    example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True)
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1714: in get_fake_value
    raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1656: in get_fake_value
    ret_val = wrap_fake_exception(
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1190: in wrap_fake_exception
    return fn()
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1657: in <lambda>
    lambda: run_node(tx.output, node, args, kwargs, nnmodule)
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1782: in run_node
    raise RuntimeError(make_error_message(e)).with_traceback(
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1769: in run_node
    return nnmodule(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/linear.py:116: in forward
    return F.linear(input, self.weight, self.bias)
venv/lib/python3.10/site-packages/torch/utils/_stats.py:20: in wrapper
    return fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:896: in __torch_dispatch__
    return self.dispatch(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:1241: in dispatch
    return self._cached_dispatch_impl(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:966: in _cached_dispatch_impl
    output = self._dispatch_impl(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:1393: in _dispatch_impl
    return decomposition_table[func](*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py:252: in _fn
    result = fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_decomp/decompositions.py:72: in inner
    r = f(*tree_map(increase_prec, args), **tree_map(increase_prec, kwargs))
venv/lib/python3.10/site-packages/torch/_decomp/decompositions.py:1421: in addmm
    out = alpha * torch.mm(mat1, mat2)
venv/lib/python3.10/site-packages/torch/utils/_stats.py:20: in wrapper
    return fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:896: in __torch_dispatch__
    return self.dispatch(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:1241: in dispatch
    return self._cached_dispatch_impl(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:966: in _cached_dispatch_impl
    output = self._dispatch_impl(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:1458: in _dispatch_impl
    r = func(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_ops.py:594: in __call__
    return self_._op(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py:252: in _fn
    result = fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_meta_registrations.py:2014: in meta_mm
    torch._check(
venv/lib/python3.10/site-packages/torch/__init__.py:1140: in _check
    _check_with(RuntimeError, cond, message)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

error_type = <class 'RuntimeError'>, cond = False, message = <function meta_mm.<locals>.<lambda> at 0x7f1ce63d5360>

    def _check_with(error_type, cond: Union[builtins.bool, SymBool], message: Callable[[], str]):  # noqa: F811
        if not isinstance(cond, (builtins.bool, torch.SymBool)):
            raise TypeError(f'cond must be a bool, but got {type(cond)}')

        from torch.fx.experimental.symbolic_shapes import expect_true
        if expect_true(cond):
            return

        # error_type must be a subclass of Exception and not subclass of Warning
        assert issubclass(error_type, Exception) and not issubclass(error_type, Warning)

        if message is None:
            message_evaluated = (
                'Expected cond to be True, but got False. (Could this error '
                'message be improved? If so, please report an enhancement request '
                'to PyTorch.)')

        else:
            if not callable(message):
                raise TypeError('message must be a callable')

            message_evaluated = str(message())

>       raise error_type(message_evaluated)
E       torch._dynamo.exc.TorchRuntimeError: Failed running call_module L__self___cls_predictions_transform_dense(*(FakeTensor(..., size=(1, 512, 30522), grad_fn=<AsStridedBackward0>),), **{}):
E       a and b must have same reduction dim, but got [512, 30522] X [768, 768].
E       
E       from user code:
E          File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1258, in torch_dynamo_resume_in_forward_at_1245
E           prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 818, in forward
E           prediction_scores = self.predictions(sequence_output)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 786, in forward
E           hidden_states = self.transform(hidden_states)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 762, in forward
E           hidden_states = self.dense(hidden_states)
E       
E       Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
E       
E       
E       You can suppress this exception and fall back to eager by setting:
E           import torch._dynamo
E           torch._dynamo.config.suppress_errors = True

venv/lib/python3.10/site-packages/torch/__init__.py:1123: TorchRuntimeError
_________________________________________________________________________ TestRemoteCompilation_1.test_compiled_return_not_cached _________________________________________________________________________

self = <tests.test_backend.TestRemoteCompilation_1 testMethod=test_compiled_return_not_cached>, mock_status = <MagicMock name='_wait_for_status' id='139762016587824'>
mock_download = <MagicMock name='_download_model' id='139762016592720'>

    @patch('os.path.isfile', new=lambda x: False)
    @patch("centml.compiler.backend.Runner._get_model_id", new=lambda x: "1234")
    @patch('centml.compiler.backend.Runner._download_model')
    @patch('centml.compiler.backend.Runner._wait_for_status')
    def test_compiled_return_not_cached(self, mock_status, mock_download):
        mock_status.return_value = True
        mock_download.return_value = MagicMock()

>       self.call_remote_compilation()

tests/test_backend.py:256: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_backend.py:236: in call_remote_compilation
    compiled_model(self.inputs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py:451: in _fn
    return fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py:1245: in forward
    outputs = self.bert(
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:921: in catch_errors
    return callback(frame, cache_entry, hooks, frame_state, skip=1)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:786: in _convert_frame
    result = inner_convert(
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:400: in _convert_frame_assert
    return _compile(
/usr/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:676: in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:262: in time_wrapper
    r = func(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:535: in compile_inner
    out_code = transform_code_object(code, transform)
venv/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py:1036: in transform_code_object
    transformations(instructions, code_options)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:165: in _fn
    return fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py:500: in transform
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2149: in run
    super().run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1219: in CALL_FUNCTION
    self.call_function(fn, args, {})
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py:336: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1260: in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:335: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:289: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:90: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1219: in CALL_FUNCTION
    self.call_function(fn, args, {})
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py:336: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1260: in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:335: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:289: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:90: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1219: in CALL_FUNCTION
    self.call_function(fn, args, {})
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py:336: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1260: in CALL_FUNCTION_EX
    self.call_function(fn, argsvars.items, kwargsvars)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:335: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:289: in call_function
    return super().call_function(tx, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/functions.py:90: in call_function
    return tx.inline_user_function_return(
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:680: in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2285: in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:2399: in inline_call_
    tracer.run()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:810: in run
    and self.step()
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:773: in step
    getattr(self, inst.opname)(inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:489: in wrapper
    return inner_fn(self, inst)
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:1219: in CALL_FUNCTION
    self.call_function(fn, args, {})
venv/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py:674: in call_function
    self.push(fn.call_function(self, args, kwargs))
venv/lib/python3.10/site-packages/torch/_dynamo/variables/nn_module.py:309: in call_function
    return wrap_fx_proxy(
venv/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py:1330: in wrap_fx_proxy
    return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs)
venv/lib/python3.10/site-packages/torch/_dynamo/variables/builder.py:1415: in wrap_fx_proxy_cls
    example_value = get_fake_value(proxy.node, tx, allow_non_graph_fake=True)
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1714: in get_fake_value
    raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1656: in get_fake_value
    ret_val = wrap_fake_exception(
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1190: in wrap_fake_exception
    return fn()
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1657: in <lambda>
    lambda: run_node(tx.output, node, args, kwargs, nnmodule)
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1782: in run_node
    raise RuntimeError(make_error_message(e)).with_traceback(
venv/lib/python3.10/site-packages/torch/_dynamo/utils.py:1769: in run_node
    return nnmodule(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1532: in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1541: in _call_impl
    return forward_call(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/nn/modules/linear.py:116: in forward
    return F.linear(input, self.weight, self.bias)
venv/lib/python3.10/site-packages/torch/utils/_stats.py:20: in wrapper
    return fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:896: in __torch_dispatch__
    return self.dispatch(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:1241: in dispatch
    return self._cached_dispatch_impl(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:966: in _cached_dispatch_impl
    output = self._dispatch_impl(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:1393: in _dispatch_impl
    return decomposition_table[func](*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py:252: in _fn
    result = fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_decomp/decompositions.py:72: in inner
    r = f(*tree_map(increase_prec, args), **tree_map(increase_prec, kwargs))
venv/lib/python3.10/site-packages/torch/_decomp/decompositions.py:1421: in addmm
    out = alpha * torch.mm(mat1, mat2)
venv/lib/python3.10/site-packages/torch/utils/_stats.py:20: in wrapper
    return fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:896: in __torch_dispatch__
    return self.dispatch(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:1241: in dispatch
    return self._cached_dispatch_impl(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:966: in _cached_dispatch_impl
    output = self._dispatch_impl(func, types, args, kwargs)
venv/lib/python3.10/site-packages/torch/_subclasses/fake_tensor.py:1458: in _dispatch_impl
    r = func(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_ops.py:594: in __call__
    return self_._op(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_prims_common/wrappers.py:252: in _fn
    result = fn(*args, **kwargs)
venv/lib/python3.10/site-packages/torch/_meta_registrations.py:2014: in meta_mm
    torch._check(
venv/lib/python3.10/site-packages/torch/__init__.py:1140: in _check
    _check_with(RuntimeError, cond, message)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

error_type = <class 'RuntimeError'>, cond = False, message = <function meta_mm.<locals>.<lambda> at 0x7f1ce65cb130>

    def _check_with(error_type, cond: Union[builtins.bool, SymBool], message: Callable[[], str]):  # noqa: F811
        if not isinstance(cond, (builtins.bool, torch.SymBool)):
            raise TypeError(f'cond must be a bool, but got {type(cond)}')

        from torch.fx.experimental.symbolic_shapes import expect_true
        if expect_true(cond):
            return

        # error_type must be a subclass of Exception and not subclass of Warning
        assert issubclass(error_type, Exception) and not issubclass(error_type, Warning)

        if message is None:
            message_evaluated = (
                'Expected cond to be True, but got False. (Could this error '
                'message be improved? If so, please report an enhancement request '
                'to PyTorch.)')

        else:
            if not callable(message):
                raise TypeError('message must be a callable')

            message_evaluated = str(message())

>       raise error_type(message_evaluated)
E       torch._dynamo.exc.TorchRuntimeError: Failed running call_module L__self___cls_predictions_transform_dense(*(FakeTensor(..., size=(1, 512, 30522), grad_fn=<AsStridedBackward0>),), **{}):
E       a and b must have same reduction dim, but got [512, 30522] X [768, 768].
E       
E       from user code:
E          File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1258, in torch_dynamo_resume_in_forward_at_1245
E           prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 818, in forward
E           prediction_scores = self.predictions(sequence_output)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 786, in forward
E           hidden_states = self.transform(hidden_states)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
E           return forward_call(*args, **kwargs)
E         File "/home/ubuntu/centml-python-client/venv/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 762, in forward
E           hidden_states = self.dense(hidden_states)
E       
E       Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
E       
E       
E       You can suppress this exception and fall back to eager by setting:
E           import torch._dynamo
E           torch._dynamo.config.suppress_errors = True

venv/lib/python3.10/site-packages/torch/__init__.py:1123: TorchRuntimeError
============================================================================================ warnings summary =============================================================================================
tests/test_server.py:48
  /home/ubuntu/centml-python-client/tests/test_server.py:48: PytestUnknownMarkWarning: Unknown pytest.mark.gpu - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html
    @pytest.mark.gpu

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================================================================= short test summary info =========================================================================================
FAILED tests/test_backend.py::TestRemoteCompilation_1::test_compiled_cached - torch._dynamo.exc.TorchRuntimeError: Failed running call_module L__self___cls_predictions_transform_dense(*(FakeTensor(..., size=(1, 512, 30522), grad_fn=<AsStridedBackward0>),), **{}):
FAILED tests/test_backend.py::TestRemoteCompilation_1::test_compiled_return_not_cached - torch._dynamo.exc.TorchRuntimeError: Failed running call_module L__self___cls_predictions_transform_dense(*(FakeTensor(..., size=(1, 512, 30522), grad_fn=<AsStridedBackward0>),), **{}):
================================================================================ 2 failed, 29 passed, 1 warning in 31.91s =================================================================================

┆Issue is synchronized with this Notion page by Unito

anandj91 commented 3 weeks ago

@yaxan this is fixed right? after you set the torch version to 2.3.1.

yaxan commented 3 weeks ago

@yaxan this is fixed right? after you set the torch version to 2.3.1.

No, the tests that were failing due to the torch version are different tests. All tests pass on github actions now, but for some reason two different tests fail when running pytest locally. @destefy might have more insight.