aidan-jackson-data commented 1 year ago

🐛 Describe the bug

The second to last code example shown here is a demonstration of using a Hugging Face model with PyTorch 2.0's compile feature.

When copying the example, having followed the PyTorch 2.0 installation instructions in a new virtual environment, an error is encountered. When removing the model compilation line, no error is encountered.

Error logs

AttributeError Traceback (most recent call last) File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/utils.py:1052, in run_node(output_graph, node, args, kwargs, nnmodule) 1051 if op == "call_function": -> 1052 return node.target(*args, **kwargs) 1053 elif op == "call_method":

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_tensor.py:40, in _handle_torch_function_and_wrap_type_error_to_not_implemented..wrapped(*args, kwargs) 39 return handle_torch_function(wrapped, args, *args, *kwargs) ---> 40 return f(args, kwargs) 41 except TypeError:

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_tensor.py:840, in Tensor.rsub(self, other) 838 @_handle_torch_function_and_wrap_type_error_to_not_implemented 839 def rsub(self, other): --> 840 return _C._VariableFunctions.rsub(self, other)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_subclasses/fake_tensor.py:886, in FakeTensorMode.__torch_dispatch__(self, func, types, args, kwargs) 885 with self: --> 886 return decomposition_table[func](*args, **kwargs) 888 with self: 889 # Decomposes CompositeImplicitAutograd ops

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_decomp/decompositions.py:1033, in rsub_Scalar(self, other, alpha) 1031 @register_decomposition(aten.rsub.Scalar) 1032 def rsub_Scalar(self: Tensor, other: float, alpha: float = 1) -> Tensor: -> 1033 return torch.sub(other, self, alpha=alpha)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_subclasses/fake_tensor.py:916, in FakeTensorMode.__torch_dispatch__(self, func, types, args, kwargs) 915 with in_kernel_invocation_manager(self): --> 916 r = func(*args, **kwargs) 917 except NotImplementedError as not_implemented_error: 918 # no meta kernel registered, fallback to kernel for the device

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_ops.py:284, in OpOverload.call(self, *args, kwargs) 283 def call(self, *args, *kwargs): --> 284 return self._op(args, kwargs or {})

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_prims_common/wrappers.py:209, in out_wrapper.._out_wrapper.._fn(out, *args, *kwargs) 207 kwargs[k] = out_attr --> 209 result = fn(args, **kwargs) 210 assert ( 211 isinstance(result, TensorLike) 212 and is_tensor 213 or isinstance(result, Tuple) # type: ignore[arg-type] 214 and len(result) == len(out_names) 215 )

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_prims_common/wrappers.py:119, in elementwise_type_promotion_wrapper.call.._fn(*args, kwargs) 117 bound.arguments.update(promoted_args) --> 119 result = fn(bound.arguments) 121 if isinstance(result, TensorLike):

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_refs/init.py:1587, in sub(a, b, alpha) 1585 b = prims.mul(b, alpha) -> 1587 return prims.sub(a, b)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_ops.py:284, in OpOverload.call(self, *args, kwargs) 283 def call(self, *args, *kwargs): --> 284 return self._op(args, kwargs or {})

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_prims/init.py:349, in _elementwise_meta(type_promotion, args_with_fixed_dtypes, args) 348 strides = utils.compute_elementwise_output_strides(args_) --> 349 shape = utils.extractshape(*args, allow_cpu_scalar_tensors=True) 351 # Acquires the dtype

AttributeError: module 'torch._prims.utils' has no attribute 'extract_shape'

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last) File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/utils.py:1011, in get_fake_value(node, tx) 1010 with tx.fake_mode, enable_python_dispatcher(): -> 1011 return wrap_fake_exception( 1012 lambda: run_node(tx.output, node, args, kwargs, nnmodule) 1013 ) 1014 except Unsupported:

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/utils.py:702, in wrap_fake_exception(fn) 701 try: --> 702 return fn() 703 except UnsupportedFakeTensorException as e:

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/utils.py:1012, in get_fake_value..() 1010 with tx.fake_mode, enable_python_dispatcher(): 1011 return wrap_fake_exception( -> 1012 lambda: run_node(tx.output, node, args, kwargs, nnmodule) 1013 ) 1014 except Unsupported:

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/utils.py:1061, in run_node(output_graph, node, args, kwargs, nnmodule) 1060 except Exception as e: -> 1061 raise RuntimeError( 1062 f"Failed running {op} {node.target}(*{args}, **{kwargs}):\n{e}\n(scroll up for backtrace)" 1063 ) from e 1064 raise AssertionError(op)

RuntimeError: Failed running call_function (*(1.0, FakeTensor(FakeTensor(..., device='meta', size=(1, 1, 1, 12)), cuda:0)), **{}): module 'torch._prims.utils' has no attribute 'extract_shape' (scroll up for backtrace)

The above exception was the direct cause of the following exception:

TorchRuntimeError Traceback (most recent call last) Cell In[3], line 4 2 text = "Replace me by any text you'd like." 3 encoded_input = tokenizer(text, return_tensors='pt').to(device="cuda:0") ----> 4 output = model(**encoded_input)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/nn/modules/module.py:1482, in Module._call_impl(self, *args, *kwargs) 1477 # If we don't have any hooks, we want to skip the rest of the logic in 1478 # this function, and just call forward. 1479 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks 1480 or _global_backward_pre_hooks or _global_backward_hooks 1481 or _global_forward_hooks or _global_forward_pre_hooks): -> 1482 return forward_call(args, **kwargs) 1483 # Do not call functions when jit is used 1484 full_backward_hooks, non_full_backward_hooks = [], []

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py:83, in OptimizedModule.forward(self, *args, kwargs) 82 def forward(self, *args, *kwargs): ---> 83 return self.dynamo_ctx(self._orig_mod.forward)(args, kwargs)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py:212, in _TorchDynamoContext.call.._fn(*args, *kwargs) 210 dynamic_ctx.enter() 211 try: --> 212 return fn(args, **kwargs) 213 finally: 214 set_eval_frame(prior)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py:333, in catch_errors_wrapper..catch_errors(frame, cache_size) 330 return hijacked_callback(frame, cache_size, hooks) 332 with compile_lock: --> 333 return callback(frame, cache_size, hooks)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py:480, in convert_frame.._convert_frame(frame, cache_size, hooks) 478 counters["frames"]["total"] += 1 479 try: --> 480 result = inner_convert(frame, cache_size, hooks) 481 counters["frames"]["ok"] += 1 482 return result

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py:103, in wrap_convert_context.._fn(*args, *kwargs) 101 torch.fx.graph_module._forward_from_src = fx_forward_from_src_skip_result 102 try: --> 103 return fn(args, **kwargs) 104 finally: 105 torch._C._set_grad_enabled(prior_grad_mode)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/utils.py:88, in dynamo_timed..time_wrapper(*args, *kwargs) 86 compilation_metrics[key] = [] 87 t0 = time.time() ---> 88 r = func(args, **kwargs) 89 latency = time.time() - t0 90 # print(f"Dynamo timer: key={key}, latency={latency:.2f} sec")

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py:339, in convert_frame_assert.._convert_frame_assert(frame, cache_size, hooks) 336 global initial_grad_state 337 initial_grad_state = torch.is_grad_enabled() --> 339 return _compile( 340 frame.f_code, 341 frame.f_globals, 342 frame.f_locals, 343 frame.f_builtins, 344 compiler_fn, 345 one_graph, 346 export, 347 hooks, 348 frame, 349 )

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py:400, in _compile(code, globals, locals, builtins, compiler_fn, one_graph, export, hooks, frame) 398 for attempt in itertools.count(): 399 try: --> 400 out_code = transform_code_object(code, transform) 401 orig_code_map[out_code] = code 402 break

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py:341, in transform_code_object(code, transformations, safe) 338 instructions = cleaned_instructions(code, safe) 339 propagate_line_nums(instructions) --> 341 transformations(instructions, code_options) 343 fix_vars(instructions, code_options) 345 dirty = True

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py:387, in _compile..transform(instructions, code_options) 374 nonlocal output 375 tracer = InstructionTranslator( 376 instructions, 377 code, (...) 385 mutated_closure_cell_contents, 386 ) --> 387 tracer.run() 388 output = tracer.output 389 assert output is not None

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:1684, in InstructionTranslator.run(self) 1682 def run(self): 1683 _step_logger()(logging.INFO, f"torchdynamo start tracing {self.f_code.co_name}") -> 1684 super().run()

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:538, in InstructionTranslatorBase.run(self) 533 try: 534 self.output.push_tx(self) 535 while ( 536 self.instruction_pointer is not None 537 and not self.output.should_exit --> 538 and self.step() 539 ): 540 pass 541 except BackendCompilerFailed:

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:501, in InstructionTranslatorBase.step(self) 499 if not hasattr(self, inst.opname): 500 unimplemented(f"missing: {inst.opname}") --> 501 getattr(self, inst.opname)(inst) 503 return inst.opname != "RETURN_VALUE" 504 except BackendCompilerFailed:

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:307, in break_graph_if_unsupported..decorator..wrapper(self, inst) 305 reason = None 306 try: --> 307 return inner_fn(self, inst) 308 except Unsupported as excp: 309 if self.has_backedge():

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:966, in InstructionTranslatorBase.CALL_FUNCTION(self, inst) 964 args = self.popn(inst.argval) 965 fn = self.pop() --> 966 self.call_function(fn, args, {})

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:435, in InstructionTranslatorBase.call_function(self, fn, args, kwargs) 430 assert isinstance(kwargs, dict) 431 assert all( 432 isinstance(x, VariableTracker) 433 for x in itertools.chain(args, kwargs.values()) 434 ) --> 435 self.push(fn.call_function(self, args, kwargs))

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py:244, in UserMethodVariable.call_function(self, tx, args, kwargs) 236 if ( 237 isinstance(self.obj, variables.NNModuleVariable) 238 and getattr(self.fn, "module", "").startswith("torch.nn.") 239 or self.is_constant 240 ): 241 return self.obj.call_method( 242 tx, self.fn.name, args, kwargs, constant=self.is_constant 243 ).add_options(self) --> 244 return super().call_function(tx, args, kwargs)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py:214, in UserFunctionVariable.call_function(self, tx, args, kwargs) 209 options = VariableTracker.propagate(self, args, kwargs.values()) 210 return invoke_and_store_as_constant( 211 tx, self.fn, self.get_name(), options, args, kwargs 212 ) --> 214 return super(UserFunctionVariable, self).call_function(tx, args, kwargs)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py:67, in BaseUserFunctionVariable.call_function(self, tx, args, kwargs) 64 def call_function( 65 self, tx, args: "List[VariableTracker]", kwargs: "Dict[str, VariableTracker]" 66 ) -> "VariableTracker": ---> 67 return tx.inline_user_function_return( 68 self, list(self.self_args()) + list(args), kwargs 69 )

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:471, in InstructionTranslatorBase.inline_user_function_return(self, fn, args, kwargs) 469 state = self.copy_graphstate() 470 try: --> 471 result = InliningInstructionTranslator.inline_call(self, fn, args, kwargs) 472 self.output.guards.update(fn.guards) 473 return result

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:1762, in InliningInstructionTranslator.inline_call(cls, parent, func, args, kwargs) 1759 @classmethod 1760 def inline_call(cls, parent, func, args, kwargs): 1761 with patch.dict(counters, {"unimplemented": counters["inline_call"]}): -> 1762 return cls.inlinecall(parent, func, args, kwargs)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:1817, in InliningInstructionTranslator.inlinecall(parent, func, args, kwargs) 1812 else: 1813 tracer = InliningInstructionTranslator( 1814 parent, code, sub_locals, parent.symbolic_globals, closure_cells, func 1815 ) -> 1817 tracer.run() 1818 assert tracer.symbolic_result is not None 1819 func.export_freevars(parent, tracer)

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:538, in InstructionTranslatorBase.run(self) 533 try: 534 self.output.push_tx(self) 535 while ( 536 self.instruction_pointer is not None 537 and not self.output.should_exit --> 538 and self.step() 539 ): 540 pass 541 except BackendCompilerFailed:

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:501, in InstructionTranslatorBase.step(self) 499 if not hasattr(self, inst.opname): 500 unimplemented(f"missing: {inst.opname}") --> 501 getattr(self, inst.opname)(inst) 503 return inst.opname != "RETURN_VALUE" 504 except BackendCompilerFailed:

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py:137, in stack_op..impl(self, inst) 135 @functools.wraps(fn) 136 def impl(self: "InstructionTranslatorBase", inst: Instruction): --> 137 self.push(fn_var.call_function(self, self.popn(nargs), {}))

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/variables/builtin.py:322, in BuiltinVariable.call_function(self, tx, args, kwargs) 318 if self.fn is operator.truediv and isinstance( 319 args[0], variables.UnspecializedPythonVariable 320 ): 321 args[0] = args[0].convert_to_constant(tx) --> 322 return wrap_fx_proxy(tx, proxy, **options) 324 except NotImplementedError: 325 unimplemented(f"partial tensor op: {self} {args} {kwargs}")

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/variables/builder.py:739, in wrap_fx_proxy(tx, proxy, example_value, options) 738 def wrap_fx_proxy(tx, proxy, example_value=None, options): --> 739 return wrap_fx_proxy_cls( 740 target_cls=TensorVariable, 741 tx=tx, 742 proxy=proxy, 743 example_value=example_value, 744 **options, 745 )

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/variables/builder.py:774, in wrap_fx_proxy_cls(target_cls, tx, proxy, example_value, ignore_subclass, **options) 772 with preserve_rng_state(): 773 if example_value is None: --> 774 example_value = get_fake_value(proxy.node, tx) 776 # Handle recursive calls here 777 elif isinstance(example_value, FakeTensor):

File ~/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/torch/_dynamo/utils.py:1031, in get_fake_value(node, tx) 1027 elif isinstance( 1028 cause, torch._subclasses.fake_tensor.DynamicOutputShapeException 1029 ): 1030 unimplemented(f"dynamic shape operator: {cause.func}") -> 1031 raise TorchRuntimeError() from e

TorchRuntimeError:

from user code: File "/home/ajax/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 987, in forward extended_attention_mask: torch.Tensor = self.get_extended_attention_mask(attention_mask, input_shape) File "/home/ajax/miniconda3/envs/movie-env-2/lib/python3.9/site-packages/transformers/modeling_utils.py", line 791, in get_extended_attention_mask extended_attention_mask = (1.0 - extended_attention_mask) * torch.finfo(dtype).min

Set torch._dynamo.config.verbose=True for more information

You can suppress this exception and fall back to eager by setting: torch._dynamo.config.suppress_errors = True

Minified repro

import torch
from transformers import BertTokenizer, BertModel
# Copy pasted from here https://huggingface.co/bert-base-uncased
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained("bert-base-uncased").to(device="cuda:0")
model = torch.compile(model) # This is the only line of code that we changed
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt').to(device="cuda:0")
output = model(**encoded_input)

Virtual Environment:

aidan-jackson-data commented 1 year ago

Given the nature of the error, and the virtual environment shown above, I believe the Issue is caused by the transformers package installing pytorch 1.12.1. The version installed from the instructions here results in the above package pytorch-triton 2.0.0+0d7e753227 being installed into the environment.

Is there any insight as to how the transformers package was installed in this example? Or how these conflicting versions should be resolved?

msaroufim commented 1 year ago

Easiest way is to install transformers and then pytorch nightlies. If that doesn't work for you, you can install pytorch nightlies first if you pip install transfomers --no-deps

aidan-jackson-data commented 1 year ago

Easiest way is to install transformers and then pytorch nightlies. If that doesn't work for you, you can install pytorch nightlies first if you pip install transfomers --no-deps

Thank you for the advice. After starting fresh and installing transformers without dependencies, and running the install commands a few times in row, it became functional.

Closing with this comment.

pytorch / torchdynamo

Unable to reporduce Hugging Face model example shown for PyTorch 2.0 #2019

🐛 Describe the bug

Error logs

Minified repro

Name Version Build Channel