Closed anijain2305 closed 2 years ago
Progress tracker:
NumpyVariables
doesn't support isinstance
(fixed by pytorch/torchdynamo#774). torch.ops.profiler._record_function_exit
returns a tensor to store underlying RecordFunction
, but dynamo can't handle this. Current we put this function into FX graph, but doesn't set its return value as one of the graph output. (https://github.com/pytorch/torchdynamo/pull/867)WithExitFunctionVariable
can't reconstruct ProfileRecordFunctionVariable
correctly.ProfileRecordFunctionVariable
misses with
context on the graph break instruction. I think we should refer how GradModeVariable
handling this case.FakeTensor
as TensorVariable
. https://github.com/pytorch/torchdynamo/pull/931aot_nop
and aot_nvfuser
work well. https://github.com/pytorch/torchdynamo/issues/953
This point has been raised multiple times. Our benchmarks today only focus on single iteration of fwd-bwd (and optimizers soon). But we lack a real training example, with loss going down.
We could start with Bert model and run for say 500 iterations - with both pre-training and fine-tuning batch sizes. Nvidia folks mentioned that this could reveal many bugs which are not visible in single iteration.