search

nod-ai / SHARK-ModelDev

Unified compiler/runtime for interfacing with PyTorch Dynamo.
Apache License 2.0
92 stars 46 forks source link

Crash due to recent change in LLVM #851

Open pdhirajkumarprasad opened 1 day ago

pdhirajkumarprasad commented 1 day ago

for the give IR

module {
  func.func @torch_jit(  %arg2: !torch.vtensor<[1],si64>, %arg3: !torch.vtensor<[?,?,576],f32>) -> !torch.vtensor<[3,?,8,?,?],f32>  attributes {torch.onnx_meta.ir_version = 7 : si64, torch.onnx_meta.opset_version = 21 : si64, torch.onnx_meta.producer_name = "pytorch", torch.onnx_meta.producer_version = "1.12.1"} {
    %1 = torch.operator "onnx.Constant"() {torch.onnx.value = dense<-1> : tensor<1xsi64>} : () -> !torch.vtensor<[1],si64> 
    %2 = torch.operator "onnx.Concat"(%arg2, %arg2, %arg2, %arg2, %1) {torch.onnx.axis = 0 : si64} : (!torch.vtensor<[1],si64>, !torch.vtensor<[1],si64>, !torch.vtensor<[1],si64>, !torch.vtensor<[1],si64>, !torch.vtensor<[1],si64>) -> !torch.vtensor<[5],si64> 
    %3 = torch.operator "onnx.Reshape"(%arg3, %2) : (!torch.vtensor<[?,?,576],f32>, !torch.vtensor<[5],si64>) -> !torch.vtensor<[?,?,3,8,?],f32> 
    %4 = torch.operator "onnx.Transpose"(%3) {torch.onnx.perm = [2 : si64, 0 : si64, 3 : si64, 4 : si64, 1 : si64]} : (!torch.vtensor<[?,?,3,8,?],f32>) -> !torch.vtensor<[3,?,8,?,?],f32> 
    return %4 : !torch.vtensor<[3,?,8,?,?],f32>
  }
}

we are seeing crash

iree-compile: /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/include/mlir/IR/StorageUniquerSupport.h:180: static ConcreteT mlir::detail::StorageUserBase<mlir::RankedTensorType, mlir::TensorType, mlir::detail::RankedTensorTypeStorage, mlir::detail::TypeUniquer, mlir::ShapedType::Trait, mlir::ValueSemantics>::get(mlir::MLIRContext *, Args &&...) [ConcreteT = mlir::RankedTensorType, BaseT = mlir::TensorType, StorageT = mlir::detail::RankedTensorTypeStorage, UniquerT = mlir::detail::TypeUniquer, Traits = <mlir::ShapedType::Trait, mlir::ValueSemantics>, Args = <llvm::ArrayRef<long> &, mlir::Type &, mlir::Attribute &>]: Assertion `succeeded( ConcreteT::verifyInvariants(getDefaultDiagnosticEmitFn(ctx), args...))' failed.
Please report issues to https://github.com/iree-org/iree/issues and include the crash backtrace.
Stack dump:
0.  Program arguments: iree-compile --iree-input-type=auto --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=llvm-cpu --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false model.torch_onnx.mlir
 #0 0x00007ff5de9948f7 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/lib/Support/Unix/Signals.inc:723:13
 #1 0x00007ff5de992b30 llvm::sys::RunSignalHandlers() /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/lib/Support/Signals.cpp:106:18
 #2 0x00007ff5de994fba SignalHandler(int) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/lib/Support/Unix/Signals.inc:413:1

command to reproduce:

iree-compile --iree-input-type=auto --iree-vm-bytecode-module-output-format=flatbuffer-binary --iree-hal-target-backends=llvm-cpu --mlir-print-debuginfo --mlir-print-op-on-diagnostic=false t1.mlir
ScottTodd commented 1 day ago

Is there any more to that stack trace? That only shows the signal, not what caused it (beyond the assert)

pdhirajkumarprasad commented 1 day ago

@ScottTodd hee is complete stack trace

iree-compile: /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/include/mlir/IR/StorageUniquerSupport.h:180: static ConcreteT mlir::detail::StorageUserBase<mlir::RankedTensorType, mlir::TensorType, mlir::detail::RankedTensorTypeStorage, mlir::detail::TypeUniquer, mlir::ShapedType::Trait, mlir::ValueSemantics>::get(mlir::MLIRContext *, Args &&...) [ConcreteT = mlir::RankedTensorType, BaseT = mlir::TensorType, StorageT = mlir::detail::RankedTensorTypeStorage, UniquerT = mlir::detail::TypeUniquer, Traits = <mlir::ShapedType::Trait, mlir::ValueSemantics>, Args = <llvm::ArrayRef<long> &, mlir::Type &, mlir::Attribute &>]: Assertion `succeeded( ConcreteT::verifyInvariants(getDefaultDiagnosticEmitFn(ctx), args...))' failed.
Please report issues to https://github.com/iree-org/iree/issues and include the crash backtrace.
Stack dump:
0.  Program arguments: iree-compile t1.mlir
 #0 0x00007fcf07ead8f7 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/lib/Support/Unix/Signals.inc:723:13
 #1 0x00007fcf07eabb30 llvm::sys::RunSignalHandlers() /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/lib/Support/Signals.cpp:106:18
 #2 0x00007fcf07eadfba SignalHandler(int) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/lib/Support/Unix/Signals.inc:413:1
 #3 0x00007fcf015a2520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x00007fcf015f69fc __pthread_kill_implementation ./nptl/./nptl/pthread_kill.c:44:76
 #5 0x00007fcf015f69fc __pthread_kill_internal ./nptl/./nptl/pthread_kill.c:78:10
 #6 0x00007fcf015f69fc pthread_kill ./nptl/./nptl/pthread_kill.c:89:10
 #7 0x00007fcf015a2476 gsignal ./signal/../sysdeps/posix/raise.c:27:6
 #8 0x00007fcf015887f3 abort ./stdlib/./stdlib/abort.c:81:7
 #9 0x00007fcf0158871b _nl_load_domain ./intl/./intl/loadmsgcat.c:1177:9
#10 0x00007fcf01599e96 (/lib/x86_64-linux-gnu/libc.so.6+0x39e96)
#11 0x00007fcf07f4cf94 (/proj/xhdhdstaff6/dhirajp/localBuild/iree-build/lib/libIREECompiler.so+0x6578f94)
#12 0x00007fcf07f4ceb8 mlir::RankedTensorType::get(llvm::ArrayRef<long>, mlir::Type, mlir::Attribute) /proj/xhdhdstaff6/dhirajp/localBuild/iree-build/llvm-project/tools/mlir/include/mlir/IR/BuiltinTypes.cpp.inc:308:3
#13 0x00007fcf0bc79703 llvm::FailureOr<mlir::RankedTensorType> mlir::iree_compiler::IREE::Flow::(anonymous namespace)::canonicalizeSubViewParts<mlir::iree_compiler::IREE::Flow::DispatchTensorStoreOp>(mlir::iree_compiler::IREE::Flow::DispatchTensorStoreOp, mlir::RankedTensorType, llvm::SmallVector<mlir::OpFoldResult, 6u>&, llvm::SmallVector<mlir::OpFoldResult, 6u>&, llvm::SmallVector<mlir::OpFoldResult, 6u>&) /proj/xhdhdstaff6/dhirajp/localBuild/iree/compiler/src/iree/compiler/Dialect/Flow/IR/FlowOpFolders.cpp:558:7
#14 0x00007fcf0bc79703 mlir::iree_compiler::IREE::Flow::(anonymous namespace)::DispatchTensorStoreOpWithOffsetSizesAndStridesConstantArgumentFolder::matchAndRewrite(mlir::iree_compiler::IREE::Flow::DispatchTensorStoreOp, mlir::PatternRewriter&) const /proj/xhdhdstaff6/dhirajp/localBuild/iree/compiler/src/iree/compiler/Dialect/Flow/IR/FlowOpFolders.cpp:670:25
#15 0x00007fcf0c2ba3de mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<llvm::LogicalResult (mlir::Pattern const&)>)::$_2::operator()() const /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Rewrite/PatternApplicator.cpp:212:13
#16 0x00007fcf0c2ba3de void llvm::function_ref<void ()>::callback_fn<mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<llvm::LogicalResult (mlir::Pattern const&)>)::$_2>(long) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#17 0x00007fcf0c2b741f mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<llvm::LogicalResult (mlir::Pattern const&)>) /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Rewrite/PatternApplicator.cpp:233:9
#18 0x00007fcf0c299eae (anonymous namespace)::GreedyPatternRewriteDriver::processWorklist() /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:0:0
#19 0x00007fcf0c296a50 (anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) &&::$_8::operator()() const /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:874:11
#20 0x00007fcf0c296a50 void llvm::function_ref<void ()>::callback_fn<(anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) &&::$_8>(long) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#21 0x00007fcf0c296a50 llvm::function_ref<void ()>::operator()() const /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#22 0x00007fcf0c296a50 void mlir::MLIRContext::executeAction<(anonymous namespace)::GreedyPatternRewriteIteration, long&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, long&) /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275:7
#23 0x00007fcf0c296a50 (anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) && /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:872:10
#24 0x00007fcf0c296a50 mlir::applyPatternsAndFoldGreedily(mlir::Region&, mlir::FrozenRewritePatternSet const&, mlir::GreedyRewriteConfig, bool*) /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:919:47
#25 0x00007fcf09ccb4a1 llvm::LogicalResult::failed() const /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#26 0x00007fcf09ccb4a1 mlir::applyPatternsAndFoldGreedily(mlir::Operation*, mlir::FrozenRewritePatternSet const&, mlir::GreedyRewriteConfig, bool*) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/mlir/include/mlir/Transforms/GreedyPatternRewriteDriver.h:160:14
#27 0x00007fcf09ccb4a1 mlir::iree_compiler::IREE::Util::ApplyPatternsPass::runOnOperation() /proj/xhdhdstaff6/dhirajp/localBuild/iree/compiler/src/iree/compiler/Dialect/Util/Transforms/ApplyPatterns.cpp:41:16
#28 0x00007fcf0805b185 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7::operator()() const /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:0:17
#29 0x00007fcf0805b185 void llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_7>(long) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45:12
#30 0x00007fcf0805b185 llvm::function_ref<void ()>::operator()() const /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68:12
#31 0x00007fcf0805b185 void mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275:7
#32 0x00007fcf0805b185 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:520:21
#33 0x00007fcf0805b8f8 llvm::LogicalResult::failed() const /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#34 0x00007fcf0805b8f8 llvm::failed(llvm::LogicalResult) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#35 0x00007fcf0805b8f8 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:592:9
#36 0x00007fcf0805dc19 mlir::PassManager::run(mlir::Operation*) /proj/rdi/staff/dhirajp/localBuild/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:0:0
#37 0x00007fcf07e00f30 llvm::LogicalResult::failed() const /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:43:43
#38 0x00007fcf07e00f30 llvm::failed(llvm::LogicalResult) /proj/xhdhdstaff6/dhirajp/localBuild/iree/third_party/llvm-project/llvm/include/llvm/Support/LogicalResult.h:71:58
#39 0x00007fcf07e00f30 mlir::iree_compiler::embed::(anonymous namespace)::Invocation::runPipeline(iree_compiler_pipeline_t) /proj/xhdhdstaff6/dhirajp/localBuild/iree/compiler/src/iree/compiler/API/Internal/CompilerDriver.cpp:997:7
#40 0x00007fcf07e00f30 ireeCompilerInvocationPipeline /proj/xhdhdstaff6/dhirajp/localBuild/iree/compiler/src/iree/compiler/API/Internal/CompilerDriver.cpp:1432:23
#41 0x00007fcf080214c8 mlir::iree_compiler::runIreecMain(int, char**)::$_2::operator()(iree_compiler_source_t*) const /proj/xhdhdstaff6/dhirajp/localBuild/iree/compiler/src/iree/compiler/Tools/iree_compile_lib.cc:254:11
#42 0x00007fcf08020d01 mlir::iree_compiler::runIreecMain(int, char**) /proj/xhdhdstaff6/dhirajp/localBuild/iree/compiler/src/iree/compiler/Tools/iree_compile_lib.cc:0:10
#43 0x00007fcf01589d90 __libc_start_call_main ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#44 0x00007fcf01589e40 call_init ./csu/../csu/libc-start.c:128:20
#45 0x00007fcf01589e40 __libc_start_main ./csu/../csu/libc-start.c:379:5
#46 0x0000557cc7ddd6d5 _start (/proj/xhdhdstaff6/dhirajp/localBuild/iree-build/tools/iree-compile+0x16d5)
Abort (core dumped)
ScottTodd commented 1 day ago

Nice, thanks. Do you also have the IR leading up to the crash?

pdhirajkumarprasad commented 1 day ago

@ScottTodd attached is detail log contain the IR for all passes

dump.log