search

iree-org / iree

A retargetable MLIR-based machine learning compiler and runtime toolkit.
http://iree.dev/
Apache License 2.0
2.47k stars 551 forks source link

Assertion `input.size() == permutation.size() && "expected input rank to equal permutation rank"' failed #17734

Closed AmosLewis closed 2 days ago

AmosLewis commented 6 days ago

What happened?

mit-b0 pytorch model to onnx failed https://github.com/nod-ai/SHARK-TestSuite/issues/270

pytorch/models/mobilebert-uncased pytorch/models/t5-base pytorch/models/t5-large has the same issue 

iree-compile: iree/third_party/llvm-project/mlir/include/mlir/Dialect/Utils/IndexingUtils.h:204: SmallVector<T> mlir::applyPermutation(ArrayRef<T>, ArrayRef<int64_t>) [T = llvm::SmallVector<long, 2>]: Assertion `input.size() == permutation.size() && "expected input rank to equal permutation rank"' failed.
Please report issues to https://github.com/iree-org/iree/issues and include the crash backtrace.
Stack dump:
0.  Program arguments: /home/chi/src/iree-build/tools/iree-compile --iree-input-demote-i64-to-i32 --iree-hal-target-backends=llvm-cpu mit-b0.default.pytorch.linalg.mlir
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  libIREECompiler.so 0x000073a30d3f033d llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 61
1  libIREECompiler.so 0x000073a30d3f082b
2  libIREECompiler.so 0x000073a30d3ee896 llvm::sys::RunSignalHandlers() + 134
3  libIREECompiler.so 0x000073a30d3f0fe5
4  libc.so.6          0x000073a301442520
5  libc.so.6          0x000073a3014969fc pthread_kill + 300
6  libc.so.6          0x000073a301442476 raise + 22
7  libc.so.6          0x000073a3014287f3 abort + 211
8  libc.so.6          0x000073a30142871b
9  libc.so.6          0x000073a301439e96
10 libIREECompiler.so 0x000073a310368e98
11 libIREECompiler.so 0x000073a310368dd9
12 libIREECompiler.so 0x000073a310368979
13 libIREECompiler.so 0x000073a3135599c5
14 libIREECompiler.so 0x000073a3135594e6
15 libIREECompiler.so 0x000073a313559057
16 libIREECompiler.so 0x000073a313558fef
17 libIREECompiler.so 0x000073a313558b45
18 libIREECompiler.so 0x000073a313558a91
19 libIREECompiler.so 0x000073a3103b87eb
20 libIREECompiler.so 0x000073a3156cf52e
21 libIREECompiler.so 0x000073a3156cf385
22 libIREECompiler.so 0x000073a30d3074a9
23 libIREECompiler.so 0x000073a3156d0d55
24 libIREECompiler.so 0x000073a3156cde27 mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>) + 1463
25 libIREECompiler.so 0x000073a3156898db
26 libIREECompiler.so 0x000073a315688b61
27 libIREECompiler.so 0x000073a315688b35
28 libIREECompiler.so 0x000073a30d3074a9
29 libIREECompiler.so 0x000073a315688285
30 libIREECompiler.so 0x000073a31568670e
31 libIREECompiler.so 0x000073a3156863f7 mlir::applyPatternsAndFoldGreedily(mlir::Region&, mlir::FrozenRewritePatternSet const&, mlir::GreedyRewriteConfig, bool*) + 231
32 libIREECompiler.so 0x000073a30d24a105
33 libIREECompiler.so 0x000073a31031f23b
34 libIREECompiler.so 0x000073a30d80335b
35 libIREECompiler.so 0x000073a30d8032f5
36 libIREECompiler.so 0x000073a30d3074a9
37 libIREECompiler.so 0x000073a30d806175
38 libIREECompiler.so 0x000073a30d7feab3 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) + 851
39 libIREECompiler.so 0x000073a30d7ff034 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) + 388
40 libIREECompiler.so 0x000073a30d8045e5
41 libIREECompiler.so 0x000073a30d804269
42 libIREECompiler.so 0x000073a30d8002eb
43 libIREECompiler.so 0x000073a30d7ffbfa mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool) + 1194
44 libIREECompiler.so 0x000073a30d7ff727 mlir::detail::OpToOpPassAdaptor::runOnOperation(bool) + 71
45 libIREECompiler.so 0x000073a30d803346
46 libIREECompiler.so 0x000073a30d8032f5
47 libIREECompiler.so 0x000073a30d3074a9
48 libIREECompiler.so 0x000073a30d806175
49 libIREECompiler.so 0x000073a30d7feab3 mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int) + 851
50 libIREECompiler.so 0x000073a30d7ff034 mlir::detail::OpToOpPassAdaptor::runPipeline(mlir::OpPassManager&, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int, mlir::PassInstrumentor*, mlir::PassInstrumentation::PipelineParentInfo const*) + 388
51 libIREECompiler.so 0x000073a30d800a78 mlir::PassManager::runPasses(mlir::Operation*, mlir::AnalysisManager) + 104
52 libIREECompiler.so 0x000073a30d8009a2 mlir::PassManager::run(mlir::Operation*) + 1138
53 libIREECompiler.so 0x000073a30d251cba
54 libIREECompiler.so 0x000073a30d251593 ireeCompilerInvocationPipeline + 35
55 libIREECompiler.so 0x000073a30d78a88e
56 libIREECompiler.so 0x000073a30d789d1e
57 libIREECompiler.so 0x000073a30d29baab ireeCompilerRunMain + 27
58 iree-compile       0x0000559f7d7347a2
59 libc.so.6          0x000073a301429d90
60 libc.so.6          0x000073a301429e40 __libc_start_main + 128
61 iree-compile       0x0000559f7d7346b5
Aborted (core dumped)

gdb backtrakce output:

(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737352719616) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737352719616) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737352719616, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fffdd442476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fffdd4287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fffdd42871b in __assert_fail_base (fmt=0x7fffdd5dd130 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", 
    assertion=0x7fffe22dc65a "input.size() == permutation.size() && \"expected input rank to equal permutation rank\"", 
    file=0x7fffe12f935b "iree/third_party/llvm-project/mlir/include/mlir/Dialect/Utils/IndexingUtils.h", line=204, function=<optimized out>) at ./assert/assert.c:92
#6  0x00007fffdd439e96 in __GI___assert_fail (assertion=0x7fffe22dc65a "input.size() == permutation.size() && \"expected input rank to equal permutation rank\"", 
    file=0x7fffe12f935b "iree/third_party/llvm-project/mlir/include/mlir/Dialect/Utils/IndexingUtils.h", line=204, 
    function=0x7fffe23b6a01 "SmallVector<T> mlir::applyPermutation(ArrayRef<T>, ArrayRef<int64_t>) [T = llvm::SmallVector<long, 2>]") at ./assert/assert.c:101
#7  0x00007fffec368e98 in mlir::applyPermutation<llvm::SmallVector<long, 2u> > (input=..., permutation=...)
    at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/Dialect/Utils/IndexingUtils.h:203

#8  0x00007fffec368dd9 in mlir::applyPermutation<llvm::SmallVector<long, 2u> > (input=..., permutation=...)
    at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/Dialect/Utils/IndexingUtils.h:214
#9  0x00007fffec368979 in mlir::applyPermutationToVector<llvm::SmallVector<long, 2u>, 1u> (inVec=..., permutation=...)
    at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/Dialect/Utils/IndexingUtils.h:225
#10 0x00007fffef5599c5 in (anonymous namespace)::applyPermutationAndReindexReassoc (reassocIndices=..., permutation=...)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp:608
#11 0x00007fffef5594e6 in (anonymous namespace)::bubbleUpPackOpThroughCollapseShape (collapseOp=..., packOp=..., rewriter=...)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp:685
#12 0x00007fffef559057 in (anonymous namespace)::BubbleUpPackOpThroughReshapeOp::matchAndRewrite(mlir::tensor::PackOp, mlir::PatternRewriter&) const::{lambda(mlir::tensor::CollapseShapeOp)#1}::operat--Type <RET> for more, q to quit, c to continue without paging--
or()(mlir::tensor::CollapseShapeOp) const (this=0x7fffffff8fc0, op=...) at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp:851
#13 0x00007fffef558fef in llvm::TypeSwitch<mlir::Operation*, mlir::LogicalResult>::Case<mlir::tensor::CollapseShapeOp, (anonymous namespace)::BubbleUpPackOpThroughReshapeOp::matchAndRewrite(mlir::tensor::PackOp, mlir::PatternRewriter&) const::{lambda(mlir::tensor::CollapseShapeOp)#1}>((anonymous namespace)::BubbleUpPackOpThroughReshapeOp::matchAndRewrite(mlir::tensor::PackOp, mlir::PatternRewriter&) const::{lambda(mlir::tensor::CollapseShapeOp)#1}&&) (this=0x7fffffff8fd0, caseFn=...) at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/TypeSwitch.h:102
#14 0x00007fffef558b45 in llvm::detail::TypeSwitchBase<llvm::TypeSwitch<mlir::Operation*, mlir::LogicalResult>, mlir::Operation*>::Case<(anonymous namespace)::BubbleUpPackOpThroughReshapeOp::matchAndRewrite(mlir::tensor::PackOp, mlir::PatternRewriter&) const::{lambda(mlir::tensor::CollapseShapeOp)#1}>((anonymous namespace)::BubbleUpPackOpThroughReshapeOp::matchAndRewrite(mlir::tensor::PackOp, mlir::PatternRewriter&) const::{lambda(mlir::tensor::CollapseShapeOp)#1}&&) (this=0x7fffffff8fd0, caseFn=...) at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/TypeSwitch.h:60
#15 0x00007fffef558a91 in (anonymous namespace)::BubbleUpPackOpThroughReshapeOp::matchAndRewrite (this=0x5555557239d0, packOp=..., rewriter=...)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Dialect/Linalg/Transforms/DataLayoutPropagation.cpp:850
#16 0x00007fffec3b87eb in mlir::detail::OpOrInterfaceRewritePatternBase<mlir::tensor::PackOp>::matchAndRewrite (this=0x5555557239d0, op=0x555556649190, rewriter=...)
    at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/IR/PatternMatch.h:331
#17 0x00007ffff16cf52e in mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>)::$_0::operator()() const (this=0x7fffffff92b0)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Rewrite/PatternApplicator.cpp:212
#18 0x00007ffff16cf385 in llvm::function_ref<void ()>::callback_fn<mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>)::$_0>(long) (callable=140737488327344)
    at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45
#19 0x00007fffe93074a9 in llvm::function_ref<void ()>::operator()() const (this=0x7fffffff91f0) at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68
#20 0x00007ffff16d0d55 in mlir::MLIRContext::executeAction<mlir::ApplyPatternAction, mlir::Pattern const&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pattern const&) (
    this=0x5555555ec710, actionFn=..., irUnits=..., args=...) at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275
#21 0x00007ffff16cde27 in mlir::PatternApplicator::matchAndRewrite(mlir::Operation*, mlir::PatternRewriter&, llvm::function_ref<bool (mlir::Pattern const&)>, llvm::function_ref<void (mlir::Pattern const&)>, llvm::function_ref<mlir::LogicalResult (mlir::Pattern const&)>) (this=0x7fffffff9ed0, op=0x555556649190, rewriter=..., canApply=..., onFailure=..., onSuccess=...)
--Type <RET> for more, q to quit, c to continue without paging--
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Rewrite/PatternApplicator.cpp:195
#22 0x00007ffff16898db in (anonymous namespace)::GreedyPatternRewriteDriver::processWorklist (this=0x7fffffff9dd0)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:615
#23 0x00007ffff1688b61 in (anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) &&::$_2::operator()() const (this=0x7fffffff9c80)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:874
#24 0x00007ffff1688b35 in llvm::function_ref<void ()>::callback_fn<(anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) &&::$_2>(long) (callable=140737488329856)
    at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45
#25 0x00007fffe93074a9 in llvm::function_ref<void ()>::operator()() const (this=0x7fffffff9c20) at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68
#26 0x00007ffff1688285 in mlir::MLIRContext::executeAction<(anonymous namespace)::GreedyPatternRewriteIteration, long&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, long&) (
    this=0x5555555ec710, actionFn=..., irUnits=..., args=@0x7fffffff9d88: 2) at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275
#27 0x00007ffff168670e in (anonymous namespace)::RegionPatternRewriteDriver::simplify(bool*) && (this=0x7fffffff9dd0, changed=0x7fffffff9fd7)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:872
#28 0x00007ffff16863f7 in mlir::applyPatternsAndFoldGreedily (region=..., patterns=..., config=..., changed=0x7fffffff9fd7)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Transforms/Utils/GreedyPatternRewriteDriver.cpp:919
#29 0x00007fffe924a105 in mlir::applyPatternsAndFoldGreedily (op=0x5555557d1cf0, patterns=..., config=..., changed=0x0)
    at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/Transforms/GreedyPatternRewriteDriver.h:159
#30 0x00007fffec31f23b in mlir::iree_compiler::GlobalOptimization::(anonymous namespace)::DataLayoutPropagationPass::runOnOperation (this=0x55555663f1d0)
    at /home/chi/src/iree/compiler/src/iree/compiler/GlobalOptimization/DataLayoutPropagation.cpp:31
#31 0x00007fffe980335b in mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1::operator()() const (this=0x7fffffffa428)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:527
#32 0x00007fffe98032f5 in llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1>(long) (
    callable=140737488331816) at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45
--Type <RET> for more, q to quit, c to continue without paging--
#33 0x00007fffe93074a9 in llvm::function_ref<void ()>::operator()() const (this=0x7fffffffa3b0) at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68
#34 0x00007fffe9806175 in mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) (this=0x5555555ec710, 
    actionFn=..., irUnits=..., args=...) at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275
#35 0x00007fffe97feab3 in mlir::detail::OpToOpPassAdaptor::run (pass=0x55555663f1d0, op=0x5555557d1cf0, am=..., verifyPasses=true, parentInitGeneration=1)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:521
#36 0x00007fffe97ff034 in mlir::detail::OpToOpPassAdaptor::runPipeline (pm=..., op=0x5555557d1cf0, am=..., verifyPasses=true, parentInitGeneration=1, instrumentor=0x5555557f5ee0, 
    parentInfo=0x7fffffffaae0) at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:593
#37 0x00007fffe98045e5 in mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_0::operator()(mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo&) const (
    this=0x7fffffffaa78, opInfo=...) at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:813
#38 0x00007fffe9804269 in mlir::failableParallelForEach<__gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_0&>(mlir::MLIRContext*, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, __gnu_cxx::__normal_iterator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> > >, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_0&) (context=0x5555555ec710, begin={passManagerIdx = 0, op = 0x5555557d1cf0, am = {impl = 0x5555557d2ce0}}, 
    end={passManagerIdx = 129, op = 0x55555578f6a0, am = {impl = 0x55555578ef00}}, func=...) at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/IR/Threading.h:46
#39 0x00007fffe98002eb in mlir::failableParallelForEach<std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> >&, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_0&>(mlir::MLIRContext*, std::vector<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo, std::allocator<mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::OpPMInfo> >&, mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl(bool)::$_0&) (
    context=0x5555555ec710, range=std::vector of length 1, capacity 1 = {...}, func=...) at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/IR/Threading.h:92
#40 0x00007fffe97ffbfa in mlir::detail::OpToOpPassAdaptor::runOnOperationAsyncImpl (this=0x555555804b30, verifyPasses=true) at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:823
#41 0x00007fffe97ff727 in mlir::detail::OpToOpPassAdaptor::runOnOperation (this=0x555555804b30, verifyPasses=true) at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:714
--Type <RET> for more, q to quit, c to continue without paging--
#42 0x00007fffe9803346 in mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1::operator()() const (this=0x7fffffffade8)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:525
#43 0x00007fffe98032f5 in llvm::function_ref<void ()>::callback_fn<mlir::detail::OpToOpPassAdaptor::run(mlir::Pass*, mlir::Operation*, mlir::AnalysisManager, bool, unsigned int)::$_1>(long) (
    callable=140737488334312) at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:45
#44 0x00007fffe93074a9 in llvm::function_ref<void ()>::operator()() const (this=0x7fffffffad70) at /home/chi/src/iree/third_party/llvm-project/llvm/include/llvm/ADT/STLFunctionalExtras.h:68
#45 0x00007fffe9806175 in mlir::MLIRContext::executeAction<mlir::PassExecutionAction, mlir::Pass&>(llvm::function_ref<void ()>, llvm::ArrayRef<mlir::IRUnit>, mlir::Pass&) (this=0x5555555ec710, 
    actionFn=..., irUnits=..., args=...) at /home/chi/src/iree/third_party/llvm-project/mlir/include/mlir/IR/MLIRContext.h:275
#46 0x00007fffe97feab3 in mlir::detail::OpToOpPassAdaptor::run (pass=0x555555804b30, op=0x5555557f5690, am=..., verifyPasses=true, parentInitGeneration=1)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:521
#47 0x00007fffe97ff034 in mlir::detail::OpToOpPassAdaptor::runPipeline (pm=..., op=0x5555557f5690, am=..., verifyPasses=true, parentInitGeneration=1, instrumentor=0x0, parentInfo=0x0)
    at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:593
#48 0x00007fffe9800a78 in mlir::PassManager::runPasses (this=0x555555739cb0, op=0x5555557f5690, am=...) at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:904
#49 0x00007fffe98009a2 in mlir::PassManager::run (this=0x555555739cb0, op=0x5555557f5690) at /home/chi/src/iree/third_party/llvm-project/mlir/lib/Pass/Pass.cpp:884
#50 0x00007fffe9251cba in mlir::iree_compiler::embed::(anonymous namespace)::Invocation::runPipeline (this=0x55555565add0, pipeline=IREE_COMPILER_PIPELINE_STD)
    at /home/chi/src/iree/compiler/src/iree/compiler/API/Internal/CompilerDriver.cpp:995
#51 0x00007fffe9251593 in ireeCompilerInvocationPipeline (inv=0x55555565add0, pipeline=IREE_COMPILER_PIPELINE_STD)
    at /home/chi/src/iree/compiler/src/iree/compiler/API/Internal/CompilerDriver.cpp:1430
#52 0x00007fffe978a88e in mlir::iree_compiler::runIreecMain(int, char**)::$_2::operator()(iree_compiler_source_t*) const (this=0x7fffffffc0e8, source=0x55555565aba0)
    at /home/chi/src/iree/compiler/src/iree/compiler/Tools/iree_compile_lib.cc:254
#53 0x00007fffe9789d1e in mlir::iree_compiler::runIreecMain (argc=4, argv=0x7fffffffdbd8) at /home/chi/src/iree/compiler/src/iree/compiler/Tools/iree_compile_lib.cc:355
#54 0x00007fffe929baab in ireeCompilerRunMain (argc=4, argv=0x7fffffffdbd8) at /home/chi/src/iree/compiler/src/iree/compiler/API/Internal/IREECompileToolEntryPoint.cpp:12
#55 0x00005555555557a2 in main (argc=4, argv=0x7fffffffdbd8) at /home/chi/src/iree/tools/iree-compile-main.cc:9

Steps to reproduce your issue

input ir: mit-b0.default.pytorch.linalg.elide.mlir

/home/chi/src/iree-build/tools/iree-compile --iree-input-demote-i64-to-i32 --iree-hal-target-backends=llvm-cpu mit-b0.default.pytorch.linalg.elide.mlir > mit-b0.default.vmfb 2>iree-compile.log

What component(s) does this issue relate to?

Compiler

Version information

iree candidate-20240624.934

Additional context

No response

ScottTodd commented 5 days ago

The sharktank llama model (https://github.com/nod-ai/sharktank/issues/22) is failing with this same error. Stack with more context:

(sharktank) λ D:\dev\projects\iree-build\tools\iree-compile D:/tmp/open_llama_3b_v2/open-llama-3b-v2-f16.mlir --iree-hal-target-backends=llvm-cpu --iree-llvmcpu-target-cpu-features=host -o D:/tmp/open_llama_3b_v2/open-llama-3b-v2-f16_cpu.vmfb
Assertion failed: input.size() == permutation.size() && "expected input rank to equal permutation rank", file D:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir/Dialect/Utils/IndexingUtils.h, line 204
Please report issues to https://github.com/iree-org/iree/issues and include the crash backtrace.
Exception Code: 0x80000003
 #0 0x00007ff601268e95 HandleAbort D:\dev\projects\iree\third_party\llvm-project\llvm\lib\Support\Windows\Signals.inc:425:0
 #1 0x00007ffe7d561881 (C:\WINDOWS\System32\ucrtbase.dll+0x71881)
 #2 0x00007ffe7d562851 (C:\WINDOWS\System32\ucrtbase.dll+0x72851)
 #3 0x00007ffe7d56426e (C:\WINDOWS\System32\ucrtbase.dll+0x7426e)
 #4 0x00007ffe7d564165 (C:\WINDOWS\System32\ucrtbase.dll+0x74165)
 #5 0x00007ffe7d5644f1 (C:\WINDOWS\System32\ucrtbase.dll+0x744f1)
 #6 0x00007ff6051bf70f mlir::applyPermutation<class llvm::SmallVector<__int64, 2>>(class llvm::ArrayRef<class llvm::SmallVector<__int64, 2>>, class llvm::ArrayRef<__int64>) D:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\Dialect\Utils\IndexingUtils.h:205:0
 #7 0x00007ff6051b9c11 mlir::applyPermutation<class llvm::SmallVector<__int64, 2>>(class llvm::SmallVectorImpl<class llvm::SmallVector<__int64, 2>> const &, class llvm::ArrayRef<__int64>) D:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\Dialect\Utils\IndexingUtils.h:214:0
 #8 0x00007ff6051b1c2b mlir::applyPermutationToVector<class llvm::SmallVector<__int64, 2>, 1>(class llvm::SmallVector<class llvm::SmallVector<__int64, 2>, 1> &, class llvm::ArrayRef<__int64>) D:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\Dialect\Utils\IndexingUtils.h:225:0
 #9 0x00007ff606741eab `anonymous namespace'::applyPermutationAndReindexReassoc D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Dialect\Linalg\Transforms\DataLayoutPropagation.cpp:610:0
#10 0x00007ff60674268d `anonymous namespace'::bubbleUpPackOpThroughCollapseShape D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Dialect\Linalg\Transforms\DataLayoutPropagation.cpp:687:0
#11 0x00007ff606743bef `anonymous namespace'::BubbleUpPackOpThroughReshapeOp::matchAndRewrite D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Dialect\Linalg\Transforms\DataLayoutPropagation.cpp:849:0
#12 0x00007ff60409bbe4 mlir::detail::OpOrInterfaceRewritePatternBase<class mlir::tensor::PackOp>::matchAndRewrite(class mlir::Operation *, class mlir::PatternRewriter &) const D:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\PatternMatch.h:332:0
#13 0x00007ff60560e8eb <lambda_033eed04a8a10a7b33015298d48d216a>::operator() D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Rewrite\PatternApplicator.cpp:212:0
#14 0x00007ff60560c275 mlir::PatternApplicator::matchAndRewrite(class mlir::Operation *, class mlir::PatternRewriter &, class llvm::function_ref<(class mlir::Pattern const &)>, class llvm::function_ref<(class mlir::Pattern const &)>, class llvm::function_ref<(class mlir::Pattern const &)>) D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Rewrite\PatternApplicator.cpp:233:0
#15 0x00007ff60448f91e `anonymous namespace'::GreedyPatternRewriteDriver::processWorklist D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Transforms\Utils\GreedyPatternRewriteDriver.cpp:617:0
#16 0x00007ff6044920e2 llvm::function_ref<void __cdecl(void)>::callback_fn<<lambda_56efa1fe2231a48e07ce9bd5369059af> > D:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:45:0
#17 0x00007ff6044914ae `anonymous namespace'::RegionPatternRewriteDriver::simplify D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Transforms\Utils\GreedyPatternRewriteDriver.cpp:872:0
#18 0x00007ff60448d38e mlir::applyPatternsAndFoldGreedily(class mlir::Region &, class mlir::FrozenRewritePatternSet const &, class mlir::GreedyRewriteConfig, bool *) D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Transforms\Utils\GreedyPatternRewriteDriver.cpp:920:0
#19 0x00007ff6051e8d1d mlir::iree_compiler::GlobalOptimization::`anonymous namespace'::DataLayoutPropagationPass::runOnOperation D:\dev\projects\iree\compiler\src\iree\compiler\GlobalOptimization\DataLayoutPropagation.cpp:31:0
#20 0x00007ff60163ead0 llvm::function_ref<void __cdecl(void)>::callback_fn<<lambda_e8f8990a45bf3495636c03506b9db479> > D:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\ADT\STLFunctionalExtras.h:45:0
#21 0x00007ff601638637 mlir::detail::OpToOpPassAdaptor::run(class mlir::Pass *, class mlir::Operation *, class mlir::AnalysisManager, bool, unsigned int) D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:533:0
#22 0x00007ff60163883d mlir::detail::OpToOpPassAdaptor::runPipeline(class mlir::OpPassManager &, class mlir::Operation *, class mlir::AnalysisManager, bool, unsigned int, class mlir::PassInstrumentor *, struct mlir::PassInstrumentation::PipelineParentInfo const *) D:\dev\projects\iree\third_party\llvm-project\mlir\lib\Pass\Pass.cpp:593:0
#23 0x00007ff60163fa5b <lambda_060c7f84c4de8022f660b122ba4cdde9>::operator() D:\dev\projects\iree\third_party\llvm-project\mlir\include\mlir\IR\Threading.h:62:0
#24 0x00007ff6016403db std::_Packaged_state<(void)>::_Call_immediate(void) C:\Program Files (x86)\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.31.31103\include\future:593:0
#25 0x00007ff60164054f std::_Deferred_async_state<void>::_Run_deferred_function(class std::unique_lock<class std::mutex> &) C:\Program Files (x86)\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.31.31103\include\future:659:0
#26 0x00007ff60163b02d std::_Associated_state<int>::_Wait(void) C:\Program Files (x86)\Microsoft Visual Studio\2022\Preview\VC\Tools\MSVC\14.31.31103\include\future:223:0
#27 0x00007ff6030c6473 llvm::StdThreadPool::processTasks(class llvm::ThreadPoolTaskGroup *) D:\dev\projects\iree\third_party\llvm-project\llvm\lib\Support\ThreadPool.cpp:103:0
#28 0x00007ff6030c79d3 llvm::thread::ThreadProxy<std::tuple<<lambda_a09d07335cf06810bfcca9f8daa525ff> > > D:\dev\projects\iree\third_party\llvm-project\llvm\include\llvm\Support\thread.h:65:0
#29 0x00007ffe7d511bb2 (C:\WINDOWS\System32\ucrtbase.dll+0x21bb2)
#30 0x00007ffe7e987344 (C:\WINDOWS\System32\KERNEL32.DLL+0x17344)
#31 0x00007ffe7f9bcc91 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x4cc91)
ScottTodd commented 5 days ago

Possible culprit: https://github.com/llvm/llvm-project/pull/93529

ScottTodd commented 5 days ago

Confirmed - my issue does not occur with that PR reverted. Going to dig a bit deeper then comment there.

hanhanW commented 5 days ago

Putting a note here:

https://github.com/llvm/llvm-project/pull/96697 exposes the control of (producer, consumer) pair to controlFn API. https://github.com/iree-org/iree/pull/17740 contains the IREE fix for the upstream change. I have a local patch which re-enables the pack->expand_shape propagation in IREE.

AmosLewis commented 5 days ago

Confirmed - my issue does not occur with that PR reverted. Going to dig a bit deeper then comment there.

mit-b0 issue goes away as well with revert this https://github.com/llvm/llvm-project/pull/93529

cd iree/third_party/llvm_project/
git revert a945f55d3e6af6be6648fb92a20c80e88e3fc2b2
cd  SHARK-TestSuite/e2eshark
python ./run.py --torchmlirbuild ../../torch-mlir/build --tolerance 0.001 0.001 --cachedir ./huggingface_cache --ireebuild ../../iree-build  -f pytorch -g models --mode onnx --report --tests pytorch/models/mit-b0 --torchtolinalg 

Status report for run: test-run using mode:onnx todtype:default backend:llvm-cpu

| tests                 | model-run   | onnx-import   | torch-mlir   | iree-compile   | inference   |
|:----------------------|:------------|:--------------|:-------------|:---------------|:------------|
| pytorch/models/mit-b0 | passed      | passed        | passed       | passed         | mismatch    |
AmosLewis commented 4 days ago

Fixed by https://github.com/llvm/llvm-project/pull/96732 need bump