Open shivadbhavsar opened 3 months ago
This seems to have its root in an MIGraphX-only bug in the pad
operator. The following test case in test_ref fails (ignore the gold value; it throws an exception before it gets that far):
TEST_CASE(pad_test)
{
migraphx::program p;
auto* mm = p.get_main_module();
migraphx::shape s{migraphx::shape::float_type, {8192, 8, 23}};
std::vector<float> ones(s.elements(), 1.0f);
auto l0 = mm->add_literal(migraphx::literal{s, ones});
mm->add_instruction(migraphx::make_op("pad", {{"pads", {0, 0, 0, 0, 0, 1}}}), l0);
p.compile(migraphx::make_target("ref"));
auto result = p.eval({}).back();
std::vector<float> results_vector(16);
result.visit([&](auto output) { results_vector.assign(output.begin(), output.end()); });
std::vector<float> gold{0, 0, 0, 0, 0, 1, 2, 0, 0, 3, 4, 0, 0, 0, 0, 0};
EXPECT(migraphx::verify::verify_rms_range(results_vector, gold));
}
It fails with the following message:
terminate called after throwing an instance of 'migraphx::version_1::exception'
what(): /workspace/pt2_benchmarks/AMDMIGraphX/src/include/migraphx/operation.hpp:366: compute_op: Not computable: pad
Aborted (core dumped)
Above replication code is non-operative: that error happened on server hyd-7c-ZT09-02
and doesn't replicate elsewhere. It seems to be independent of this issue.
Note to self: the following reduced pass list in target::get_passes()
AMDMIGraphX/src/targets/gpu/target.cpp
replicates the fail. All 3 of these passes must be present to create the error:
return
{
enable_pass(mlir_enabled(), fuse_mlir{&ctx}),
lowering{&ctx, options.offload_copy},
compile_ops{&ctx, options.exhaustive_tune},
};
Note to self: adding the following tostruct softmax_compiler : compiler
if(axis < 0)
axis += inputs.front().ndim();
solves one bug, but failures continue when we revert to the full pass list (see above).
add: similar fix in src/fuse_reduce.cpp
line 424
I haven't verified that this fix will also work on an MI300--will get on that now.
Error observed in TIMM torch benchmark model: eca_halonext26ts
Full message:
To Repro:
migraphx-driver compile /mnt/nas_share/migraphx/models/torch_benchmarks/eca_halonext26ts/eca_halonext26ts.mxr