[Good first issue]: [API CONFORMANCE] [TEMPLATE PLUGIN] `CachingModelTests` partually are failing on not-supported operation in interpreter

[iefode]

Description:

API conformance suite is OV validation tool checks a plugin conditions from API implementation perspective. ov_plugin*CompileModelCacheTestBase* fails on TEMPLATE Plugin` fails on TEMPLATE plugin according not supported operations in interpreter backend.

How to reproduce:

1. Build the OV using -DENABLE_TESTS=ON -DENABLE_FUNACTIONAL_TESTS=ON
2. Build ov_api_conformance_tests target
3. run API conformance: ov_api_conformance_tests --gtest_filter=*ov_plugin_numeric*CompileModelCacheTestBase* --device=TEMPLATE
4. Get the result for unsupported properties:

   
   [ RUN      ] ov_plugin_numeric/CompileModelCacheTestBase.CompareWithRefImpl/SimpleFunctionMultiply_u16_batch1_TEMPLATE

MEM_USAGE=75876KB Access PluginCache ov core. OV Core use count: 0 Created ov core.

MEM_USAGE=80980KB [ CONFORMANCE ] Influence coefficient: 0 /home/efode/repo/openvino/src/tests/functional/plugin/shared/src/behavior/ov_plugin/caching_tests.cpp:241: Failure Failed Can't loadNetwork without cache for SimpleFunctionMultiply with precision u16 Exception [Exception from src/inference/src/infer_request.cpp:231: Check 'it != map.end()' failed at src/plugins/template/backend/int_executable.cpp:212: Interpreter backend doesn't implement evaluate method for OP Multiply

]

Reset PluginCache. OV Core use count: 1 [ FAILED ] ov_plugin_numeric/CompileModelCacheTestBase.CompareWithRefImpl/SimpleFunctionMultiply_u16_batch1_TEMPLATE, where GetParam() = ((32-byte object <D3-11 68-67 F2-55 00-00 2C-20 22-54 45-4D 50-4C 79-D0 69-67 F2-55 00-00 EE-CF 69-67 F2-55 00-00>, "SimpleFunctionMultiply"), u16, 1, "TEMPLATE", {}) (83 ms)

## How to fix

## Useful links:
* [Intel DevHub Discord channel](https://discord.gg/wPuqAujS) - engage in discussions, ask questions and talk to OpenVINO developers
* [Conformance readme](https://github.com/openvinotoolkit/openvino/tree/master/src/tests/functional/plugin/conformance/test_runner)
* [TEMPLATE plugin](https://github.com/openvinotoolkit/openvino/blob/master/src/plugins/template/README.md)
* [OV CONTRIBUTION GUIDE](https://github.com/openvinotoolkit/openvino/tree/master/?tab=readme-ov-file#how-to-contribute)

## Contacts:
* @iefode @sbalandi 

[LucaTamSapienza]

Hi, I'm looking into this issue and trying to replicate it. I built OV using the command

cmake -DENABLE_TESTS=ON -DENABLE_FUNCTIONAL_TESTS=ON ..

and then attempted to compile the mentioned target using the command

make --jobs=$(nproc --all) ov_api_conformance_tests

(by the way, this command didn't work for me initially; I had to add TARGET= before ov_api_conformance_tests to make it work). However, during the compilation, I consistently encounter the same error when it reaches 60%:**make: * [Makefile:156: all] Error 2. I checked the Makefile for a solution, but couldn't find one. I've restarted the build several times, but it didn't work. I'm currently stuck and unable to work on the issue, even though I would like to. Does anyone know what could be the problem?

[LucaTamSapienza]

This is the full error clang: error: invalid linker name in argument '-fuse-ld=lld' [ 55%] Linking CXX static library /home/luca/Documents/OpenSourceProject/openvino/bin/intel64/Release/libopenvino_onednn_cpu.a make[2]: [thirdparty/protobuf/protobuf/cmake/CMakeFiles/protoc.dir/build.make:99: /home/luca/Documents/OpenSourceProject/openvino/bin/intel64/Release/protoc-3.20.3.0] Error 1 make[1]: [CMakeFiles/Makefile2:2388: thirdparty/protobuf/protobuf/cmake/CMakeFiles/protoc.dir/all] Error 2 make[1]: Waiting for unfinished jobs.... [ 60%] Built target openvino_transformations_obj [ 65%] Built target openvino_core_obj [ 65%] Built target dnnl make: [Makefile:156: all] Error 2

[iefode]

@LucaTamSapienza Hello, Sorry for late reply.

Do you use latest master branch in OpenVINO? I have tried to reproduce the following behavior locally, and my build was successful on Linux. I used

cd openvino/build
cmake -DENABLE_FUNATIONAL_TESTS=ON -DENABLE_TESTS=ON ..
make -j24 ov_api_conformance_tests

Looks like you have a problem with protobuf. Which OS do you use? Do you have installed protobuf (it can lead to misalignment to ov protobuf)?

[LucaTamSapienza]

@iefode I apologize, it was my mistake; I didn't have the branch updated. I ran the following command ov_api_conformance_tests --gtest_filter=*ov_plugin_numeric*CompileModelCacheTestBase* --device=TEMPLATE, and I obtained the following output(as you mentioned above):

[ RUN      ] ov_plugin_numeric/CompileModelCacheTestBase.CompareWithRefImpl/SimpleFunctionMultiply_u16_batch1_TEMPLATE

MEM_USAGE=373156KB

MEM_USAGE=373156KB
[ CONFORMANCE ] Influence coefficient: 0
src/tests/functional/plugin/shared/src/behavior/ov_plugin/caching_tests.cpp:238: Failure
Failed
Can't loadNetwork without cache for SimpleFunctionMultiply with precision u16
Exception [Exception from src/inference/src/cpp/infer_request.cpp:223:
Check 'it != map.end()' failed at src/plugins/template/backend/int_executable.cpp:212:
Interpreter backend doesn't implement evaluate method for OP Multiply

]

[  FAILED  ] ov_plugin_numeric/CompileModelCacheTestBase.CompareWithRefImpl/SimpleFunctionMultiply_u16_batch1_TEMPLATE, where GetParam() = ((32-byte object <90-60 05-A8 B7-55 00-00 00-00 00-00 00-00 00-00 E0-68 03-A8 B7-55 00-00 20-63 03-A8 B7-55 00-00>, "SimpleFunctionMultiply"), u16, 1, "TEMPLATE", {}) (6 ms)

Since I still don't feel confident in taking on the request, I will work on it and read some documentation about it in the next few days. If I manage to find a way to resolve it, I will take it on. Thank you very much for your help. I'll write here if i find something.

[iefode]

@LucaTamSapienza Please let Me know if you need any additional help to fix the issue. I will share technical details to resolve the problem. It should be easier to fix then it looks :)

[LucaTamSapienza]

@iefode Hi! I've looked around a bit, and I believe a proper approach to solving this issue is to first add u16 to the supported types in the <Multiply>evaluate function and <Multiply>hasEvaluate (i.e. multiply.cpp). Additionally, since the error suggests a problem thrown in the int_backend file, particularly in int_executable, within the evaluate_node function, I think the issue might be that the evaluate operation is not registered, even though it's correctly implemented for the Multiply class. Before proceeding, I'd like to have your feedback to know if I'm on the right track. Thank you very much!

[iefode]

@LucaTamSapienza

Hi! I've looked around a bit, and I believe a proper approach to solving this issue is to first add u16 to the supported types in the evaluate function and hasEvaluate (i.e. multiply.cpp).

In general, you are right. But OV is an inference on edge solution. It means that we have an limitation for OV binary size library which contains operation classes. As you know, evaluate calls template function inside to cast tensors to pointer based on element types. This approach will increase OV library size ...

Additionally, since the error suggests a problem thrown in the int_backend file, particularly in int_executable, within the evaluate_node function, I think the issue might be that the evaluate operation is not registered, even though it's correctly implemented for the Multiply class.

As a consequence of first one point we have developer the second approach to call a reference function which is places of ov lib and linked to test utilities. The approach is based on eveluates map where node type is template arg. My suggestion is extend this approach to fix the original problem.

Great work! Thanks!

[LucaTamSapienza]

@iefode Perfect, i think i can fix it, i'll prepare a PR as soon as possible to discuss possible problems that i could encounter. Thank you so much!

[LucaTamSapienza]

.take

[github-actions[bot]]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[LucaTamSapienza]

@iefode Hi, here is my PR #22832