Please make sure to check off these prerequisites before submitting a bug report.
[x] Test that the bug appears on the current version of the master branch. Make sure to include the commit hash of the commit you checked out.
[x] Check that the issue hasn't already been reported, by checking the currently open issues.
[x] If there are steps to reproduce the problem, make sure to write them down below.
[ ] If relevant, please include the hls4ml project files, which were created directly before and/or after the bug.
Quick summary
Created an LSTM neural network with Keras which achieves 98.40% AUC score on the test set. However, the HLS-model only achieves 54.75% when the "Strategy" is set to "Resource". If "Strategy" is set to "Latency", the expected AUC scores are received, i.e. 98.40% (Keras) and 98.16% (HLS4ML).
Details
The problem occurs with the latest release (v0.7.0) as with the main branch 51be56eeefe7c1f46945528143ef0ac38d174a22 .
Prerequisites
Please make sure to check off these prerequisites before submitting a bug report.
Quick summary
Created an LSTM neural network with Keras which achieves 98.40% AUC score on the test set. However, the HLS-model only achieves 54.75% when the "Strategy" is set to "Resource". If "Strategy" is set to "Latency", the expected AUC scores are received, i.e. 98.40% (Keras) and 98.16% (HLS4ML).
Details
The problem occurs with the latest release (v0.7.0) as with the main branch 51be56eeefe7c1f46945528143ef0ac38d174a22 .
With strategy=resource:
Results:
With strategy=latency:
Results:
Overview of the LSTM network:
Steps to Reproduce
See previous section.
Expected behavior
As I understood the strategy shouldn't influence the AUC score/performance of the model.
Actual behavior
Different strategies result in different AUC scores.