fastmachinelearning / hls4ml

Machine learning on FPGAs using HLS
https://fastmachinelearning.org/hls4ml
Apache License 2.0
1.3k stars 419 forks source link

Initial values for the hidden/cell state for LSTM and GRU models in Pytorch #1120

Open JanFSchulte opened 2 weeks ago

JanFSchulte commented 2 weeks ago

This PR addresses https://github.com/fastmachinelearning/hls4ml/issues/1074 and implements the passing of initial values for the hidden and cell states in GRU and LSTM models, which is supported in pytorch. This first version implements this only for the pytorch parser, but it should be able to be extended it for keras and other parsers.

I have tested this for Vivado, Vitis, and Quartus. I don't have access to Catapult or oneAPI at the moment, so I haven't implemented this fort those backends.

Note that this currently only works in io_parallel. In io_stream I was having some conceptual issues and was unsure if I should treat these initial states are streamed inputs or not. Might be good enough for now and I can revisit io_stream if there are any suggestions how to tackle that.

Type of change

Tests

Tested in both standalone scripts and also the pytests to ensure that model parsing and evaluation work with and without passing these optional tensors.

Checklist