LSTM in HydPy?

[tyralla]

We thought about implementing long short-term memory (LSTM) in HydPy, which currently seems to be the most promising artificial neural network type for some hydrological applications (e.g. flood forecasting). I open this issue as a notebook for the following deeper discussions. The key questions are:

• What is the best way to implement LSTM in HydPy?
• What would be the advantages and disadvantages of including LSTM in HydPy (compared to combining HydPy and an independent LSTM library in Python scripts)?
• Depending on the expected necessary work, do we really want to do it?

[BGWKlein]

We should also think about the requirements:

• LSTM parameters should be stored similary to the parameters of the other hydrological models
• States of the LSTM should be stored similary in initial condition files that we can stop and restart LSTM simulations at any time
• Number of input variables should be flexible as in LSTMs any input variable could be used instead of the fixed input variables (precipitation, temperature,...) defined by the different hydrological model concepts
• Should the input variables be fixed for all LSTM models in a hydpy network or could they vary from catchment (e.g. only precipitation is used) to catchment (e.g. precipitation and temperature is used as input)

[tyralla]

Tensorflow / Keras

@HenningOp and I discussed the possibility of wrapping the LSTM implementation of Tensorflow/Keras into HydPy. Tensorflow seems to be the current standard tool for the given purpose, and Keras provides a more convenient API for it.

Tensorflow is available on PyPI, which is good news, as we only support installing HydPy via pip and not via conda.

HydPy should always work under all active Python versions, currently 3.8, 3.9, 3.10, and 3.11. The latest Tensorflow version (2.13.0) is available for exactly these Python versions. I am not aware of any general promises regarding the supported Python versions, but I do not expect Tensorflow to start dropping active Python versions soon.

PyPI provides "win", "macosx", and "manylinux" Tensorflow wheels, so we could still use HydPy under Linux and Windows after wrapping Tensorflow.

Tensorflow uses the Apache License 2.0, whereas HydPy uses the GNU Lesser General Public License 3.0. The Apache License seems more permissive, so it should allow us to pack Tensorflow into our HydPy installer. However, someone should check this more carefully.

johnnydep told me that tensorflow-intel is Tensorflow's only dependency. However, when installing Tensorflow in my virtual environment for developing HydPy, pip also installed or changed the following packages:

• absl-py-1.4.0
• astunparse-1.6.3
• flatbuffers-23.5.26
• gast-0.4.0
• google-auth-2.22.0
• google-auth-oauthlib-1.0.0
• google-pasta-0.2.0
• grpcio-1.56.0
• h5py-3.9.0
• keras-2.13.1
• libclang-16.0.6
• markdown-3.4.3
• numpy-1.24.3
• oauthlib-3.2.2
• opt-einsum-3.3.0
• protobuf-4.23.4
• pyasn1-0.5.0
• pyasn1-modules-0.3.0
• requests-oauthlib-1.3.1
• rsa-4.9
• tensorboard-2.13.0
• tensorboard-data-server-0.7.1
• tensorflow-2.13.0
• tensorflow-estimator-2.13.0
• tensorflow-intel-2.13.0
• tensorflow-io-gcs-filesystem-0.31.0
• termcolor-2.3.0
• typing-extensions-4.5.0
• werkzeug-2.3.6

For installing tensorflow-intel, pip needed to download 276.6 MB. So, adding Tensorflow to the HydPy installer (if it works at all) would vastly increase its size (currently about 100 MB).

Installing Tensorflow downgraded numpy from version 1.25.0 to 1.24.3, as its requirements.txt files specify dependency versions via "==". I would not be surprised if Tensorflow actually requires particular numpy versions due to its high level of performance optimisation. So this could mean trouble when working with other libraries that rely on different specific numpy versions (possibly the case for arcpy, which wishes 1.20.1?).

We could not think of serious programming difficulties wrapping a Tensorflow-LSTM in a HydPy model. My gut feeling is that this would require similar work to implementing the LSTM equations from scratch. The main advantage of the wrapping approach would be that we could easily add further convenience functions to the HydPy model that rely on other TensorFlow functionalities, most notably the training algorithms. On the downside, the huge increase in dependencies (with specific version requirements) might result in a long-term rise in maintenance costs.