Problems Calculating Jacobian and Hessian in Converted PyTorch Model using L4CasADi

[nacereddinemoumai]

Hi @Tim-Salzmann,

I am working on applying Model Predictive Control (MPC) to regulate indoor temperature within a defined range while minimizing energy consumption for cooling. To achieve this, I trained a neural network that takes a window of past data as input—specifically, (disturbance[t-window], u[t-window], and x[t-window])—to predict x(t+1). I then constructed an L4CasADi model from the PyTorch model so that I can use this network as a prediction function with CasADi.

Initially, I reshaped my CasADi input to a vector and then reshaped it back to the expected dimensions inside my PyTorch model, as you recommended. However, I am encountering a problem when trying to compute the Jacobian and Hessian matrices for the L4CasADi model. Below is a snippet of the code I am using:

# model avec une fenetre
import l4casadi as l4c
class AirModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(input_size=3, hidden_size=50, num_layers=2, batch_first=True)
        self.linear = nn.Linear(50, 1)

    def forward(self, x):
        # Reshape input from (15, 1) to (1, 5, 3)
        #x = torch.tensor(x).reshape(1, 5, 3)
        x = x.reshape(1, 5, 3)
        print(x.shape)

        # Passer à travers le LSTM
        x, _ = self.lstm(x)
        x = self.linear(x[:, -1, :])
        return x

# import model without normalization

# link off pytorch model in drive : https://drive.google.com/file/d/1LOm-ePWmse8liARicJDstxIS5KhUWPQQ/view?usp=sharing

checkpoint = torch.load(
    "/content/pytorch_sansNormalisation_model.pt",
    map_location=torch.device('cpu'),

)
model = checkpoint["model"]

# Construct L4CasADi Model from PyTorch Model
from casadi import *
l4c_model = l4c.L4CasADi(model, model_expects_batch_dim=True, device='cpu')

#create a casadi function
import casadi as cs
d = cs.MX.sym('Text',5,1)          # distrubance (externel temp)
beta = cs.MX.sym('beta',5,1)       # control
Tin = cs.MX.sym('tin',5,1)         # room temp (state)

#reshaping CasADi input to a vector and, inside PyTorch model, reshaping it back to the expected dimensions Tensor[1,5,3]
input_window = cs.vertcat(d,beta,Tin)
print(input_window.shape) #(15,3)

x_next = l4c_model(input_window)
print(x_next .shape) #(1,1)
F = cs.Function("F", [input_window], [x_next])
df = cs.Function('dy', [input_window], [cs.jacobian(x_next, input_window)])
ddf = cs.Function('ddy', [input_window], [cs.hessian(x_next, input_window)[0]])

#extract window of state and control
d_window  = np.array([11.5648576 , 11.59015729, 11.61566968, 11.64138387, 11.66728878])
u_window = np.array([0.61402444, 0.61402444, 0.61402444, 0.61402444, 0.61402444])
y_window = np.array([19.6568679 , 18.79865259, 18.57413771, 18.49717339, 18.4556606 ])

x = cs.DM(np.concatenate([d_window, u_window, y_window]))
print(l4c_model(inp))
print(F(inp))
print(df(inp))
print(ddf(inp))

Here is the error that I am encountering:

Thanks in advance!

Best regards,

[Tim-Salzmann]

I tried your code. It works for me:

# model avec une fenetre
import torch.nn as nn
import torch
import l4casadi as l4c
class AirModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(input_size=3, hidden_size=50, num_layers=2, batch_first=True)
        self.linear = nn.Linear(50, 1)

    def forward(self, x):
        # Reshape input from (15, 1) to (1, 5, 3)
        #x = torch.tensor(x).reshape(1, 5, 3)
        x = x.reshape(1, 5, 3)
        print(x.shape)

        # Passer à travers le LSTM
        x, _ = self.lstm(x)
        x = self.linear(x[:, -1, :])
        return x

# import model without normalization

# link off pytorch model in drive : https://drive.google.com/file/d/1LOm-ePWmse8liARicJDstxIS5KhUWPQQ/view?usp=sharing

# checkpoint = torch.load(
#     "/content/pytorch_sansNormalisation_model.pt",
#     map_location=torch.device('cpu'),
#
# )
model = AirModel()#checkpoint["model"]

# Construct L4CasADi Model from PyTorch Model
from casadi import *
l4c_model = l4c.L4CasADi(model, model_expects_batch_dim=True, device='cpu')

#create a casadi function
import casadi as cs
d = cs.MX.sym('Text',5,1)          # distrubance (externel temp)
beta = cs.MX.sym('beta',5,1)       # control
Tin = cs.MX.sym('tin',5,1)         # room temp (state)

#reshaping CasADi input to a vector and, inside PyTorch model, reshaping it back to the expected dimensions Tensor[1,5,3]
input_window = cs.vertcat(d,beta,Tin)
print(input_window.shape) #(15,3)

x_next = l4c_model(input_window)
print(x_next .shape) #(1,1)
F = cs.Function("F", [input_window], [x_next])
df = cs.Function('dy', [input_window], [cs.jacobian(x_next, input_window)])
ddf = cs.Function('ddy', [input_window], [cs.hessian(x_next, input_window)[0]])

#extract window of state and control
d_window  = np.array([11.5648576 , 11.59015729, 11.61566968, 11.64138387, 11.66728878])
u_window = np.array([0.61402444, 0.61402444, 0.61402444, 0.61402444, 0.61402444])
y_window = np.array([19.6568679 , 18.79865259, 18.57413771, 18.49717339, 18.4556606 ])

x = cs.DM(np.concatenate([d_window, u_window, y_window]))
inp = np.zeros((15, 1))
print(l4c_model(inp))
print(F(inp))
print(df(inp))
print(ddf(inp))

[nacereddinemoumai]

Hi @Tim-Salzmann ,

Thank you for your response. However, I'm still encountering the same error, even though I'm running my code on Google Colab. Do you think that could be the cause? (I can't manage to install l4casadi on Windows.)

Thanks in advance!

[Tim-Salzmann]

Do you clone the git in Colab or pip install a release version?

[nacereddinemoumai]

@Tim-Salzmann Hi , I cloned the Git repository in Colab.

[Tim-Salzmann]

I tried your code. It works for me:

# model avec une fenetre
import torch.nn as nn
import torch
import l4casadi as l4c
class AirModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm = nn.LSTM(input_size=3, hidden_size=50, num_layers=2, batch_first=True)
        self.linear = nn.Linear(50, 1)

    def forward(self, x):
        # Reshape input from (15, 1) to (1, 5, 3)
        #x = torch.tensor(x).reshape(1, 5, 3)
        x = x.reshape(1, 5, 3)
        print(x.shape)

        # Passer à travers le LSTM
        x, _ = self.lstm(x)
        x = self.linear(x[:, -1, :])
        return x

# import model without normalization

# link off pytorch model in drive : https://drive.google.com/file/d/1LOm-ePWmse8liARicJDstxIS5KhUWPQQ/view?usp=sharing

# checkpoint = torch.load(
#     "/content/pytorch_sansNormalisation_model.pt",
#     map_location=torch.device('cpu'),
#
# )
model = AirModel()#checkpoint["model"]

# Construct L4CasADi Model from PyTorch Model
from casadi import *
l4c_model = l4c.L4CasADi(model, model_expects_batch_dim=True, device='cpu')

#create a casadi function
import casadi as cs
d = cs.MX.sym('Text',5,1)          # distrubance (externel temp)
beta = cs.MX.sym('beta',5,1)       # control
Tin = cs.MX.sym('tin',5,1)         # room temp (state)

#reshaping CasADi input to a vector and, inside PyTorch model, reshaping it back to the expected dimensions Tensor[1,5,3]
input_window = cs.vertcat(d,beta,Tin)
print(input_window.shape) #(15,3)

x_next = l4c_model(input_window)
print(x_next .shape) #(1,1)
F = cs.Function("F", [input_window], [x_next])
df = cs.Function('dy', [input_window], [cs.jacobian(x_next, input_window)])
ddf = cs.Function('ddy', [input_window], [cs.hessian(x_next, input_window)[0]])

#extract window of state and control
d_window  = np.array([11.5648576 , 11.59015729, 11.61566968, 11.64138387, 11.66728878])
u_window = np.array([0.61402444, 0.61402444, 0.61402444, 0.61402444, 0.61402444])
y_window = np.array([19.6568679 , 18.79865259, 18.57413771, 18.49717339, 18.4556606 ])

x = cs.DM(np.concatenate([d_window, u_window, y_window]))
inp = np.zeros((15, 1))
print(l4c_model(inp))
print(F(inp))
print(df(inp))
print(ddf(inp))

Did you try running exactly this?! It is slightly different from your original code.

[nacereddinemoumai]

@Tim-Salzmann I just re-ran this code, and I still have the same error.

[nacereddinemoumai]

Here is the link to the Colab notebook :(https://colab.research.google.com/drive/1qi8BSfPmW5vLfbjqPX3ivxhZuhhwI47C?usp=sharing)

[Tim-Salzmann]

At this point I am not sure why it fails in Colab. I suggest you try outside of colab.

Best Tim