QUERY: optimize: Doubts about `minimize`

[Always-Stude]

Why do functions with the same functionality produce different results when computed on a CPU versus a GPU? I defined a function using numpy arrays and rewrote it in PyTorch. The test data confirmed that the results were consistent, but there is a significant difference in results when using minimize optimization. This is my function

def S1fun_pm1(b,tdata,tt): # rho = np.exp(b[2]) # d = 1 # sigma1 = np.exp(b[0]) # sigma0 = np.exp(b[1])

#
NN = 2 * np.max(tdata)

#
num = 2 * NN
#
Q = np.zeros((num, num))
#
one = np.ones((num,1))

# 
indices = np.arange(NN - 1)  # 从 0 到 NN-2 的数组
values = (indices + 1) * d  # 计算 (i + 1) * d

# 
Q[indices + 1, indices] = values
Q[indices + NN + 1, indices + NN] = values
Q[indices + NN, indices + NN + 1] = rho

# 
np.fill_diagonal(Q[:NN, NN:], sigma1)

# 
np.fill_diagonal(Q[NN:, :NN], sigma0)

temp = np.dot(Q, one)
#
np.fill_diagonal(Q, -temp)

#
Qmod = Q.copy()
Qmod[:, -1] = 1

#
vec = np.zeros((1,num))
#
vec[0, -1] = 1
#
vec = vec.flatten()

if np.linalg.matrix_rank(Qmod) == Qmod.shape[0]: 
    # 
    ssd=scipy.linalg.solve(Qmod.T,vec)
else:       
    # 
    Qmod_pinv = scipy.linalg.pinv(Qmod)
    #
    ssd= vec @ Qmod_pinv

# 
dist = ssd[:NN] + ssd[NN:]
dist=dist+1e-10

#
S1=0 
for i in range(len(tdata)) :
    S1=S1- tt.iloc[i] * np.log(dist[i])

return S1

def S1fun_pm1_pytorch(params, tdata, tt): dtype=torch.float32 device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') b = torch.tensor(params, dtype=dtype,device=device)
tt_tensor = torch.tensor(tt, dtype=dtype,device=device)

rho = torch.exp(b[2])
d = torch.tensor(1.0,device=device)
sigma1 = torch.exp(b[0])
sigma0 = torch.exp(b[1])

NN = 2 * torch.max(torch.tensor(tdata, dtype=torch.int32,device=device)).item()

num = 2 * NN

Q = torch.zeros((num, num), dtype=dtype,device=device)
one = torch.ones(num, 1, dtype=dtype,device=device)

indices = torch.arange(NN - 1,device=device)
values = (indices + 1) * d
values = values.to(dtype=dtype)

Q[indices + 1, indices] = values
Q[indices + NN + 1, indices + NN] = values
Q[indices + NN, indices + NN + 1] = rho

Q[:NN, NN:].fill_diagonal_(sigma1)
torch.diagonal(Q[NN:, :NN]).fill_(sigma0)

# 
temp = torch.matmul(Q, one)
# 
Q.diagonal().copy_(-temp.flatten())

Qmod = Q.clone()
Qmod[:, -1] = 1

vec = torch.zeros(num, dtype=dtype,device=device)
vec[-1] = 1

if torch.linalg.matrix_rank(Qmod) == Qmod.size(0):
    ssd = torch.linalg.solve(Qmod.T, vec)
else:
    Qmod_pinv = torch.linalg.pinv(Qmod)
    ssd = torch.matmul(vec, Qmod_pinv)

dist = ssd[:NN] + ssd[NN:]
dist=dist+1e-10

S1 = torch.tensor(0.0, dtype=dtype,device=device)
# 
for i in range(len(tdata)):
    # 
    S1 = S1 - tt_tensor[i] * torch.log(dist[i])

return S1.item()

[andyfaff]

Please format your code, it's hard to figure out what you're doing. It's hard to understand from your post what you think is going wrong. Furthermore, there's no call to minimize here, so you're not actually demonstrating an issue.

[Always-Stude]

"I apologize for not being clear. What I mean is that I have two functions, S1fun_pm1_pytorch and S1fun, which produce the same results for the same data. However, result = minimize(S1fun_pm1_pytorch, np.log([lain, gain, vin]), args=(tdata, tt), method='nelder-mead') and result = minimize(S1fun, np.log([lain, gain, vin]), args=(tdata, tt), method='nelder-mead') yield different results."

------------------ 原始邮件 ------------------ 发件人: "Andrew @.>; 发送时间: 2024年9月8日(星期天) 下午2:05 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [scipy/scipy] Doubts about minimize optimization (Issue #21521)

Please format your code, it's hard to figure out what you're doing. It's hard to understand from your post what you think is going wrong. Furthermore, there's no call to minimize here, so you're not actually demonstrating an issue.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[ilayn]

@Always-Stude Please spend some effort on making clear issues. We cannot guess what you are trying to do based on your copy/pasted code and we don't use PyTorch. At the very least make sure that you are formatting your code properly in the Github issue. Otherwise we don't have much choice other than closing this issue.

[Always-Stude]

Will there be a significant difference in the final results of minimize when using double precision versus single precision?

------------------ 原始邮件 ------------------ 发件人: "Ilhan @.>; 发送时间: 2024年9月8日(星期天) 下午2:48 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [scipy/scipy] Doubts about minimize optimization (Issue #21521)

@Always-Stude Please spend some effort on making clear issues. We cannot guess what you are trying to do based on your copy/pasted code and we don't use PyTorch. At the very least make sure that you are formatting your code properly in the Github issue. Otherwise we don't have much choice other than closing this issue.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[andyfaff]

Double and single precision can have an effect, it would depend on the problem. When you say there's a big difference in the result, what do you classify as a big difference?

[Always-Stude]

When using single precision, the optimal parameters are [2.78, 2.54, 3.55], but when using double precision, the optimal parameters are [8.69, 2.78, 4.55].

------------------ 原始邮件 ------------------ 发件人: "Andrew @.>; 发送时间: 2024年9月8日(星期天) 晚上6:03 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [scipy/scipy] Doubts about minimize optimization (Issue #21521)

Double and single precision can have an effect, it would depend on the problem. When you say there's a big difference in the result, what do you classify as a big difference?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[Always-Stude]

Because my calculations involve a large number of exponentials and logarithms

------------------ 原始邮件 ------------------ 发件人: "Andrew @.>; 发送时间: 2024年9月8日(星期天) 晚上6:03 收件人: @.>; 抄送: @.>; @.>; 主题: Re: [scipy/scipy] Doubts about minimize optimization (Issue #21521)

Double and single precision can have an effect, it would depend on the problem. When you say there's a big difference in the result, what do you classify as a big difference?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>