CG method produces flattened outputs

[calvinmccarter]

In the following optimization over a 5x5 matrix, we see that CG works, but it produces a flattened version of the correct result:

import torch
import torchmin as tm

Y = torch.eye(5) + torch.normal(0, 0.1, size=(5, 5))
X = torch.eye(5) + torch.normal(0, 0.1, size=(5, 5))

def myfun(B):
    obj = torch.sum((Y - X @ B) ** 2)
    return obj
B_init = torch.eye(5)

res = tm.minimize(
    myfun,
    B_init,
    method="l-bfgs",
    max_iter=10,
    disp=3,
)    
B_lbfgs = res.x.numpy()
print(B_lbfgs)

res = tm.minimize(
    myfun,
    B_init,
    method="cg",
    max_iter=10,
    disp=3,
)    
B_cg = res.x.numpy()
print(B_cg)

produces:

initial fval: 0.5372
iter   1 - fval: 0.2338
iter   2 - fval: 0.0060
iter   3 - fval: 0.0004
iter   4 - fval: 0.0000
iter   5 - fval: 0.0000
iter   6 - fval: 0.0000
iter   7 - fval: 0.0000
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 7
         Function evaluations: 8
[[ 1.1526148   0.0012578   0.02067411 -0.04399937  0.01775152]
 [ 0.23119657  1.0350143  -0.26021296 -0.06592292 -0.01872858]
 [ 0.00481756 -0.2991033   0.83702314  0.0673951  -0.13733503]
 [-0.26384994 -0.02407601  0.06922133  0.88163185  0.12506227]
 [ 0.14160359  0.25184008 -0.1872299  -0.07943757  1.1720723 ]]
initial fval: 0.5372
iter   1 - fval: 0.0450
iter   2 - fval: 0.0016
iter   3 - fval: 0.0000
iter   4 - fval: 0.0000
iter   5 - fval: 0.0000
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 5
         Function evaluations: 11
[ 1.1526111   0.00125925  0.02067351 -0.04400145  0.01775517  0.2311999
  1.0350143  -0.26021218 -0.06592047 -0.01873206  0.00481421 -0.2991013
  0.83702505  0.06739324 -0.13733163 -0.2638538  -0.02407451  0.0692211
  0.8816287   0.12506813  0.14159894  0.25184155 -0.18722953 -0.07944129
  1.1720781 ]
​```

[rfeinman]

Hi @calvinmccarter - I am not able to reproduce the flattened output issue on my machine (see below). Can you please pull the latest repo and try again? Please use fixed matrices X/Y or provide a random seed. Thanks for pointing this out.

Code:

torch.manual_seed(191)
Y = torch.eye(5) + torch.normal(0, 0.1, size=(5, 5))
X = torch.eye(5) + torch.normal(0, 0.1, size=(5, 5))

def myfun(B):
    obj = torch.sum((Y - X @ B) ** 2)
    return obj

B_init = torch.eye(5)

res = tm.minimize(
    myfun,
    B_init,
    method="cg",
    max_iter=10,
    disp=3,
)
print(res.x.numpy())

Output:

initial fval: 0.3555
iter   1 - fval: 0.0245
iter   2 - fval: 0.0025
iter   3 - fval: 0.0002
iter   4 - fval: 0.0000
iter   5 - fval: 0.0000
Optimization terminated successfully.
         Current function value: 0.000000
         Iterations: 5
         Function evaluations: 11
[[ 0.7799 -0.114   0.0043 -0.1361 -0.0143]
 [-0.0608  0.8359  0.1543  0.1881  0.0663]
 [-0.0425 -0.0986  0.8894 -0.0299  0.0949]
 [-0.1785 -0.0352 -0.0761  1.0242 -0.0109]
 [-0.0007  0.0523 -0.1453  0.0983  1.0643]]

[calvinmccarter]

Updating from 8c5da1c7 to latest fixed this -- thanks so much!