Indexing images at calculated offset

[fwindolf]

Hey, I have an optimization problem that relates a vector unknown to several images.

Given the unknown, I calculate positions in images of corresponding pixels. However, I do not find a way to make it work within Opt. bad argument #1 to 'Offset' expected 'number*' but found 'table' at list index 1

local N,C = Dim("N",0), Dim("C", 1)
local G = Graph("G", 0,                
               "iC", { C }, 1,
               "iD", { N }, 2)

-- First image 
local I1 = Image("Image1", float3, { N }, 3)
local D1 = Image("Depth1", float, { N }, 4)

-- Second image
local I2 = Image("Image2", float3, { N }, 5)
local D2 = Image("Depth2", float, { N }, 6)

local P = Unknown("Pose", float6, { C }, 7)

function transform(pt)
    -- some transformation of graph index pt with pose
    return i -- pixel in which an Image should be accessed
end

Energy(I1(G.iD) - I2(transform(G.iD)))

Is that a general limitation of your approach? I could imagine if the pixels would be used explicitly it would be a problem... Or ist it just not featured in the library yet?

[Mx7f]

I believe you are running into an implementation limitation, but cannot quite make sense of what you are trying to do. What are the Unknowns in this problem? Second, does transform take in a point, as the comments suggest, or an index (G.iD), as the code suggests?

[fwindolf]

I updated the example. The Pose was supposed to be Unknown.

Basically what I want to do is to calculate a position in which data is accessed. Imagine camera tracking, where the pose can be found by minimizing a photometric error. Therein, pixel (x,y) in the reference image corresponds to pixel (x+i, y+j) in the current image. i and j can be determined by the current pose estimate, which gets optimized by finding the "right" correspondences of pixels in both images. I want to find i and j (here i, as I have a linearized image) in the function transform. Is such a thing generally possible?

So to rephrase my question: Is it possible to calculate positions in which data is accessed?

If that does not work, can you imagine updating the graph after each iteration - whenever the pose changes to get the lastest correspondences?