Converting lemnatec scanner metadata to absolute geographic positions

[dlebauer]

From @dlebauer on May 25, 2016 20:54

From @solmazhajmohammadi on March 18, 2016 20:31

@czender, At the moment there is no accurate GPS at the gantry box. It is planned to add geographical coordinates to metadata, based on the fixed point on the ground and a RTK GPS location. Here are locations from each corner of the field that @rjstrand sent from his phone:

SE Corner 33° 04.470' N / -111° 58.485' W SW Corner. 33° 04.474' N / -111° 58.505' W NW Corner. 33° 04.592' N / -111° 58.505' W NE Corner. 33° 04.591' N / -111° 58.487' W

I have used linear scaling formula to transfer the coordinates, it should be enough for the current GPS location.

To determine the location of each pixel within the image, Extrinsic and Intrinsic calibration matrices are needed. 3 coordinate systems exist:

• Field coordinate frame (Xf , Yf , Zf)
• Camera coordinate (Xc , Yc , Zc)
• Image coordinate (Xp , Yp)

1 ) Extrinsic calibration matrix: The extrinsic calibration parameters specify the transformation from the field to the camera coordinates, and it is represented as, [Xc Yc Zc]'= Ro [Xf Yf Zf]'+ to Since there is no rotation (Ro), MEX is only translating vector to. to is transition vector from the control point (0,0,0) to camera position [Xc Yc Zc]. Considering that gantry is moving with constant speed in x direction, and metadata information shows the starting time and start position of a scan.
to = [+xg+Vx*(t) +yg +zg]'

Where (xg ,yg ,zg ) is the camera position in gantry box. For Hyperspectral camera (xg, yg ,zg ) is (1.9 , 0.855 , 0.635). Vx is velocity in x direction. t represents time difference. ()' is transpose operation.

2 ) Intrinsic calibration matrix: Notes:

• The geometric distortions that exist between the 272 bands are considered to be small.
• No rectification is done.
• No distortion and lens model is considered.
• Mirror angle change is not considered.

The intrinsic calibration parameters specify the transformation from the camera coordinate to the pixel coordinates. Coordinate transformation between camera plane and image plane: [Xp Yp f]'= A [ I 0 ] [Xc Yc Zc]' Simple orientation matrix is: αx = αy =α and it's focal length divided by pixel pitch. u0 and v0 denote the principal point (ideally center of image) For SWIR camera focal length is 24 mm, and pixel pitch is equal to 25 um. I dont know how much is γ. I have contacted the Headwall group and hopefully soon, they will provide me with more information on calibration of the camera. So, I will update your response soon.

Copied from original issue: terraref/documentation#9

Copied from original issue: terraref/reference-data#32

[dlebauer]

From @rjstrand on May 25, 2016 21:8

@dlebauer Thanks for the reminder. I just uploaded a .zip file to Box that contains RTK GIS measurements in various shape files. It is located in UA Site Information\GIS Shape Files. There are shape files for rail extents, foundation locations, sprinkler locations, a set of calibration points through the field, and a single point that represents the reference (0,0) for the current setup. Each shape file has an associated dbf file that can be opened in Excel by filtering for all files (.) and selecting it. I suggest saving the dbf in a separate file in native Excel format as the dbf is a component of the shape file assembly. The dbf has columns X, Y, Z that represent the coordinates in UTMs. The last two columns are lat-long coordinates in decimal degrees.

I hope this helps. Let me know if there is further information that you need. I will be back on site on May 31st.

• Bob

[dlebauer]

Thanks @rjstrand since the Box folder is restricted I uploaded them here for quick reference:

Gantry_Adjusted.zip

[dlebauer]

From @czender on May 25, 2016 21:18

We are at an impasse with calibrating the radiometric data, so this is a good time to try to implement the above geolocation. @FlyingWithJerome please read this thread and try to implement the suggested algorithm within JsonDealer, i.e., decorate the hyperspectral output file lat(x,y) and lon(x,y) where x and y are the existing pixel dimensions.

[dlebauer]

From @SolmazHajmohammadi-LemnaTec on May 25, 2016 22:59

@czender @FlyingWithJerome I think it would be better to wait for the triggered data from Hyperspectral camera. I think it is currently working for VNIR. Using external trigger will solve the problem, and you only need to convert from Gantry coordinate system to GPS coordinate(if you want GPS location)

[dlebauer]

@rjstrand, I think cz is referring to issues computing down welling radiance a from the ocean optics hyper spectral sensor https://github.com/terraref/reference-data/issues/30

[dlebauer]

I meant @SolmazHajmohammadi

[dlebauer]

From @rjstrand on May 26, 2016 14:49

@czender @dlebauer Just for clarification, the SWIR is functioning with the external trigger. The VNIR is the instrument that has been giving us problems with the external trigger. Additionally, VNIR is currently down. I will look at it when I return to Maricopa on 31 May.

[dlebauer]

From @caicai89- on May 27, 2016 21:16

When calculating the positions, we found that we need more information about the gantry and field.

1. Are the sensors fixed on the gantry?
2. In what direction can gantry move, or what is the direction of the track?
3. What is the exact point on gantry that have the position recorded in the metadata? And what is the coordinate origin, SE corner?
4. What is the exact point on camera that have the position recorded in the metadata? And what is the coordinate origin, Up-Left corner? What is the relation between camera box and gantry?
5. What is the field size? Are the edges of field straight North-South and East-West?

Also, when doing the position calculation. It is not safe to do the linear interpolation based on LatLon. So we transform the LatLon into UTM XY coordinate. However, the transformation has some errors. And we are not sure about how large the errors could be. Maybe we need some points with gantry locations and RTK LatLon to verify the error. But we are not sure about the necessity.

[dlebauer]

@caicai89- see some of the discussion here: https://github.com/terraref/reference-data/issues/25#issuecomment-219126297

I'll answer but will want someone from Lemnatec (@rjstrand or @TinoDornbusch?) to confirm.

First lets define rails, gantry is the big two-legged bit attached to rails and sensor box is moved around by the gantry.

1. Are the sensors fixed on the gantry?

The sensors are fixed to the sensor box. the sensor box moves EW on the gantry, the gantry moves N/S on the rails

1. In what direction can gantry move, or what is the direction of the track?

North / South

1. What is the exact point on gantry that have the position recorded in the metadata?

Not sure. I assume the SE corner of the sensor box and then the sensor position is relative to this

And what is the coordinate origin, SE corner?

I believe the center of the SE most concrete piling on the rails. [0,0,0] in the figure above

1. What is the exact point on camera that have the position recorded in the metadata?

I believe this is the center of the camera sensor, which I assume is pretty close is the center of the field of view (I think for z dimension, add focal distance to calculate center of field of view)

And what is the coordinate origin, Up-Left corner? What is the relation between camera box and gantry?

In https://github.com/terraref/reference-data/issues/25#issuecomment-219126297 @TinoDornbusch says "Field of view means a box with dimension in X and Y given with the camera X,Y as center of mass"

1. What is the field size? Are the edges of field straight North-South and East-West?

I believe the planted and scanned area is approximately 200 m N/S and 20 m E/W

Also, when doing the position calculation. It is not safe to do the linear interpolation based on LatLon. So we transform the LatLon into UTM XY coordinate. However, the transformation has some errors. And we are not sure about how large the errors could be. Maybe we need some points with gantry locations and RTK LatLon to verify the error.

Just say where / how many measurements you need.

[dlebauer]

@caicai89- see also https://github.com/terraref/reference-data/issues/32

[dlebauer]

From @TinoDornbusch on May 28, 2016 6:11

@dlebauer all correct. Godd drawing. May be the schematic drawing of the sensorbox helps to understand the Relations inside the box. 7100019_20160215_Kamerabox_Koordinaten.pdf

[dlebauer]

From @rjstrand on May 28, 2016 15:26

A few clarifications...

(0,0) - the X,Y origin is not at the SE foundation. If you look at the set of shape files, there is one called "reference". This is the single point representing the SE corner of the camera box when the gantry coordinates read (0,0). This location was set by our electrical/control contractor. We did not have a chance to modify this before the experiment started, so we decided to leave it as is for this year.

The accuracy of this GPS measurement is roughly +/- 1 cm. The reference point is offset from the SE foundation and is currently marked with a pin flag. I hope to determine the offsets from some fixed point on the SE foundation before I leave on 10 June.

This is are repeat, but I'll include to be complete... The shape file sets include a .dbf file that can be opened in Excel. It is best to save the .dbf as a separate file. Modifying it can have consequences if you attempt to import the shape set into a GIS package. The .dbf has X,Y columns that represent UTM coordinates. I believe this is zone 12N. "Lat" & "Lon" are in decimal degrees.

Additionally, there is a set of calibration points in the shape files... labeled "CalibrationPnts". Tino has the gantry coordinates that correspond to these GPS points. @TinoDornbusch, maybe you could upload those.

We will probably have time to get more points in the next two weeks, but I would suggest that we do it sooner than later. The sorghum will soon get in the way of the plumb bob that I use to mark points on the field. We also have to arrange to borrow the RTK and we have to mark the additional points between imaging runs. So, it can be done, but some things have to be arranged. Let us know if you need some additional points.

With regard to the reported position, David is correct. The reported position represents the location of the SE corner of the camera box. Instrument positions are then based on offsets. Tino can speak to those.

Current extents of the system are (3.8, 0.0, 0.0) to (207.3, 22.135, 5.5) all in meters. Again, this is the extents of motion for the SE corner of the camera box.

Finally, the gantry coordinate system is right-handed - Origin in the SE part of the field, X increases going from south to north, Y increases from east to the west.

[dlebauer]

From @caicai89- on June 8, 2016 14:11

@rjstrand With the .dbf file and gantry coordinates from @TinoDornbusch , I think we can check if the error. Where could I find the file? Do you mean current field is with size of 203.5 (207.3-3.8) * 22.135?

[dlebauer]

From @caicai89- on June 8, 2016 16:51

@rjstrand the gantry coordinate system is right-handed.

1. Is the camera box coordinate also right-handed?
2. to = [+xg+Vx*(t) +yg +zg]', for this formula, where can I find the t value? The sample metadata is the following. "gantry_system_variable_metadata": { "Time": "05/07/2016 15:58:43", "Position x [m]": "203.6", "Position y [m]": "2.499", "Position z [m]": "0", "Velocity x [m/s]": "0", "Velocity y [m/s]": "0", "Velocity z [m/s]": "0", "Camnera box light 1 is on": "False", "Camnera box light 2 is on": "False", "Camnera box light 3 is on": "False", "Camnera box light 4 is on": "False" },

[dlebauer]

From @gsrohde on June 9, 2016 17:15

@dlebauer @rjstrand I just thought I should clarify "right-handed" coordinate system vs. "left-handed" coordinate system. "right-handed" (also called "standard orientation") is what mathematicians usually use. I means that if the positive x-axis is going off to your right and the positive y-axis is going up, then the positive z-axis is coming toward you. What's unusual here is that in a geographical context, it seems more conventional to have the positive x-axis going toward the east and the positive y-axis going toward the north (and the positive z-axis going toward increasing altitude of course) whereas here you have the positive x-axis going toward the north and the positive y-axis going toward the west. In other words, the axis are rotated 90 degrees from what is usual. But both systems are right-handed coordinate systems.

[dlebauer]

From @rachelshekar on July 7, 2016 14:1

@caicai89- please update this issue

[dlebauer]

From @rachelshekar on July 29, 2016 20:27

@yanliu-chn - please update this issue

[dlebauer]

From @yanliu-chn on July 29, 2016 20:34

@caicai89- could you update the status of this issue? as we discussion, we need a timeline to deliver the first version of this feature.

[dlebauer]

From @caicai89- on July 29, 2016 21:34

This work will be closed once https://github.com/terraref/computing-pipeline/issues/101 is finished.

[dlebauer]

From @caicai89- on August 18, 2016 4:54

N(x)

^

|

|

|

W(y)<------SE

SE point has gantry position (3.8,0.0) and GPS position (33.0745,-111.97475).

When transforming a point from gantry coordinate to GPS position, we follow the following 3 steps.

1. Calculate the UTM position of the SE point
2. Calculate the UTM position of the point taking SE point as reference
3. Get the GPS position based on UTM

I tested the extreme situation, the NW point, and found some errors I cannot explain.

The NW point has gantry position (207.3, 22.135) and GPS position (33.0765333333,-111.9750833333). The UTM position for SE is (409017.7305875577, 3659968.4471026724). So the UTM position for NW is (409017.7305875577-22.135,3659968.4471026724+207.3-3.8) =(408995.5955875577, 3660171.9471026724)

However, the UTM position of NW point can also be calculated from GPS position of NW point, which is (408988.710283526, 3660194.1676153513).

As a result, there is a difference (6.885304031718988,-22.220512678846717).

This is too large!

Is there any wrong with the gantry position for SE (3.8,0.0) and NW (207.3,22.135)?

[dlebauer]

@rjstrand @smarshall-bmr any ideas?

[dlebauer]

From @max-zilla on August 18, 2016 18:52

@caicai89- You mentioned current metadata format during the meeting. I looked at several sensor metadata JSON files from August 17th for the geospatial format.

The gantry data (gantry_system_variable_metadata) is available with every sensor:

"gantry_system_variable_metadata": {
      "time": "08/17/2016 11:23:14",
      "position x [m]": "207.013",
      "position y [m]": "3.003",
      "position z [m]": "0.68",
      "speed x [m/s]": "0",
      "speed y [m/s]": "0.33",
      "speed z [m/s]": "0",
      "camera box light 1 is on": "True",
      "camera box light 2 is on": "True",
      "camera box light 3 is on": "True",
      "camera box light 4 is on": "True",
      "y end pos [m]": "22.135",
      "y set velocity [m/s]": "0.33",
      "y set acceleration [m/s^2]": "0.1",
      "y set decceleration [m/s^2]": "0.1"
    },

More detailed information (sensor_fixed_metadata) varies by sensor as you might expect. Here are the fields per sensor, limited to only include the fields we might use for this calculation.

stereoTop

"sensor_fixed_metadata": {
      "cameras alignment": "cameras optical axis parallel to XAxis, perpendicular to ground",
      "optics focus setting (both)": "2.5m",
      "optics apperture setting (both)": "6.7",
      "location in gantry system": "camera box, facing ground",
      "location in camera box x [m]": "0.877",
      "location in camera box y [m]": "2.276",
      "location in camera box z [m]": "0.578",
      "field of view at 2m in X- Y- direction [m]": "[1.857 1.246]",
      "bounding Box [m]": "[1.857     1.246]",
    },

cropCircle

"sensor_fixed_metadata": {
      "location in gantry system": "camera box, facing ground",
      "location in camera box x [m]": "0.480",
      "location in camera box y [m]": "1.920",
      "location in camera box z [m]": "0.6",
    },

co2Sensor

"sensor_fixed_metadata": {
      "location in gantry system": "camera box, facing ground",
      "location in camera box x [m]": "0.35",
      "location in camera box y [m]": "2.62",
      "location in camera box z [m]": "0.7",
    },

flirIrCamera

"sensor_fixed_metadata": {
      "location in gantry system": "camera box, facing ground",
      "location in camera box x [m]": "0.877",
      "location in camera box y [m]": "1.361",
      "location in camera box z [m]": "0.520",
      "field of view x [m]": "1.496",
      "field of view y [m]": "1.105",
    },

ndviSensor

"sensor_fixed_metadata": {
      "location in gantry system": "top of gantry, facing up, camera box, facing ground",
      "location in camera box x [m]": "0.33",
      "location in camera box y [m]": "2.50",
    },

priSensor

"sensor_fixed_metadata": {
      "location in gantry system": "top of gantry, facing up, camera box, facing ground",
      "location in camera box x [m]": "0.400",
      "location in camera box y [m]": "2.470",
    },

SWIR

"sensor_fixed_metadata": {
      "location in gantry system": "camera box, facing ground",
      "location in camera box x [m]": "0.877",
      "location in camera box y [m]": "2.325",
      "location in camera box z [m]": "0.635",
      "field of view y [m]": "0.75",
      "optics focal length [mm]": "25",
      "optics focus apperture": "2.0",
    },

scanner3DTop

N/A

So the following fields are available across most sensors:

• "location in camera box x [m]"
• "location in camera box y [m]"

I don't think z should matter in this case. But if your extractor uses these 2 fields along with the gantry_system_variable_metadata above, we should get geo data for all but 1 sensor listed here.

[dlebauer]

@max-zilla why don't you think z would matter?

For a fixed bounding box describing the field of view, if the box is on the ground the view at the top of the canopy is much smaller than the box. I think the field of view is always defined as 2m from the base of the sensor box see also original post / comments here and terraref/computing-pipeline#101

[dlebauer]

From @max-zilla on August 18, 2016 19:59

@dlebauer just meant not every sensor has a "location in camera box z" in the metadata like they do x and y. We have the gantry z position however. Perhaps this is already accounted for in @caicai89-'s script.

[dlebauer]

@max-zilla two sensors facing up (PRI and NDVI) neither have z nor field of view.

@TinoDornbusch and @SolmazHajmohammadi-LemnaTec @markus-radermacher-lemnatec is there a reason that scanner3DTop doesn't provide any information about sensor position or field of view in its metadata?

[TinoDornbusch]

@dlebauer Should be included in Fixed Meta data of the 3D Scanner:

    <entry9 type="KeyValueUnitTriple">
        <Key type="String" value="scanner east location in camera box x" />
        <Value type="String" value="2.070" />
        <Unit type="String" value="m" />
    </entry9>
    <entry10 type="KeyValueUnitTriple">
        <Key type="String" value="scanner east location in camera box y" />
        <Value type="String" value="0.306" />
        <Unit type="String" value="m" />
    </entry10>
    <entry11 type="KeyValueUnitTriple">
        <Key type="String" value="scanner east location in camera box z" />
        <Value type="String" value="1.135" />
        <Unit type="String" value="m" />
    </entry11>
    <entry12 type="KeyValueUnitTriple">
        <Key type="String" value="scanner west location in camera box x" />
        <Value type="String" value="2.070" />
        <Unit type="String" value="m" />
    </entry12>
    <entry13 type="KeyValueUnitTriple">
        <Key type="String" value="scanner west location in camera box y" />
        <Value type="String" value="2.726" />
        <Unit type="String" value="m" />
    </entry13>
    <entry14 type="KeyValueUnitTriple">
        <Key type="String" value="scanner west location in camera box z" />
        <Value type="String" value="1.135" />
        <Unit type="String" value="m" />
    </entry14>
    <entry15 type="KeyValueUnitTriple">
        <Key type="String" value="field of view y" />
        <Value type="String" value="0.800" />
        <Unit type="String" value="m" />