opendilab / LMDrive

[CVPR 2024] LMDrive: Closed-Loop End-to-End Driving with Large Language Models
Apache License 2.0
526 stars 48 forks source link

How to generate corresponding instructions in a new Town? Such as Town13? #19

Closed kevinchiu19 closed 4 months ago

kevinchiu19 commented 5 months ago

Hi!

I found that both './tools/data_parsing' and 'InstructionPlanner' have some predefined values, such as some coordinates corresponding to each Town. If these values ​​are in a new Town, such as leaderboard2.0's Town13, how can these be generated?

Looking forward to your reply, thank you~

For example:

  1. leaderboard/team_code/planner.py 'self.highway_mapping = {"Town04":[[-487.84,361.47,2.84,44.26],[-19.73,18.43,-279.10,278.82],[94.88,333.41,-360.95,-398.73],[-376.92,-93.35,400.16,440.26],[-517.71,-478.04,37.87,319.51]], \ "Town05":[[-257.43,-217.99,-179.75,175.86],[184.14,218.66,-175.28,174.62],[-204.68,162.79,-217.05,-181.86],[-210.39,179.10,182.47,218.67]], \ "Town06":[[-302.75,625.72,-8.10,-26.18],[-278.19,651.95,35.46,54.74],[-286.53,649.00,135.70,155.15],[-323.99,647.78,236.17,254.11],[656.33,673.07,12.02,228.79],[-372.63,-359.37,13.01,230.13]] }'

  2. tools/data_parsing/turn_rules.py class TurnFalse(Turn): def init(self, direction): super().init() self.sample_range = 100 self.direction = direction self.direction_index_mapping = {"left": 0, "right": 1} self.direction_command_mapping = {"left": 2, "right": 1} self.total_range = 128 self.Tjunction_loc_mapping = {"1": [[[74.31,2.50,2],[140.33,2.33,2],[320.09,2.32,2],[92.69,72.83,2],[173.38,55.50,2],[334.70,41.42,2],[92.29,147.24,2],[334.53,114.74,2],[92.81,215.65,2],[334.66,179.34,2],[107.98,326.93,2],[354.03,326.60,2]],[[107.74,-1.54,2],[173.88,-1.73,2],[352.97,-1.82,2],[88.03,41.60,2],[139.77,59.79,2],[339.00,73.74,2],[88.19,115.30,2],[339.00,148.54,2],[88.08,181.66,2],[338.96,214.46,2],[73.55,330.99,2],[319.83,331.13,2]]],\ "2": [[[-3.23,205.09,2],[29.32,192.01,2],[117.29,192.16,2],[189.37,175.07,2],[45.80,253.94,2],[148.54,237.29,2],[189.51,225.80,2],[58.12,302.90,2]],[[-7.67,176.45,2],[58.12,187.84,2],[150.38,187.84,2],[193.96,204.03,2],[41.60,223.79,2],[119.50,241.48,2],[193.75,253.49,2],[28.17,307.19,2]]],\ "3": [[[-18.78,-195.73,3],[149.05,-151.50,3],[84.34,-58.19,3],[147.97,-88.64,3],[103.13,-4.93,3],[172.18,-3.80,3],[233.99,-22.35,3],[232.47,37.33,3],[146.53,62.99,2],[22.64,195.33,3]],[[26.14,-205.40,3],[153.95,-116.42,2],[152.70,-58.14,2],[79.87,-88.97,2],[58.55,6.42,3],[129.03,7.83,3],[243.73,27.13,3],[190.25,58.98,2],[241.73,86.48,3],[-24.61,204.59,3]]],\ "4": [[[179.01,-369.13,5],[242.51,-307.31,2],[81.06,-173.40,2],[148.32,-172.98,2],[388.35,-193.76,5],[272.62,-122.34,2],[386.55,-90,5]],[[272.72,-310.59,2],[44.36,-169.96,2],[116.76,-169.19,2],[239.88,-118.47,2]]],\ "5": [[[-149.31,-136.65,3],[-101.78,145.94,3],[7.10,-189.40,5],[55.46,191.61,5],[33.34,162.25,3],[151.32,-18.97,2],[36.20,-129.62,3]],[[-103.21,-143.83,3],[-149.73,153.17,3],[30.16,-165.07,3],[26.36,125.75,3],[155.49,17.09,2]]],\ "6": [[],[[665.92,66.59,5],[664.88,168.47,5]]],\ "7": [[[-197.95,-147.58,2],[-85.62,-158.09,2],[-4.47,-171.80,2],[-5.13,-121.57,2],[-98.38,-50.71,2],[-5.83,-76.91,2],[-199.07,-22.44,2],[-99.01,15.57,2],[-199.03,64.57,2],[-138.96,48.95,2],[10.28,58.64,2],[-95.56,116.56,2]],[[-202.47,-174.34,2],[-202.13,-49.13,2],[-203.63,36.88,2],[-122.42,117.61,2],[-0.51,-145.93,2],[-1.19,-50.03,2]]],\ "10": [[[64.47,64.40,3]],[[19.83,71.62,3]]]} self.intersection_loc_mapping = {"1": [[90.30,0.51,25],[156.93,1.09,25],[336.86,1.39,25],[337.33,326.93,40],[90.95,327.01,40],[92.37,196.73,30],[91.87,131.36,30],[92.17,57.97,25],[156.05,55.61,25],[335.12,57.68,25],[335.78,130.58,30],[336.32,196.97,30]],\ "2": [[43.31,304.10,30],[-5.34,190.45,30],[192.52,189.99,30],[190.68,239.30,30],[134.06,238.50,30],[43.77,238.49,30],[43.51,190.68,30],[133.09,189.44,30]],\ "3": [[3.93,-199.79,35],[236.43,0.77,30],[237.27,61.02,30],[-1.39,196.76,35],[151.53,-132.98,30],[149.59,-72.75,30],[80.90,-74.44,30],[148.68,-5.98,30],[78.58,-5.19,30],[169.12,64.11,30],[-226.23,-2.30],[-223.15,103.26],[83.79,-257.12],[157.84,-256.18],[-146.60,-1.44],[-84.86,133.58],[-2.82,132.36],[-81.72,-137.82],[2.44,-135.59],[83.89,-135.75],[85.39,-199.39],[153.65,-198.61]],\ "4": [[257.15,-308.29,25],[256.30,-122.12,25],[128.78,-172.50,30],[61.36,-174.60,25],[15.01,-172.33,25],[205.67,-364.69,30],[393.50,-171.28,25],[381.09,-67.54,30],[203.01,-309.33],[202.12,-247.58],[200.61,-171.29],[256.94,-248.01],[256.49,-170.93],[313.26,-248.37]],\ "5": [[34.01,-182.82,20],[40.02,-147.67,20],[153.47,-0.52,25],[40.85,142.48,25],[30.24,198.96,30],[-126.12,-137.57,20],[-124.06,148.97,25],[-268.82,-1.19,30],[-189.88,-90.40,30],[-189.49,0.79,30],[-190.41,89.65,30],[-127.13,-89.45,30],[-126.58,1.19,30],[-125.56,89.59,30],[-49.85,-89.76,30],[-49.13,0.86,30],[-49.28,89.65,30],[31.55,-89.33,30],[29.53,0.28,30],[29.20,89.69,30],[101.55,-0.07,30]],\ "6": [[662.70,41.96,40],[662.41,144.54,40],[-1.63,-17.53,40],[-1.84,49.77,40],[-0.50,141.78,40],[1.29,244.84,40]],\ "7": [[-197.22,-161.53,40],[-1.85,-238.09,40],[67.08,-1.04,35],[67.25,60.09,35],[-109.01,113.97,35],[-198.61,49.24,25],[-198.65,-36.34,25],[-151.27,48.35,25],[-100.17,-34.76,15],[-100.46,-63.77,15],[-101.47,-96.25,10],[-85.31,-111.70,10],[-73.35,-159.14,30],[-3.43,-159.27,30],[-4.05,-107.83,15],[-4.45,-64.86,20],[-4.79,57.83,35],[-101.62,53.08],[-3.78,-1.48],[-150.54,-35.13]],\ "10": [[-44.76,-55.94,30],[96.00,-21.14,30],[96.84,68.01,30],[-46.40,127.21,30],[-99.79,19.70,30],[-38.44,65.96,30],[41.59,66.94,20],[41.08,30.14,20],[-47.38,19.22]]}

deepcs233 commented 5 months ago

Hi! These specific coordinates are manually annotated by the authors :)

If you also need to obtain these values, I could give you some suggestions:

  1. generate the corresponding .xodr map file by the Carla Python API
  2. open the website: https://odrviewer.io/ , and upload the .xodr map file to this website. This website can help you visualize the Carla map
  3. find the coordinates that you need. Some coordinate transforms may need to be applied. I remember that the y-axis value should multiply -1 when used in the Carla simulator.
kevinchiu19 commented 5 months ago

Thank you for your detailed reply. I have studied and tried it based on it these days, and I think I understand it.

In addition, I found a problem. In the town map used in the currently provided project(Town01, Town10HD...), in the ''geoReference'' field of the a file, the displacements of ''lat_0=0 +lon_0=0'' are all 0, but in the town13 used in Leaderboard2.0, ''lat_0=41.09374957104912 + lon_0=-81.59374804575259'', which will cause some problems:

In calculations involving "gps to location", such as those used in the following two files, I think the mean and scale used in different towns will change with lat_ref and lon_ref:

  1. leaderboard/team_code/lmdriver_agent.py
  2. leaderboard/team_code/planner.py self.mean = np.array([0.0, 0.0]) # for carla 9.10 self.scale = np.array([111324.60662786, 111319.490945]) # for carla 9.10

I referred to the code in leaderboard (leaderboard/leaderboard/utils/route_manipulation.py -> _location_to_gps), I learned from it and worked backwards, and got what I think is the correct conversion method:

` def _gps_to_location(lat_ref, lon_ref, lat, lon, z): """ 参考自 leaderboard/leaderboard/utils/route_manipulation.py -> _location_to_gps """

EARTH_RADIUS_EQUA = 6378137.0   # pylint: disable=invalid-name
scale = math.cos(lat_ref * math.pi / 180.0)
mx = scale * lon_ref * math.pi * EARTH_RADIUS_EQUA / 180.0
my = scale * EARTH_RADIUS_EQUA * math.log(math.tan((90.0 + lat_ref) * math.pi / 360.0))

mx_ = lon / 180.0 * (math.pi * EARTH_RADIUS_EQUA * scale)
my_ = math.log(math.tan((lat + 90.0) * math.pi / 360.0)) * (EARTH_RADIUS_EQUA * scale)
location_z = z

location_x = mx_ - mx
location_y = -my_ + my

return {'x': location_x, 'y': location_y, 'z': location_z}

`

deepcs233 commented 4 months ago

Hi! Sorry! I'm not familiar with the leaderboard 2.0. I remembered that the coordinate system is a little special(The GPS value was corrupted by noise. If the AV didn't move, the GPS of it was still changing?).

I think the best way is to check your config file by running it in the simulator.

kevinchiu19 commented 4 months ago

Thanks!