Test Report is generated only for single map in unittest

[JuliaEmelianova]

Description The "Iterations" section in Test Report displays only iterations for tests having the same Map, however there are tests with different Maps inside the test case which was run wth a Python API script.

Steps to reproduce issue

1. Run latest Apollo with SVL Simulator 2021.1.1
2. Select the "API Only" Simulation in SVL Simulator and click "Run Simulation"
3. Run the following Python API script as unittest:

`import os import lgsvl import lgsvl.dreamview import time import unittest

MAX_POV_ROTATION = 5 TIME_LIMIT = 20
TIME_DELAY = 3 MIN_POV_TEST_SPEED = 4

SIMULATOR_HOST = os.environ.get("SIMULATOR_HOST", "127.0.0.1") SIMULATOR_PORT = int(os.environ.get("SIMULATOR_PORT", 8181)) BRIDGE_HOST = os.environ.get("BRIDGE_HOST", "127.0.0.1") BRIDGE_PORT = int(os.environ.get("BRIDGE_PORT", 9090))

modules = [ 'Localization', 'Perception', 'Transform', 'Routing', 'Prediction', 'Planning', 'Traffic Light', 'Control' ]

class TestVehicleFollowing(unittest.TestCase):

def test_vehFlw_forward_BorregasAve_BoxTruck(self):

    collisions = []
    def on_collision(agent1, agent2, contact):
        collisions.append([agent1, agent2, contact])

    MAX_POV_SPEED = MIN_POV_TEST_SPEED
    sim = lgsvl.Simulator(SIMULATOR_HOST, SIMULATOR_PORT)
    if sim.current_scene == "BorregasAve":
        sim.reset()
    else:
        sim.load("BorregasAve")

    sim.set_time_of_day(18)

    egostate = lgsvl.AgentState()
    egoX = 53.3642539978027
    egoY = -3.04669451713562
    egoZ = -34.0701026916504

    egostate.transform = sim.map_point_on_lane(lgsvl.Vector(egoX, egoY, egoZ))
    egostate.transform.rotation.x = 0
    egostate.transform.rotation.y = 104.415573120117    
    egostate.transform.rotation.z = 0

    ego = sim.add_agent("2e9095fa-c9b9-4f3f-8d7d-65fa2bb03921", lgsvl.AgentType.EGO, egostate) #config ID for Apollo 6.0(modular testing)

    ego.connect_bridge(BRIDGE_HOST, BRIDGE_PORT)

    POVState = lgsvl.AgentState()
    npcX = 69.6873321533203
    npcY = -3.3256733417511
    npcZ = -37.787654876709

    POVState.transform = sim.map_point_on_lane(lgsvl.Vector(npcX, npcY, npcZ))
    POVState.transform.rotation.x = 0
    POVState.transform.rotation.y = 109.090927124023
    POVState.transform.rotation.z = 0

    POV = sim.add_agent("BoxTruck", lgsvl.AgentType.NPC, POVState)

    collisions = []
    ego.on_collision(on_collision)

    dv = lgsvl.dreamview.Connection(sim, ego, BRIDGE_HOST)
    dv.set_hd_map('BorregasAve')
    dv.set_vehicle('Lincoln2017MKZ')

    dv.set_destination(355.711364746094,-114.557281494141,-7.63640594482422)
    dv.setup_apollo(355.711364746094, -114.557281494141, modules)

    sim.run(TIME_DELAY)  # The EGO should start moving first

    POV.follow_closest_lane(True, MAX_POV_SPEED, False)        
    sim.run(TIME_LIMIT)
    sim.close()

    self.assertEqual(len(collisions), 0)

def test_vehFlw_forward_SingleLaneRoad_Jeep(self):

    collisions = []
    def on_collision(agent1, agent2, contact):
        collisions.append([agent1, agent2, contact])

    MAX_POV_SPEED = MIN_POV_TEST_SPEED

    sim = lgsvl.Simulator(SIMULATOR_HOST, SIMULATOR_PORT)
    if sim.current_scene == "SingleLaneRoad":
        sim.reset()
    else:
        sim.load("SingleLaneRoad")

    sim.set_time_of_day(12)

    egostate = lgsvl.AgentState()
    egoX = -43.4099502563477
    egoY = 0
    egoZ = -1.82721841335297

    egostate.transform = sim.map_point_on_lane(lgsvl.Vector(egoX, egoY, egoZ))
    egostate.transform.rotation.x = 0
    egostate.transform.rotation.y = 89.9640731811523    
    egostate.transform.rotation.z = 0

    ego = sim.add_agent("2e9095fa-c9b9-4f3f-8d7d-65fa2bb03921", lgsvl.AgentType.EGO, egostate) #config ID for Apollo 6.0(modular testing)

    ego.connect_bridge(BRIDGE_HOST, BRIDGE_PORT)

    POVState = lgsvl.AgentState()
    npcX = -31.1504936218262
    npcY = 0
    npcZ = -1.81953167915344

    POVState.transform = sim.map_point_on_lane(lgsvl.Vector(npcX, npcY, npcZ))
    POVState.transform.rotation.x = 0
    POVState.transform.rotation.y = 89.9640731811523
    POVState.transform.rotation.z = 0

    POV = sim.add_agent("Jeep", lgsvl.AgentType.NPC, POVState)

    collisions = []
    ego.on_collision(on_collision)

    dv = lgsvl.dreamview.Connection(sim, ego, BRIDGE_HOST)
    dv.set_hd_map('SingleLaneRoad')
    dv.set_vehicle('Lincoln2017MKZ')

    dv.set_destination(94.2320556640625, -2.09937024116516, 0)
    dv.setup_apollo(94.2320556640625, -2.09937024116516, modules)

    sim.run(TIME_DELAY)  # The EGO should start moving first

    POV.follow_closest_lane(True, MAX_POV_SPEED, False)        
    sim.run(TIME_LIMIT)
    sim.close()

    self.assertEqual(len(collisions), 0)`

4. Stop "API Only" Simulation in SVL Simulator.
5. View generated Test Report navigating to Test Results tab in SVL Simulator.

Expected results

3. -> 2 tests ran successfully
4. -> Test Report has two iterations in the "Iterations" section

Actual results

3. -> 2 tests ran successfully
4. -> Test Report has one iteration in the "Iterations" section

System details Intel(R) Core(TM) i9-7900X CPU @ 3.30GHz Nvidia GeForce GTX 1080 Ti (GPU memory 11GB) Ubuntu 20.04.2 LTS Apollo master (latest Apollo was cloned at 2021-06-05) Docker version: apolloauto/apollo:dev-x86_64-18.04-20210517_1712 SVL Simulator 2021.1.1

[EricBoiseLGSVL]

Hmm, we create a test report with the api every time api.reset() is called in the logic. This might be a bug. Please add api.reset to your python between iterations and see if it fixes it. Thanks for the issue.

[JuliaEmelianova]

@EricBoiseLGSVL Do you mean to add sim.reset() in my python between iterations? If yes, it does not help (((

[EricBoiseLGSVL]

ok this is a bug, we will look into. Thanks for posting

[JuliaEmelianova]

Also thanks for your comments )