Camera rotation

[Hokyjack]

Hello Braden @bradenhurl. I am extending your projetct to include multiple cameras, however, I came to a problem where when the segmentations and detected objects are being exported, it doesn't take into account the camera rotation and it breaks the exports and detection boxes - I guess it just takes the collector car direction since it consideres only one camera facing directly on the hood. Can you please point me into direction what should I look into when fix this issue and take the actual rotation from the camera? Thank you so much in advance!

[Hokyjack]

the issue comes, when I try to rotate the camera inside Scenario::setRenderingCam method: CAM::SET_CAM_ROT(camera, rotation.x, rotation.y, rotation.z + 10, 0);

Then it doens't detect pedestrians and many car boundings are either missing or wrong.

[bradenhurl]

I think this may be due to the frame buffer not being rendered before collecting. I had many issues synchronizing while developing. There are a couple things to note: 1. the frame buffer needs to be flushed multiple times to ensure synchronization (see CAM::RENDER_SCRIPT_CAMS(TRUE, FALSE, 0, FALSE, FALSE)) 2. The game needs to be unpaused (can set speed to 0 so there will be minimal difference between frames, although not exact.

If you look at the function generateSecondaryPerspective and it's subfunctions you will see the unpause, render three times, pause, collect procedure that seemed to work best during my experimentation. Hopefully this helps, let me know if you're still having trouble.

[Hokyjack]

Hello, I don't know if this is because of the synchronization, because it happens only when i change camera rotation different from the default heading rotation.z. If I change the camera FOV or position it still works, but chaning the rotation makes the issues - black boxes in the instanceSegmentations and messed up annotations.

Thanks!

[bradenhurl]

Have you had any luck fixing this issue?

[Hokyjack]

Have you had any luck fixing this issue?

unfortunately no, I still don't know what is causing issue. If i just translate/offset the camera it's ok, if I rotate the camera (even by just 1 degree) it breakes the annotations with that black boxes as shown in above images

[bradenhurl]

Okay, I'll have time to debug in a few days. I'll get back to you next week.

[Hokyjack]

Awesome, thank you very much for checking it! If we can sort this issue I can then refactor my code and push a branch with my experimantal multi-camera setup here.

[bradenhurl]

Hey, sorry for the delay. I've been super busy at work. I think I found the issue. There are multiple places where GET_ENTITY_ROTATION is being used to set the camera parameters from the vehicle direction. The ones which should definitely match are:

1. Scenario.cpp -> 593 (this is where you changed it in setRenderingCam)
2. Within the submodule GTAV_ObjectDetection/ObjectDetection.cpp -> line 2117 (the s_camParams are used all over the place for calculations)

Let me know if this works! If not I can take another look.

[Hokyjack]

Hey, sorry for the delay. I've been super busy at work. I think I found the issue. There are multiple places where GET_ENTITY_ROTATION is being used to set the camera parameters from the vehicle direction. The ones which should definitely match are:

1. Scenario.cpp -> 593 (this is where you changed it in setRenderingCam)
2. Within the submodule GTAV_ObjectDetection/ObjectDetection.cpp -> line 2117 (the s_camParams are used all over the place for calculations)

Let me know if this works! If not I can take another look.

searching for usage of ObjectDetection::setCamParams, I haven't found a single usage of this method. The only one used is withing Scenario class.

This is so strange. it has to be caused by rotation somewhere, because even if I just use only 1 main camera and rotate that one I get the messed up image you can see on the right:

Do you have some other tips? Or maybe some hint how to debug this? I was thinking about some realtime overlaying the segmentation buffer on top of game or something, because turning off game chaning line of code, compiling, turning game on waiting for game and mod to load 5 minutes again, shutting it checking exported images... it drives me crazy 😵

[bradenhurl]

Hmm, you're correct. A few things to help debug:

1. Can you show your code?
2. Did you simply modify generateSecondaryPerspective to obtain secondary perspectives from different views?
3. Can you log the camera parameters where they are set in the scenario then when they are used in ObjectDetection? Are they the same?
4. Can you show the depth map associated with the images?

[bradenhurl]

I feel your pain with debugging. This was my life for a while. I recommend reading through and triple checking before testing. The overlays are an option. Other people have made the overlays work. It's a bit more difficult with the segmentation as it obtains the data from the graphics buffer and the buffers are refreshed a few times.

[Hokyjack]

Here is the stecil and depthmap of that right camera: They both look ok.

Yes I modified a extraPerspective, basically everything just happens in these tow functions:

void Scenario::generateExtraCameraPerspective(Vector3 cameraRotation, Vector3 cameraPosition, const char* cameraName) {

    m_pObjDet->currentExtraCameraName = cameraName;
    // todo change to extra rotations

    setRenderingCamWithRotAndPos(m_ownVehicle, CAM_OFFSET_UP, CAM_OFFSET_FORWARD, cameraRotation, cameraPosition, camera);

    log("capture of camera");
    log(cameraName);

    GAMEPLAY::SET_GAME_PAUSED(true);

    capture();
    capture();

    setDepthBuffer();
    setStencilBuffer();

    FrameObjectInfo fObjInfo = m_pObjDet->generateMessage(depth_map, m_stencilBuffer);
    m_pObjDet->exportDetections(fObjInfo);
    std::string filename = m_pObjDet->getStandardFilename("image_2", ".png");
    m_pObjDet->exportImage(screenCapturer->pixels, filename);

    log("capture exported");
    GAMEPLAY::SET_GAME_PAUSED(false);

    m_pObjDet->currentExtraCameraName = "";
}

void Scenario::setRenderingCamWithRotAndPos(Vehicle v, int height, int length, Vector3 newRotation, Vector3 newPosition, Cam xcam) {
    Vector3 position;
    Vector3 fVec, rVec, uVec;
    Vector3 rotation = ENTITY::GET_ENTITY_ROTATION(v, 0);
    ENTITY::GET_ENTITY_MATRIX(v, &fVec, &rVec, &uVec, &position);

    position.x += newPosition.x;
    position.y += newPosition.y;
    position.z += newPosition.z;

    rotation.x += newRotation.x;
    rotation.y += newRotation.y;
    rotation.z += newRotation.z;

    Vector3 offset;
    offset.x = 0;
    offset.y = length / 2;
    offset.z = height;
    Vector3 offsetWorld = camToWorld(offset, fVec, rVec, uVec);
    //Since it's offset need to subtract the cam position
    offsetWorld.x -= s_camParams.pos.x;
    offsetWorld.y -= s_camParams.pos.y;
    offsetWorld.z -= s_camParams.pos.z;

    GAMEPLAY::SET_TIME_SCALE(0.0f);
    GAMEPLAY::SET_GAME_PAUSED(false);
    GAMEPLAY::SET_TIME_SCALE(0.0f);

    CAM::SET_CAM_COORD(xcam, position.x + offsetWorld.x, position.y + offsetWorld.y, position.z + offsetWorld.z);
    CAM::SET_CAM_ROT(xcam, rotation.x, rotation.y, rotation.z, 0);
    scriptWait(0);
    GAMEPLAY::SET_GAME_PAUSED(true);

    std::ostringstream oss;
    oss << "EntityID/rotation/position: " << v << "\n" <<
        position.x << ", " << position.y << ", " << position.z <<
        "\n" << rotation.x << ", " << rotation.y << ", " << rotation.z <<
        "\nOffset: " << offset.x << ", " << offset.y << ", " << offset.z <<
        "\nOffsetworld: " << offsetWorld.x << ", " << offsetWorld.y << ", " << offsetWorld.z;
    //log(oss.str(), true);
    log(oss.str());
}

[bradenhurl]

Everything in these two functions looks fine to me. Are you also changing the cam in Scenario::setCamParams? Alternatively, you could not use a new camera and simply change the position/rotation of the global camera ('camera'). This is what I did to generate secondary perspectives. The following two lines could be causing your problems (lines 864/865 in scenario.cpp):

//These values change frame to frame
s_camParams.theta = CAM::GET_CAM_ROT(camera, 0);
s_camParams.pos = CAM::GET_CAM_COORD(camera);

[Hokyjack]

Yes, actually I also tried creating multiple cameras before, but with the same result. Here, you can see, I am just passing reference of the main camera object to the function: setRenderingCamWithRotAndPos(m_ownVehicle, CAM_OFFSET_UP, CAM_OFFSET_FORWARD, cameraRotation, cameraPosition, camera); so I am using the same global camera (just changing rotation), so the setCamParams should not be causing problems, since they refer to the same camera variable.

[bradenhurl]

Okay, and where are you calling generateExtraCameraPerspective? In the same spot as generateSecondaryPerspectives?

[Hokyjack]

yes, exactly

[bradenhurl]

How are the 3D bounding boxes? Tip for debugging: set up a stationary scene (the airport would be good as there isn't much traffic), place one vehicle and log the vehicle position from the first and second camera's perspective. Set the maximum range for entities to be small so you only get the one vehicle.

[meteorshowers]

@bradenhurl I just need the stereo images and depths map? Do these data right? Thanks a lot!

[Hokyjack]

How are the 3D bounding boxes? Tip for debugging: set up a stationary scene (the airport would be good as there isn't much traffic), place one vehicle and log the vehicle position from the first and second camera's perspective. Set the maximum range for entities to be small so you only get the one vehicle.

I'll try it. Also I found this function ExportEntity Could you please check this and maybe elaborate if this could be causing troubles?

[bradenhurl]

@bradenhurl I just need the stereo images and depths map? Do these data right? Thanks a lot!

Yes, I believe the depth and camera images are correct. You can easily obtain stereo by setting two cameras in the desired locations.

[bradenhurl]

This could potentially be a problem. Have you tried commenting out the checks for if the entity is occluded and if there is a clear LOS? Not sure if you are aware but there can sometimes be useful information on the functions located at: http://www.dev-c.com/nativedb/

[Hokyjack]

Disabling LOS checks didn't help. However, today I might have found some lead: When I ouput the unused stencil points image, there is just a few in the front facing camera, but when I do on rotated camera there is plenty - and furthermore these pixels seem to be the complementary to the missing pixels of segmentation objects - so they should be in the instSeq image instead.

Here is the comparision of instSeq image - white and unusedPixels image - red

I have no idea why these pixels are missclassified, because when I look at the stencilImage, everything seems to be OK:

Any thoughts?