Is it possible to stream and capture more than one camera view?

IanKirwan commented 6 years ago

I see in the natives API there are camera create functions. Is it possoble to have multiple cameras running simultaneously to stream for control and capture?

sibojia commented 6 years ago

I believe it's possible as I came across this TwoPlayerMod. Hope it helps.

dominikandreas commented 6 years ago

I don't see two camera views being used in that mod. Both players share the same view

IanKirwan commented 6 years ago

I'm thinking about left, right and rear view cameras for autonomous vehicle control. The API includes:

Cam CREATE_CAM(char *camName, BOOL p1)
Cam CREATE_CAMERA(Hash camHash, BOOL p1) 
void SET_CAM_ACTIVE(Cam cam, BOOL active)
void SET_CAM_ROT(Cam cam, float rotX, float rotY, float rotZ, int rotationOrder)
void SET_CAM_COORD(Cam cam, float posX, float posY, float posZ)

Not sure why there are 2 camera create functions. Two different mechanisms I guess. I'm hoping you can have more than one cam active and that set_cam_coord is relative to vehicle.

Was just wondering if anyone had attempted to do this, might save me some pain to get some info.

ndrplz commented 6 years ago

I indeed tried to capture the same scene from multiple cameras for this work. Still, I didn't find a clean way to do so. The workaround I used is to create the two cameras and then swap camera at each game iteration, while recording frames. However, there are some downsides:

The two streams are recorded at a lower framerate. Say, if you run the game at 30 fps, you'll find yourselves with two streams at approximately 15fps. For obvious reasons this does not scale well to more than a couple of cameras.
Entities will be slightly disaligned between the two views, because an iteration has passed. The difference is usually negligible, but it could be annoying if the entity is moving fast.
You can't play the game yourself, indeed you almost become epileptic if you watch the screen while the cameras are swapping at 30 fps :-) Of course this is a problem only if you want to control the entity yourself

Would be very interested if someone finds a cleaner solution! @IanKirwan

IanKirwan commented 6 years ago

Hi @ndrplz

Thanks for the info. I think a control algorithm could cope with the out of phase synchronicity caused by the camera switching. From what you are saying it sounds like GTAV only allows a single view port. Not sure about the frame rate. Is there a GTAV limitation to frame rate or is it down to the GPU? I'm currently using a GTX770 but have access to a 1080ti if needed. I have seen a 3 screen set up but I think it was just one view on an extended desktop. I'm wondering if multiple gpus will allow multiple views.

Cheers Ian

PS I had your paper in my Mendelay account waiting to be read :o)

Cuky88 commented 6 years ago

@IanKirwan Look at this video: https://www.youtube.com/watch?time_continue=12&v=CdDPK4hOA5U You can use at least 2 cameras side by side....maybe there is a way to use it for your purpose. I'm planning also to use left, center, right and rear view cameras for autonomous vehicle control.

The video is from this mod, but according to the author, the splitscreen is not available, yet. But there is another version of this mod with experimental splitscreen.

It would be nice if you could share your insights with me.

Cheers, Selcuk

IanKirwan commented 6 years ago

Hi Selcuk,

Thank you kindly for this. I can't view the video for some reason. Possibly because I am using my phone. Will try again with a PC tomorrow.

Would be pleased to hear more about your plans if that's possible.

Many thanks again Ian

On 27 Sep 2017 16:29, "Cuky88" notifications@github.com wrote:

@IanKirwan https://github.com/iankirwan Look at this video: https://www.youtube.com/watch?time_continue=12&v=CdDPK4hOA5U You can use at least 2 cameras side by side....maybe there is a way to use it for your purpose. I'm planning also to use left, center, right and rear view cameras for autonomous vehicle control.

The video is from this mod https://www.gta5-mods.com/scripts/twoplayermod-net-beta#description_tab, but according to the author, the splitscreen is not available, yet. But there is another version of this mod https://www.gta5-mods.com/scripts/twoplayermod-net-split with experimental spitscreen.

It would be nice if you could share your insights with me.

Cheers, Selcuk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ai-tor/DeepGTAV/issues/68#issuecomment-332560291, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab7e0TWpxyRX3FTo0BA6rOggpKiW9g52ks5smmnsgaJpZM4PHvbw .

IanKirwan commented 6 years ago

Wrt insights I'll help if I can. What sort of insights are you thinking of?

On 27 Sep 2017 16:29, "Cuky88" notifications@github.com wrote:

@IanKirwan https://github.com/iankirwan Look at this video: https://www.youtube.com/watch?time_continue=12&v=CdDPK4hOA5U You can use at least 2 cameras side by side....maybe there is a way to use it for your purpose. I'm planning also to use left, center, right and rear view cameras for autonomous vehicle control.

The video is from this mod https://www.gta5-mods.com/scripts/twoplayermod-net-beta#description_tab, but according to the author, the splitscreen is not available, yet. But there is another version of this mod https://www.gta5-mods.com/scripts/twoplayermod-net-split with experimental spitscreen.

It would be nice if you could share your insights with me.

Cheers, Selcuk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ai-tor/DeepGTAV/issues/68#issuecomment-332560291, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab7e0TWpxyRX3FTo0BA6rOggpKiW9g52ks5smmnsgaJpZM4PHvbw .

Cuky88 commented 6 years ago

@IanKirwan I was asking if you could share your insights regarding multiple cams :)

I'm currently working on a robust mod to extract 2D and 3D bounding boxes out of the game. Next step will be to extract segmentations and depth information and do some tests, if I can use the image footage to train CNNs and SegNets, which can be applied to real footage. We are building a race car, which is currently using 3 cameras and 2 LIDARs. I want to simulate the car in GTA V and assess, if we can use GTA V as a simulation environment for image and sensor data.

Cheers, Selcuk

mindThomas commented 6 years ago

@Cuky88 Sounds very interesting. I would like to know how this goes. But why have no one investigated a way to extract the bounding boxes (vehicle, pedistrian, object locations) instead of always trying to estimate them based on different algorithmic approaches (machine learning or not).

IanKirwan commented 6 years ago

@Cuky88 I'm afraid I have limited experience. Multiple cam streams will be new to me.

There are 2 papers you should look at for segmentation data from GTAV: 'driving in the matrix' and 'playing for data'. They both have GitHub projects that are easy to fine. The latter includes a sizeable cache of images, their segmented counterparts and ground truths. I'll be looking to do some SegNets stuff soon too.

Hth

On 30 Sep 2017 14:12, "Cuky88" notifications@github.com wrote:

@IanKirwan https://github.com/iankirwan I was asking if you could share your insights regarding multiple cams :)

I'm currently working on a robust mod to extract 2D and 3D bounding boxes out of the game. Next step will be to extract segmentations and depth information and do some tests, if I can use the image footage to train CNNs and SegNets, which can be applied to real footage. We are building a race car, which is currently using 3 cameras and 2 LIDARs. I want to simulate the car in GTA V and assess, if we can use GTA V as a simulation environment for image and sensor data.

Cheers, Selcuk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ai-tor/DeepGTAV/issues/68#issuecomment-333307644, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab7e0fTgkuor9OO8VO9Ulit2chsnZUNEks5snj4pgaJpZM4PHvbw .

On 30 Sep 2017 14:12, "Cuky88" notifications@github.com wrote:

@IanKirwan https://github.com/iankirwan I was asking if you could share your insights regarding multiple cams :)

I'm currently working on a robust mod to extract 2D and 3D bounding boxes out of the game. Next step will be to extract segmentations and depth information and do some tests, if I can use the image footage to train CNNs and SegNets, which can be applied to real footage. We are building a race car, which is currently using 3 cameras and 2 LIDARs. I want to simulate the car in GTA V and assess, if we can use GTA V as a simulation environment for image and sensor data.

Cheers, Selcuk

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ai-tor/DeepGTAV/issues/68#issuecomment-333307644, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab7e0fTgkuor9OO8VO9Ulit2chsnZUNEks5snj4pgaJpZM4PHvbw .

Cuky88 commented 6 years ago

@IanKirwan Thanks for the hint. I saw these projects before. I would like to generate segmentation data directly out of the game when running. So, "playing for data" records the game with renderdoc and then extracts the data. This would be the last option for me. "driving in the matrix" seems to be very interesting, but the buffers I get, are noisy. So, I need to invest more time in that. But for now, 2D and 3D bounding boxes do work fine on my workstation.

@mindThomas Sry, I didn't understand you question :) The bounding boxes are extracted directly from the game engine, so that no manual work is needed. I just transform it to the needed format, at the end.

mindThomas commented 6 years ago

@Cuky88 Sorry, I got that after investigating more about DeepGTAV and especially Script Hook V

JasmineDeng commented 5 years ago

@ndrplz I'm trying to see if multiple camera view is possible, and would be curious to know the exact code you used for your multi-camera view? I'm new to DeepGTAV and I'm not sure how that should be done/where should be implemented.

Also, just following this issue for further updates.

mkwetherell commented 5 years ago

Similar question to the above, so I thought I'd mention it here before starting a new thread. I would like to simply use a different camera view than the default setting in deepgtav. Is it possible to switch the camera view to one which includes the vehicle? Many thanks!

Matt

JasmineDeng commented 5 years ago

I actually got multiple camera to work by adding a "cameraNumber" variable in Scenario.cpp and incrementing it once a frame is sent in Server.cpp. Changing the SET_TIME_SCALE to 0 while recording different camera views makes sure roughly the same frame rate is kept.

And I got different camera views via changing the ATTACH_CAM_TO_ENTITY in buildScenario() and SET_CAM_ROT in run() appropriately.

ghost commented 4 years ago

I actually got multiple camera to work by adding a "cameraNumber" variable in Scenario.cpp and incrementing it once a frame is sent in Server.cpp. Changing the SET_TIME_SCALE to 0 while recording different camera views makes sure roughly the same frame rate is kept.

And I got different camera views via changing the ATTACH_CAM_TO_ENTITY in buildScenario() and SET_CAM_ROT in run() appropriately.

In which file is this SET_TIME_SCALE variable?

aitorzip / DeepGTAV

Is it possible to stream and capture more than one camera view? #68