search

Navideck / Romo-iOS-SDK

Romo SDK supporting up to iOS 17!
Other
44 stars 15 forks source link

Romo app #5

Closed cnnn closed 6 years ago

cnnn commented 6 years ago
fotiDim commented 6 years ago

@cnnn just did some testing. Video is not working right with iPhone 5s (Romo app). I do get a video feed but it is very glitchy. I see mostly green frames and only occasionally frames from the camera feed. It seems that the behaviour is dependent on where Romo is facing. On some orientations in my room the camera feed goes normal but as soon as I move Romo a bit it goes green again.

At the same time my iPhone SE works fine, so it must be something with the 5s hardware encoding. In both tests I used my iPad 4 with iOS 10 as Romo Control.

Do you have any suggestions?

cnnn commented 6 years ago

@fotiDim thanks for suggestions. I agree it's better to keep the original code as much as possible. I'll try not to remove any file. Hopefully the diff should be much smaller.

By telepresence I was referring to the group folder Telepresence, which is the remote telepresence via OpenTok. I believe the local telepresence is called WiFi Drive in the project.

The green frames on iPhone 5s sound like key frames get dropped. It could be due to hardware encoding capacity on the 5s. Did you get any error on the console from socket? In - (void)gotEncodedData:(NSData*)data isKeyFrame:(BOOL)isKeyFrame, is the data size for key frame greater than 65535?

If you don't see any error, try tweaking these parameters Camera output resolution https://github.com/fotiDim/Romo/pull/5/files#diff-b49e537a6db193b004415aa316bf4500R98 Encoding output resolution https://github.com/fotiDim/Romo/pull/5/files#diff-b49e537a6db193b004415aa316bf4500R156 Encoding profile https://github.com/fotiDim/Romo/pull/5/files#diff-28a368d8d9b79fd4dd33eb4f00ff5bf7R167

fotiDim commented 6 years ago

This is my console output:

objc[473]: Class CvAbstractCamera is implemented in both /private/var/containers/Bundle/Application/C98E4F1B-802B-482B-ADAA-912634681432/Romo.app/Frameworks/RMVision.framework/RMVision (0x101704950) and /var/containers/Bundle/Application/C98E4F1B-802B-482B-ADAA-912634681432/Romo.app/Romo (0x1009b1ae8). One of the two will be used. Which one is undefined.
objc[473]: Class CvPhotoCamera is implemented in both /private/var/containers/Bundle/Application/C98E4F1B-802B-482B-ADAA-912634681432/Romo.app/Frameworks/RMVision.framework/RMVision (0x1017049a0) and /var/containers/Bundle/Application/C98E4F1B-802B-482B-ADAA-912634681432/Romo.app/Romo (0x1009b1b38). One of the two will be used. Which one is undefined.
objc[473]: Class CvVideoCamera is implemented in both /private/var/containers/Bundle/Application/C98E4F1B-802B-482B-ADAA-912634681432/Romo.app/Frameworks/RMVision.framework/RMVision (0x1017049f0) and /var/containers/Bundle/Application/C98E4F1B-802B-482B-ADAA-912634681432/Romo.app/Romo (0x1009b1b88). One of the two will be used. Which one is undefined.
2018-04-05 06:57:27.902320+0200 Romo[473:46127] [DYMTLInitPlatform] platform initialization successful
2018-04-05 06:57:28.221832+0200 Romo[473:45898] [HockeySDK] WARNING: Detecting crashes is NOT enabled due to running the app with a debugger attached.
2018-04-05 06:57:29.257342+0200 Romo[473:45898] registerForRemoteNotificationTypes: is not supported in iOS 8.0 and later.
2018-04-05 06:57:29.345960+0200 Romo[473:45898] [] <<<< AVOutputDeviceDiscoverySession (FigRouteDiscoverer) >>>> -[AVFigRouteDiscovererOutputDeviceDiscoverySessionImpl outputDeviceDiscoverySessionDidChangeDiscoveryMode:]: Setting device discovery mode to DiscoveryMode_None (client: Romo)
2018-04-05 06:57:29.793728+0200 Romo[473:45898] [] <<<< AVOutputDeviceDiscoverySession (FigRouteDiscoverer) >>>> -[AVFigRouteDiscovererOutputDeviceDiscoverySessionImpl outputDeviceDiscoverySessionDidChangeDiscoveryMode:]: Setting device discovery mode to DiscoveryMode_Presence (client: Romo)
objc[473]: __weak variable at 0x10b94f5d0 holds 0x109e90003 instead of 0x109e981a0. This is probably incorrect use of objc_storeWeak() and objc_loadWeak(). Break on objc_weak_error to debug.

2018-04-05 06:57:42.357828+0200 Romo[473:46119] [MC] System group container for systemgroup.com.apple.configurationprofiles path is /private/var/containers/Shared/SystemGroup/systemgroup.com.apple.configurationprofiles
2018-04-05 06:57:42.359055+0200 Romo[473:46119] [MC] Reading from public effective user settings.
2018-04-05 06:57:42.424351+0200 Romo[473:46368] H264: VTCompressionSessionCreate 0
2018-04-05 06:57:51.844266+0200 Romo[473:46353] KeyFrame 36
2018-04-05 06:57:51.859419+0200 Romo[473:46353] KeyFrame 2610
2018-04-05 06:57:53.042426+0200 Romo[473:46352] KeyFrame 31
2018-04-05 06:57:53.042919+0200 Romo[473:46352] KeyFrame 53258
2018-04-05 06:57:53.043984+0200 Romo[473:46352] KeyFrame 38012
2018-04-05 06:57:54.298899+0200 Romo[473:46352] KeyFrame 31
2018-04-05 06:57:54.299508+0200 Romo[473:46352] KeyFrame 59622
2018-04-05 06:57:54.300694+0200 Romo[473:46352] KeyFrame 43246
2018-04-05 06:57:55.547679+0200 Romo[473:46119] KeyFrame 31
2018-04-05 06:57:55.549298+0200 Romo[473:46119] KeyFrame 59710
2018-04-05 06:57:55.550814+0200 Romo[473:46119] KeyFrame 43931
2018-04-05 06:57:56.795972+0200 Romo[473:46351] KeyFrame 31
2018-04-05 06:57:56.800329+0200 Romo[473:46351] KeyFrame 59853
2018-04-05 06:57:56.801597+0200 Romo[473:46351] KeyFrame 43888
2018-04-05 06:57:58.045649+0200 Romo[473:46352] KeyFrame 31
2018-04-05 06:57:58.047165+0200 Romo[473:46352] KeyFrame 56102
2018-04-05 06:57:58.051080+0200 Romo[473:46352] KeyFrame 41022
2018-04-05 06:57:58.251433+0200 Romo[473:45898] sent notification SegmentioDidSendRequest
2018-04-05 06:57:58.675972+0200 Romo[473:45898] sent notification SegmentioRequestDidSucceed
2018-04-05 06:57:59.287689+0200 Romo[473:46351] KeyFrame 31
2018-04-05 06:57:59.288606+0200 Romo[473:46351] KeyFrame 56417
2018-04-05 06:57:59.289558+0200 Romo[473:46351] KeyFrame 40872
2018-04-05 06:58:00.544146+0200 Romo[473:46351] KeyFrame 31
2018-04-05 06:58:00.546596+0200 Romo[473:46351] KeyFrame 56138
2018-04-05 06:58:00.551633+0200 Romo[473:46351] KeyFrame 40830
2018-04-05 06:58:01.792852+0200 Romo[473:46352] KeyFrame 31
2018-04-05 06:58:01.794882+0200 Romo[473:46352] KeyFrame 53430
2018-04-05 06:58:01.799701+0200 Romo[473:46352] KeyFrame 37808
2018-04-05 06:58:03.069788+0200 Romo[473:46441] KeyFrame 31
2018-04-05 06:58:03.070940+0200 Romo[473:46441] KeyFrame 55899
2018-04-05 06:58:03.071851+0200 Romo[473:46441] KeyFrame 40576
2018-04-05 06:58:04.294174+0200 Romo[473:46352] KeyFrame 31
2018-04-05 06:58:04.298636+0200 Romo[473:46352] KeyFrame 52677
2018-04-05 06:58:04.300118+0200 Romo[473:46352] KeyFrame 37342
2018-04-05 06:58:05.543626+0200 Romo[473:46119] KeyFrame 31
2018-04-05 06:58:05.545450+0200 Romo[473:46119] KeyFrame 52269
2018-04-05 06:58:05.548341+0200 Romo[473:46119] KeyFrame 37215
2018-04-05 06:58:06.791734+0200 Romo[473:46119] KeyFrame 31
2018-04-05 06:58:06.792625+0200 Romo[473:46119] KeyFrame 52327
2018-04-05 06:58:06.794815+0200 Romo[473:46119] KeyFrame 37129
2018-04-05 06:58:08.042212+0200 Romo[473:46441] KeyFrame 31
2018-04-05 06:58:08.043189+0200 Romo[473:46441] KeyFrame 52397
2018-04-05 06:58:08.047290+0200 Romo[473:46441] KeyFrame 37355
2018-04-05 06:58:09.291099+0200 Romo[473:46352] KeyFrame 31
2018-04-05 06:58:09.293500+0200 Romo[473:46352] KeyFrame 52246
2018-04-05 06:58:09.297340+0200 Romo[473:46352] KeyFrame 37175
2018-04-05 06:58:10.541325+0200 Romo[473:46497] KeyFrame 31
2018-04-05 06:58:10.541859+0200 Romo[473:46497] KeyFrame 52467
2018-04-05 06:58:10.550140+0200 Romo[473:46497] KeyFrame 37415
2018-04-05 06:58:11.780983+0200 Romo[473:46119] KeyFrame 31
2018-04-05 06:58:11.782599+0200 Romo[473:46119] KeyFrame 52381
2018-04-05 06:58:11.783192+0200 Romo[473:46119] KeyFrame 37450
2018-04-05 06:58:13.044732+0200 Romo[473:46497] KeyFrame 31
2018-04-05 06:58:13.048754+0200 Romo[473:46497] KeyFrame 52578
2018-04-05 06:58:13.052207+0200 Romo[473:46497] KeyFrame 37145

I don't see any error and my keyframes seem to be smaller than 65535. I tried tweaking with the parameters you mentioned. Although they did improve things slightly the overall experience is more or less unusable.

I also tried using the 5S as Romo Control and this works fine. I guess the encoding process is heavier that decoding. Do you have access to any 5S devices?

You can see a video of the effect here

cnnn commented 6 years ago

I’ll try to get a 5s to test. Can you try setting he H264 encoding profile to Main Level 3.1?

fotiDim commented 6 years ago

@cnnn it is already kVTProfileLevel_H264_Main_3_1, no? But anyway I tried almost all values there. Didn't seem to make much difference.

fotiDim commented 6 years ago

@cnnn any updates on this?

cnnn commented 6 years ago

I got a 5S. Will do some test over the weekend.

cnnn commented 6 years ago

Looks like the issue with 5S is not due to encoding/decoding capability. The encoded packets on 5S when converted to element stream are splitted into two, whereas on SE it’s one whole frame.

I’m not sure yet what is the right way to fix it. Encoding seems correct. Problem is with decoding.

fotiDim commented 6 years ago

Hmm, I need to deep dive into the code to be able to investigate this. I will have time for that during next week. In the meantime can you check this guide to see if you find it helpful https://mobisoftinfotech.com/resources/mguide/h264-encode-decode-using-videotoolbox?

cnnn commented 6 years ago

That’s a good tutorial. I also found this one https://github.com/BrennanJones/BebopDroneVideoStreaming-iOS/blob/master/BebopDroneVideoStreaming-iOS/DroneVideoView.m with which I replace the h264 decode code and it fixed part of the problem. On 5S, the video is no longer black/green but actual image. However, the video freezes after a few samples. If I force every frame to be key frame, the video looks almost perfect. This problem still only exist on 5S, not SE.

fotiDim commented 6 years ago

@cnnn so if you set all frames as keyframes both problems are fixed? Is there any drawback to that approach? Can you commit that code so I can test it out?

cnnn commented 6 years ago

Closing this PR. I'll open two new PRs, one for general Romo app that can eventually get merged, and another used for 5S H264 investigation.