Update of periscope streaming API

[DmT021]

Hi, As I can see, you create a broadcast with /liveorigin path in the rtmp URL. When I create a broadcast i get vidmanlive application in response of createBroadcast function. Do you get the same?

[DmT021]

Oh, I got it. I used to specify User Agent as "Periscope/2699 (iPhone; iOS 8.1.2; Scale/2.00)" within a request and periscope returns vidmanlive. If I do not specify anything periscope returns liveorigin.

[Pmmlabs]

Thanks for info, but if I specify user agent and get url to vidman application, ffmpeg crashes with error RTMP_ReadPacket, failed to read RTMP packet header. If user agent not specified, liveorigin application returns, and streaming runs normally, besides broadcast can't be viewed on smartphone, only in browser. Do you get it working with vidmanlive ? Both on smartphone and in browser?

[DmT021]

Nope, can't fight vidmanlive. But I found something very intresting in the RTMP client of periscope for android. I let it crash for my sample broadcast and checked out the crash report. If I'm correct it fails with NullPointerException (getting array length of null) of something called trackinfo of onMetaData object.

    public final boolean m10316(aun o_aun) {
        aun o_aun2 = o_aun;
        if (o_aun.bMN == 18) {
            o_aun2 = o_aun;
            Object[] objArr = o_aun.bMO;
            if (((String) objArr[0]).equals("onMetaData") && this.boH == null) {
                C0467 c0467;
                C1778 c1778;
                Map map = (Map) objArr[1];
                afy o_afy = this;
                Object[] objArr2 = (Object[]) r9.get("trackinfo");
                for (int i = 0; i < objArr2.length; i++) { // crashes here
                    Map map2 = (Map) objArr2[i];
                    String str = (String) map2.get("type");
                    if (str.equals("video")) {
                        Double d = (Double) r9.get("height");
                        o_afy.boH = new agb(i, (String) map2.get("sprop-parameter-sets"), ((Double) r9.get("width")).intValue(), d.intValue(), o_afy.boI);
                        o_afy.boH.m7692(o_afy.boF);
                    } else if (str.equals("audio")) {
                        o_afy.boG = new afx(i, (String) map2.get("config"), ((Double) r9.get("audiochannels")).intValue(), ((Double) r9.get("audiosamplerate")).intValue());
                        o_afy.boG.m7692(o_afy.boF);
                    }
                }
                if (o_afy.boH == null) {
                    c0467 = null;
                    C1942.log("Stream with no video encountered: " + o_afy.boC);
                } else {
                    c0467 = new C2102(o_afy, o_afy.boH, 1, 5000, o_afy.boD.jf, o_afy.boD, 50);
                }
                if (o_afy.boG == null) {
                    c1778 = null;
                    C1942.m9770(new Exception("Stream with no audio encountered: " + o_afy.boC));
                } else {
                    c1778 = new C1778(o_afy.boG);
                }
                C0467[] c0467Arr = new C0467[5];
                c0467Arr[0] = c0467;
                c0467Arr[1] = c1778;
                o_afy.boE.m7649(null, null, c0467Arr);
            }

As I can understand it sends/recieves some additional metadata with streams' parameters. I'm not an expert in RTMP protocol, so I can't explain what exactly type of RTPM message it sends, but I know that it encode it as string with RTMPMessage prefix and hexadeciamal representation of data.

[Pmmlabs]

Metadata can be posted using -metadata option of FFmpeg. Without it, error in logcat is

java.lang.NullPointerException: Attempt to get length of null array

And with -metadata trackinfo="<someinfo>" option, error is

java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Object[]

Instead <someinfo> I tried JSON-encoded object, but no luck. Seems to be trackinfo must be some complex structure in Metadata section, as here. And I have not found, how to do it with ffmpeg.

[Pmmlabs]

Sample onMetaData object (from logs.txt in pingBroadcast request):

{
  frameHeight=568,
  trackinfo=[
    {
      sampledescription=[{
                    sampletype = "H264";
                }],
      language="eng",
      description = "{H264CodecConfigInfo: codec:H264, profile:Main, level:2.1, frameSize:320x568, displaySize:320x568, crop: l:0 r:0 t:0 b:0}"; // iPhone: l:0 r:0 t:0 b:8
      type="video",
      profile-level-id="640015",   // iPhone: 4d0015
      timescale=90000,
      sprop-parameter-sets="Z2QAFazSBQEn5YBtChNQAA==,aM4G4sA=" // iPhone: "Z00AFatgoCT8qA==,KO4GcMA="
    },
    {
      config="1208",
      sampledescription=[{
                    sampletype = "mpeg4-generic";
                }],
      description = "{AACFrame: codec:AAC, channels:1, frequency:44100, samplesPerFrame:1024, objectType:LC}";
      language="eng",
      type="audio",
      timescale=90000
    }
  ],
  audiocodecid="mp4a",
  frameWidth=320,
  rtpsessioninfo={
    name="Live stream from Periscope",
    connectiondata="In IP4 0.0.0.0",
    protocolversion=0,
    timing="0 0"
  },
  audiosamplerate=44100,
  height=568,
  width=320,
  displayWidth=320,
  videocodecid="avc1",
  displayHeight=568,
  audiochannels=1
}

[DmT021]

Yep, I checked out ffmpeg sources https://github.com/FFmpeg/FFmpeg/blob/a0174f67298ba9494c146183dd360e637b03db64/libavformat/flvenc.c#L205 There is no way to pass anything but string. Probably there is another streaming software that can do it, but I didn't find. I going to check out RTMP libs with streaming feature.

[Pmmlabs]

There is something called SDP for posting metadata. Here is example for rtpsessioninfo object, I think for trackinfo must be similarly. UPD: no, it will not help.

[Pmmlabs]

Found something: rtmp_conn ffmpeg option. Allow to write arbitrary AMF connection parameters, even objects.

[DmT021]

rtmp_conn - Append arbitrary AMF data to the Connect message As I can understand the rtmp flow is following: handshake send connect send createStream send publish send onMetadata

[Pmmlabs]

I tried to patch ffmpeg, but one parameter, sprop-parameter-sets, somehow doesn't seen in Periscope (i.e. null). I don't know, why.

[Pmmlabs]

I edited the patch and got it working. Unfortunately, on the screen I see gray porridge instead of video. Audio working great.

[Pmmlabs]

Got video working! But only on Android. iPhone still crashes. Patch fixed.

[MichaelZaporozhets]

@Pmmlabs Do you have any insight into why it's crashing on iphone? I tried passing through meta data but it still crashes on iphone for some reason..

[DmT021]

It's much harder for me to debug iOS version of periscope. I'm sure both versions works with the same metadata, but actually iOS and android decode h264/avc differently. So it's possible that iOS simply can't decode current config of h264 stream. Did you tried different options for profile (baseline, main, high) and profile-level (2.0, 2.1, 4.0, 4.1)?

[MichaelZaporozhets]

@DmT021 I'm trying a bunch of option combinations now. Will update on results

[MichaelZaporozhets]

@DmT021 I'm on a jail broken iPhone too- hoping to find a way to look at the video buffer

[Pmmlabs]

I tried Main 2.1, because it was in log.txt from pingBroadcast. Yes, maybe need to try other options. I have no iPhone, therefore testing will be long.

[MichaelZaporozhets]

@Pmmlabs perhaps this will help

[MichaelZaporozhets]

Here is the crash log for periscope on the iphone:

Mon Mar 14 00:58:08 2016: Periscope (com.bountylabs.periscope): Error response Error Domain=PubNub Channel is nil Code=0 "(null)"
Mon Mar 14 00:58:08 2016: Periscope (SecLogging):  SecTrustEvaluate  [leaf AnchorTrusted ValidLeaf ValidRoot]
Mon Mar 14 00:58:22 2016: Periscope (com.bountylabs.periscope): -[__NSCFConstantString objCType]: unrecognized selector sent to instance 0x5346c8
Mon Mar 14 00:58:22 2016: Periscope (com.bountylabs.periscope): *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFConstantString objCType]: unrecognized selector sent to instance 0x5346c8'
*** First throw call stack:
(0x24e4b86b 0x3654adff 0x24e51035 0x24e4ec8f 0x24d7e2b8 0x24d669e7 0x14f849 0x14f157 0x14ebbd 0x1c01ab 0x1c0fb7 0x1bbecd 0x36c4ced7 0x36c4cec3 0x36c51729 0x24e0e595 0x24e0ca8f 0x24d5f1e9 0x24d5efdd 0x2e003af9 0x28fc418d 0x97ec7 0x36c75873)
Mon Mar 14 00:58:23 2016: ReportCrash (Crash Reporter): Formulating report for corpse[6258] Periscope
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Notice: Injecting: com.bountylabs.periscope [Periscope] (1240.10)
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Notice: Loading: /Library/MobileSubstrate/DynamicLibraries/SSLKillSwitch2.dylib
Mon Mar 14 00:58:24 2016: Periscope (com.bountylabs.periscope): === SSL Kill Switch 2: Preference set to 1.
Mon Mar 14 00:58:24 2016: Periscope (com.bountylabs.periscope): === SSL Kill Switch 2: Subtrate hook enabled.
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Notice: Loading: /Library/MobileSubstrate/DynamicLibraries/SpeedIntensifier.dylib
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector slowAnimations
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector slowDownFactor
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector slowAnimations
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector slowDownFactor
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector backlightFadeDuration
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector setIsEditing:
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector _beginEditing
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Warning: nil class argument for selector _stopEditing
Mon Mar 14 00:58:24 2016: Periscope (com.apple.console): MS:Notice: Loading: /Library/MobileSubstrate/DynamicLibraries/XMScreenShotEx.dylib
Mon Mar 14 00:58:24 2016: Periscope (com.apple.coregraphics): CGContextSaveGState: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Mon Mar 14 00:58:24 2016: Periscope (com.apple.coregraphics): CGContextTranslateCTM: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Mon Mar 14 00:58:24 2016: Periscope (com.apple.coregraphics): CGContextRestoreGState: invalid context 0x0. If you want to see the backtrace, please set CG_CONTEXT_SHOW_BACKTRACE environmental variable.
Mon Mar 14 00:58:25 2016: Periscope (com.bountylabs.periscope): [Crashlytics] Version 3.6.0 (99)
Mon Mar 14 00:58:25 2016: Periscope (com.bountylabs.periscope): <WARNING> -[Twitter session] will soon be deprecated. It is recommended that users use -[TWTRSessionStore session] or -[TWTRSessionStore sessionForUserID:] if they are managing multiple users
Mon Mar 14 00:58:25 2016: Periscope (com.apple.console): MS:Warning: message not found [UITextField becomeFirstResponder:]
Mon Mar 14 00:58:25 2016: Periscope (com.apple.console): MS:Warning: message not found [UITextField becomeFirstResponder:]
Mon Mar 14 00:58:25 2016: Periscope (com.bountylabs.periscope): [Crashlytics:Crash:Reports] Submitting async /var/mobile/Containers/Data/Application/45D70B75-4330-4129-9AFE-1A95BCF796C7/Library/Caches/com.crashlytics.data/com.bountylabs.periscope/v3/prepared/6CD26C40-3C7F-4FE3-8C26-87475D544C7D.multipartmime
Mon Mar 14 00:58:26 2016: Periscope (com.bountylabs.periscope): Unbalanced calls to begin/end appearance transitions for <SENavigationController: 0x15398800>.
Mon Mar 14 00:58:26 2016: Periscope (SecLogging):  SecOSStatusWith error:[-50] Error Domain=NSOSStatusErrorDomain Code=-50 "query missing class name" UserInfo={NSDescription=query missing class name}
Mon Mar 14 00:58:28 2016: Periscope (com.bountylabs.periscope): Error response Error Domain=NSURLErrorDomain Code=-1002 "unsupported URL" UserInfo={NSErrorFailingURLStringKey=, NSErrorFailingURLKey=, NSLocalizedDescription=unsupported URL, NSUnderlyingError=0x146dd5a0 {Error Domain=kCFErrorDomainCFNetwork Code=-1002 "(null)"}}
Mon Mar 14 00:58:30 2016: Periscope (com.bountylabs.periscope): Error response Error Domain=PubNub Channel is nil Code=0 "(null)"
Mon Mar 14 00:58:31 2016: Periscope (SecLogging):  SecTrustEvaluate  [leaf AnchorTrusted ValidLeaf ValidRoot]
Mon Mar 14 00:58:31 2016: Periscope (com.apple.lsuseractivity): Caching encoded userInfo to use until we are marked dirty again (UAUserActivity.m #1567)
Mon Mar 14 00:58:31 2016: Periscope (user): LaunchServices: disconnect event received for service com.apple.lsd.modifydb
Mon Mar 14 00:58:36 2016: Periscope (com.bountylabs.periscope): -[__NSCFConstantString objCType]: unrecognized selector sent to instance 0x4f96c8
Mon Mar 14 00:58:36 2016: Periscope (com.bountylabs.periscope): *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFConstantString objCType]: unrecognized selector sent to instance 0x4f96c8'
*** First throw call stack:
(0x24e4b86b 0x3654adff 0x24e51035 0x24e4ec8f 0x24d7e2b8 0x24d669e7 0x114849 0x114157 0x113bbd 0x1851ab 0x185fb7 0x180ecd 0x36c4ced7 0x36c4cec3 0x36c51729 0x24e0e595 0x24e0ca8f 0x24d5f1e9 0x24d5efdd 0x2e003af9 0x28fc418d 0x5cec7 0x36c75873)
Mon Mar 14 00:58:36 2016: ReportCrash (Crash Reporter): Formulating report for corpse[6276] Periscope

[Pmmlabs]

@MichaelZaporozhets from 3-rd party libraries, only libx264 is needed. What error do you get while compiling?

[MichaelZaporozhets]

@Pmmlabs I overcame the error; It was a whitespace issue. That being said, how do we pass meta data with the patch?

[Pmmlabs]

@MichaelZaporozhets nohow, it's hardcoded for a while.

[Pmmlabs]

@MichaelZaporozhets thanks for the crashlog, NSInvalidArgumentException means that periscope expected some type, but got string. I think, the reason is not in video encoding, but in metadata.

[MichaelZaporozhets]

@Pmmlabs I assume it's looking for a pubnub channel to be passed with the stream somehow?

[Pmmlabs]

@MichaelZaporozhets android app don't passes pubnub channel with the stream. but streams, created with android, can be viewed on iPhone. Therefore, that is not the reason.

[MichaelZaporozhets]

I assumed that Error response Error Domain=PubNub Channel is nil Code=0 "(null)" referred to the metadata- no?

[MichaelZaporozhets]

@Pmmlabs Would it help at all if I gave you the .app file, you could probably decompile some with Hopper

[Pmmlabs]

@MichaelZaporozhets I don't know, how to decompile iphone apps and is it possible. I am zero in iphone development.

[lepikhinb]

@MichaelZaporozhets have you ever tried to mess up with Hopper?

[MichaelZaporozhets]

@Pmmlabs yeah, i'm the same

@lepikhinb nope, but know it's quite popular for his sort of thing

[lepikhinb]

@MichaelZaporozhets bump that .app file on my email please - lepikhinb[at]gmail.com I will try to look into this

[MichaelZaporozhets]

@lepikhinb will do. Thanks a lot

[MichaelZaporozhets]

@lepikhinb let me know if you've received the package

[lepikhinb]

@MichaelZaporozhets got it, thanks!

[MichaelZaporozhets]

Just an FYI, offering a cash bounty for anyone who can fix this within the next day or so

[DmT021]

Yeah, with the log i'm also sure the thing is in the metadata. That's a good news, no need to dive in h264 encoding details. Another possible solution is to dump tcp traffic with some tool compatible with wireshark and compare original meta and @Pmmlabs ' . Also there is a simple server in rtmpdump. It's also very helpful.

[lepikhinb]

@DmT021 do you have any ideas how to dump rtmp from original app? I've tried rvictl tool on OSX and all I got is encoded text (despite the fact that SSLKiller is already installed)

[Pmmlabs]

I think, there is no need to dump rtmp traffic, because all sent metadata are logged in "log.txt" in "pingBroadcast" request. I updated my comment and added comments, where is difference between Android and iPhone. But! Error in Michael's crashlog tell us, that there is no lack of metadata (NullPointerException would be thrown), but Periscope is trying to use some string variable as number (i.e. double), actually to compare with another double. It says the following log:

Last Exception Backtrace:
0 CoreFoundation 0x23b1f866 0x23a2a000 + 0xf5866 // __exceptionPreprocess + 0x7a
1 libobjc.A.dylib 0x3521edfa 0x35218000 + 0x6dfa // objc_exception_throw + 0x22
2 CoreFoundation 0x23b25030 0x23a2a000 + 0xfb030 // -[NSObject(NSObject) doesNotRecognizeSelector:] + 0xb8
3 CoreFoundation 0x23b22c8a 0x23a2a000 + 0xf8c8a // ___forwarding___ + 0x2ba
4 CoreFoundation 0x23a522b4 0x23a2a000 + 0x282b4 // _CF_forwarding_prep_0 + 0x14
5 CoreFoundation 0x23a3a9e2 0x23a2a000 + 0x109e2 // -[__NSCFNumber compare:] + 0x66
6 Periscope (*) 0x000da844 0x00014000 + 0xc6844 

And yes, I already tried to write all possible metadata from log.txt in my patch, and it doesn't solve the problem. :disappointed:

[DmT021]

@lepikhinb nope, I used to dump all traffic with a kind of vpn-based sniffer on my android. And then I parse it byte-by-byte with wireshark. It was painful... And I shoud also mention that there are two rtmp urls: rtmp and rtmps. For some reason it starts with rtmp and then the session closes and rtmps sessions opens. But metadata is in both. So I didn't use any kind of ssl termination or something like this. @Pmmlabs agree. But in your dump all variables lack of type info. AMF stores the type for each field and array item and I'd like to compare exactly type infos.

[MichaelZaporozhets]

@DmT021 @Pmmlabs @lepikhinb Let me know if there is anything else I can provide. I'm currently trying to change any numbers we pass as strings to numbers to see if that will work.

[MichaelZaporozhets]

Also, do we know why replays do not work and/or what can be done to make it work?

[DmT021]

Api provides replay upload URL, which is a link to s3. So, I think a replay should be uploaded. If so, that is not pretty good solution, for me. Same for thumbnails.

[lepikhinb]

@MichaelZaporozhets saw absolutely nothing in Hopper. Not sure if I'm dumb but it works different way unlike pc-based debuggers (for example Olly Debugger).

[lepikhinb]

@MichaelZaporozhets I guess that replays have to be uploaded different way. Also I have no idea why I don't see any traffic of thumbnail uploading in Charles.

[MichaelZaporozhets]

@lepikhinb ah damn. I looked at it too but I'm a web developer, not particularly comfortable with reverse engineering obj-C

[MichaelZaporozhets]

To be honest, just getting this iOS crash bug fixed would be more than enough to make this viable. Also, I'm pretty sure that periscope pulls the thumbnail from the uploaded video to signer.periscope.tv

Edit: I've made a pull request to add thumbnails.

[MichaelZaporozhets]

I have also found that instead of using flv, using mpegts as a format doesn't crash on iOS. However, the stream does not come up. Stuck on loading...

[Pmmlabs]

@MichaelZaporozhets, using mpegts, broadcast doesn't plays in browser and in Android app, because m3u url returns 404. No stream - no crash.

[lepikhinb]

@MichaelZaporozhets how did you add a thumbnail?

[Pmmlabs]

@lepikhinb see #2 and update the OpenPeriscope