Undefined constant or missing '(' in 'high' error when transcoding certain segments using NETINT

yondonfu commented 2 years ago

[h264_ni_enc @ 0x7f7832291080] [Eval @ 0x7f78517f91a0] Undefined constant or missing '(' in 'high'
[h264_ni_enc @ 0x7f7832291080] Unable to parse option value "high"
[h264_ni_enc @ 0x7f7832291080] Error setting option profile to value high.
ERROR: encoder.c:240] Error opening video encoder : Invalid argument

MikeIndiaAlpha commented 2 years ago

@yondonfu One comment I have here - unless I understood something wrong, referred issue was on Windows and here we are talking about Netint, so Linux, right? Any hint on how can I reproduce the problem described above?

yondonfu commented 2 years ago

@MikeIndiaAlpha

referred issue was on Windows and here we are talking about Netint, so Linux, right?

Yep that is correct - https://github.com/livepeer/FFmpeg/issues/19 was only a problem on Windows and the error described in this issue was observed on a Linux machine using NETINT.

Any hint on how can I reproduce the problem described above?

Unfortunately I don't have additional reproduction steps as this error was just observed in the logs of a node using NETINT. We can keep an eye out for additional reports of this error and follow up if there are clearer reproduction steps that can be taken.

MikeIndiaAlpha commented 2 years ago

@yondonfu Thanks for a clarification. 'high' is likely to be h264 stream profile name, so I'll try to look around for profile setting. Other than that I guess you're right, need to have more.

thomshutt commented 2 years ago

fname /home/nvme/.lpData/arbitrum-one-mainnet/f3843dd4fef32d3f4a10.tempfile
[h264_ni_enc @ 0x7f6704a44740] [Eval @ 0x7f68c3ffe200] Undefined constant or missing '(' in 'high'
[h264_ni_enc @ 0x7f6704a44740] Unable to parse option value "high"
[h264_ni_enc @ 0x7f6704a44740] Error setting option profile to value high.
ERROR: encoder.c:261] Error opening video encoder : Invalid argument
ERROR: transcoder.c:209] Unable to open output : Invalid argument
E0412 07:48:41.935128   39791 ffmpeg.go:790] Transcoder Return : Invalid argument
E0412 07:48:41.935230   39791 orchestrator.go:555] manifestID=335498b7-3ab3-4042-8e85-ddb19e438078 seqNo=813 orchSessionID=b5d87b17 clientIP=89.187.185.153 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Error transcoding segName= err="Invalid argument"
E0412 07:48:41.935472   39791 segment_rpc.go:234] manifestID=335498b7-3ab3-4042-8e85-ddb19e438078 seqNo=813 orchSessionID=b5d87b17 clientIP=89.187.185.153 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Could not transcode err="Invalid argument"
E0412 07:48:43.841583   39791 orchestrator.go:178] manifestID=335498b7-3ab3-4042-8e85-ddb19e438078 seqNo=814 orchSessionID=b5d87b17 clientIP=89.187.185.153 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E Error receiving ticket sessionID=b5d87b17 recipientRandHash=4e05857562430c1edfd11917ee2c52a197a494ae3d8b9305f838d326dd270e1d senderNonce=94: TicketParams expired
E0412 07:48:43.841633   39791 segment_rpc.go:107] manifestID=335498b7-3ab3-4042-8e85-ddb19e438078 seqNo=814 orchSessionID=b5d87b17 clientIP=89.187.185.153 sender=0xc3c7c4C8f7061B7d6A72766Eee5359fE4F36e61E error processing payment: TicketParams expired

This is the latest set of logs - should we add some additional logging of params received and ship a patched version to help diagnose this since it's down in the ffmpeg call, or is there anything useful we can get from finding out the params passed to the binary? @MikeIndiaAlpha @cyberj0g

cyberj0g commented 2 years ago

It occurs here, but I'm still unable to reproduce it: https://github.com/livepeer/lpms/blob/b0c6462e7b364e008b949ee50cace8970bd39020/ffmpeg/encoder.c#L259 I believe profile:high should be passed there always, but it doesn't seem to be the case, or it doesn't trigger error with test data. Maybe add some logging around this bit, WDYT @MikeIndiaAlpha?

This issue seem to be specific to h264_ni_enc and is not related to other similar errors. I think one of previous experimental versions for Netint used xcoderparams structure to set profile, maybe it won't work otherwise, then we need to make sure profile is passed differently through this structure, in case of Netint: https://github.com/livepeer/lpms/blob/b0c6462e7b364e008b949ee50cace8970bd39020/ffmpeg/encoder.c#L256

MikeIndiaAlpha commented 2 years ago

Looking at the FFmpeg code at that server I can also see that NETINT guys did modify option parser. I think I can check, maybe they introduced some problems there. The question is, where I can find current/latest NETINT ffmpeg changes. Do you happen to know @cyberj0g?

MikeIndiaAlpha commented 2 years ago

Update:

Checked over NETINT changes in the option parser, doesn't seem like this is the problem
Had a hunch that maybe option value is the problem. Code complains about "high", while proper profile name is (I think) "High". 'git grep' in ffmpeg sources seem to confirm. So my current theory is that basically the problem is with uppercase vs lowercase 'H'. Digging around to learn more.

MikeIndiaAlpha commented 2 years ago

It looks like video options are actually set at upper level Golang code (ffmpeg.go function Transcode, which does pretty complex parameters processing before actually calling lpms_transcode). So this would be the best place for printing the options out, as I don't believe they are changed deeper (in the C code)

cyberj0g commented 2 years ago

@MikeIndiaAlpha thanks for insights, I tried passing profile as "High" and it didn't work either. Looks like Netint encoder expect profile to be passed differently. I updated transcoding script on the server to make reproducing the error easier, it will always use high profile now:

sudo ./transcoding ~/bbb_265.mp4 P720p25fps16x9 nt 0

And that's how profile is passed through Ffmpeg API: https://github.com/livepeer/lpms/blob/b0c6462e7b364e008b949ee50cace8970bd39020/ffmpeg/encoder.c#L259 The AVDictionary will have profile:high in it.

The error:

[h264_ni_enc @ 0x4104540] [Eval @ 0x7fff5d83cd70] Undefined constant or missing '(' in 'high'
[h264_ni_enc @ 0x4104540] Unable to parse option value "high"
[h264_ni_enc @ 0x4104540] Error setting option profile to value high.

xdwq2008 commented 2 years ago

We use "-xcoder-params" to set the FW params on ffmpeg. You can use below code: av_opt_set(c->priv_data, "xcoder-params", "xxx", 0);

MikeIndiaAlpha commented 2 years ago

Not sure if this is relevant, but when going through ffmpeg.go configuration parsing I found out that the "High" profile and "ConstrainedHigh" profile are handled differently. For "High" profile there is special handling for Netint HW and profile is set via xcoderOutParamsStr. For "ConstrainedHigh" default handling is used BUT in videoprofile.go this profile name is changed to "high" as well.

So to sum things up, we have two profiles that look different in the config: High and ConstrainedHigh. Internally they both map to "high" profile name. And then the same name is handled differently on Netint HW: for ConstrainedHigh it is passed as normal encoder options (incorrectly?) and for High profile is passed by xcoderOutParamsStr (correctly?).

What do you think @cyberj0g ?

cyberj0g commented 2 years ago

@MikeIndiaAlpha 'constrained high' is 'high' without B-frames, which is how it is configured for Software mode and Nvidia. For Netint, we got clarifications from them, that profile indeed should be passed through xcoderParams structure, which is how it currently implemented for high profile, but we didn't yet get info on how to disable B-frames to make it 'constrained'. So you are right, 'constrained high' is not configured correctly in current version, because we don't know how to do it. Now, it will result in above error and Broadcaster will select a different Orchestrator, which, I think, is better than producing output not as requested.

livepeer / lpms

Undefined constant or missing '(' in 'high' error when transcoding certain segments using NETINT #312