Closed pglowacky closed 2 years ago
gioelecerati
commented
2 years ago Looks like it's not a player issue, but a scaling issue with the 360p resolution! Link for the specific resolution with the green bar: https://livepeercdn.com/recordings/871da71b-9c15-405d-b34f-df65501d3641/360p0.m3u8
In any case, this mean that by default the player is loading the lower resolution on the first .ts instead of the higher one, could be changed to avoid the grainy video at the start
gioelecerati
commented
2 years ago This also happens in Live streams, so it's almost surely related to how it's transcoded, cc @AlexKordic
MikeIndiaAlpha
commented
2 years ago OK, so let's look into this. I got all relevant streams. I won't post original video because it is huge. It is 2160x3840 (so 9:16 aspect ratio, or 16:9 in "portrait"). Now, we are talking about version transcoded to "360p" profile. 3840/360=10.666... and 2160 divided by 10.666... is about 202.5. And indeed, the stream with the green bar has 202x360 resolution, and the test stream transcoded on my Mac using transcoding tool also has 202x360 resolution BUT no green bar. Looking into the stream, both have the same geometry. That is, they are 13 macro blocks wide and 23 macro blocks tall, which means 208x368 resolution of the frames. Then the cropping rectangle in both is left = 0, top = 0, right = 6, bottom = 8 so displayed image starts at top-left corner of the frame and is 202x360 pels. So far so good.
Let's make a screenshots of the very first frames of respective streams, for example by ffmpeg -I stream.h264 -frames:v 1 -f image2 first_frame.png. Now, comparing these frames side-by side (my preferred way of doing this is to have both frames in the same directory with preview enabled, and use arrow keys to move preview between the files) we can easily see that while SW transcoded file has neatly scaled and filtered image, and no green bar, the green bar file has very ugly scalling (like without filtering at all, just "nearest" sampling), and there is green bar encoded in the image itself.
So the first conclusions would be: h.264 encoder seems innocent, because stream geometry is identical in both cases. The culprit seems to be scaling before the encoding - in green stream case it is done badly, with little or no filtering and it doesn't fit the proper geometry, either. Please see attached files for yourself.
green_bar.ts.zip
(green_bar.ts got zipped because GitHub refuses to upload .ts)
MikeIndiaAlpha
commented
2 years ago I decided that it is better to upload original video as well - problem may be related to something in that so here it is:
MikeIndiaAlpha
commented
2 years ago After some more testing: 1) "graininess" or generally more high frequencies in the image seems to be the property of Nvidia encoders. See attached software and nvidia transcoded videos. There is no "right" or "wrong" here, it is just a different approach to quality. Software encoded video is blurry and loses details, hardware encoded image is too sharp and shows up aliasing. Note that the original video is not really h264-friendly, for example the way the hand and the cat is cropped out the background will challenge any block-based format. 2) But the video encoded on the machine I have access to doesn't show "green bar" :-(. I am thinking it may be issue with particular hw/driver combination. I will look up the code to see how the scaling is done, AFAIR scaling is a "filter" for the Nvidia HW encoders...
gioelecerati
commented
2 years ago Thank you for all of this Michal! Do you think it may be due to the scaling made before encoding in Livepeer or on the client? As far as we know the iPhone submitting the video may be scaling down the 4k video before pushing it to the RTMP endpoint
However, I found some references online about this issue happening to some people while using scale_qsv or while scaling using QuickSync
MikeIndiaAlpha
commented
2 years ago @gioelecerati No problem, this is actually something I specialise in and am happy to help.
Funny thing, I had to upload the video to notice - NV encoded version has different dimensions than SW encoded one. When I was testing with transcoding cat.mp4 P360p30fps16x9 sw I got the 202x360 resolution, when I did transcoding cat.mp4 P360p30fps16x9 nv 0 I am getting 360x640 one. So while we don't have exactly same problem, we do have a problem and again it is a scaling one. I will now look carefully into LPMS source to remind myself the details of scaling, but I seem to remember that scaling is done differently for SW and HW pipelines. In fact, I think it is done different way on SW/HW/Nvidia HW which may explain various problems.
QSV is yet another tech (Intel's), but this is definitely before the encoder. In fact, this issue is not assigned correctly, it is a problem with transcoder, of that I am sure. It ain't client.
gioelecerati
commented
2 years ago I don't know if this can help, but trying to push the same original file from the terminal with vcodec copy and acodec copy the green bar is not present on the lower resolution, while the issue appears if I set up vcodec libx264.
However, with h264_nvenc the green bar appers, but not on the lower resolution, 360x640 is fine and 480x854 has the green bar!
In general I have 2 separate master manifests, this (with the green bar)
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=21860800,RESOLUTION=2160x3840,FRAME-RATE=24,CODECS="avc1.640034"
0/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1773816,RESOLUTION=406x720,FRAME-RATE=30,CODECS="avc1.4d401f"
1/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=887200,RESOLUTION=270x480,FRAME-RATE=30,CODECS="avc1.4d401f"
2/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=441664,RESOLUTION=202x360,FRAME-RATE=30,CODECS="avc1.4d401e"
3/index.m3u8Or this
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1627152,RESOLUTION=2160x3840,FRAME-RATE=24,CODECS="avc1.4d4033"
0/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=1980376,RESOLUTION=720x1280,FRAME-RATE=30,CODECS="avc1.4d401f"
1/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=990480,RESOLUTION=480x854,FRAME-RATE=30,CODECS="avc1.4d401f"
2/index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=501176,RESOLUTION=360x640,FRAME-RATE=30,CODECS="avc1.4d401e"
3/index.m3u8@gioelecerati Great catch!
transcoding tool has no 854x480 by default, but it is easy to create one by adding to videoprofile.go. For example like that:
ERROR = VideoProfile{Name: "ERROR", Bitrate: "1000k", Framerate: 30, AspectRatio: "16:9", Resolution: "854x480"}
Now, when I encode with the "ERROR" profile name on sw encoder, I get proper stream in 854x480, no green bar. But when I switch to nv encoder, I get 854x480 stream all right, but with a green bar. Great.
Basically it seems that we found two issues with transcoder. One being that some profiles cause encoding in different output res depending on the encoder in use, and the other one that some profiles may cause NVenc to emit green bars in output streams. Great.
Not sure what you guys want to do, but my proposition is as follows:
What do you think @gioelecerati ?
gioelecerati
commented
2 years ago Not sure what you guys want to do, but my proposition is as follows:
- You close the issue on your side
- I will open two issues on the LPMS side
What do you think @gioelecerati ?
Makes sense! I am going to close this issue, so you can go on and open the ones on LPMS! cc @pglowacky
Currently, portrait assets played in the dashboard appear grainy in the first few frames and are accompanied by a small green bar at the bottom. Assets played on other test players playback flawlessly.
Ex:
Since we're swapping out the player by launch, it's likely not necessary to address in the meantime, however it should be on our radar as something to test before launch.
cc @clacladev