Question about the future; HEVC/AV1

[CristianGrecu]

Hi, as you may know, more and more streamers are testing 4k HEVC streams. Rumors point to additional announcements around Twitchcon US.

I know you've said you don't really want to add support for AV1 since not many devices are capable of hardware decoding. It seems that short term this HEVC codec will be available for everyone to use. Is it even possible to integrate it in this app?

I'm just thinking that most TVs nowdays are 4k so it would be a massive improvement. Leaving resolution aside, I think the other incredible thing is that the bitrate behaves differently, so when a lot of stuff happens on stream, it doesn't become a blurry mess anymore.

We all know that Twitch's official app will be horrible, so if more and more streamers start using 4k HEVC permanently, a regular Desktop PC migh be mandatory for watching on a TV.

Thank you!!

[fgl27]

Is possible, but you need a av1 ready device.

The app will continue to show avc streams as is the only supported by all devices.

The avc will continue to be used even if the stream is av1, so content will just work

The day this is a real feature I can add a way to detect supported devices.

[CristianGrecu]

AV1 is probably more of a 2025 thing, but what about this HEVC that everyone is testing? If now we are using H.264 and they are testing H.265, shouldn't most devices support it since it's some sort of avc+? I assume that the transcoding is still mostly done by twitch unlike AV1 which makes the users handle most of it.

Also, I thought when a device can't hardware decode it just falls back to software decoding if the CPU is decent. My Sony TV shows AV1 when watching some Youtube 4k videos, but I don't think it has dedicated hardware.

I'm going to be honest, I even love using this app on a tablet, even if it doesn't have regular touch controls.I don't even want to think about using the official apps, mobile or otherwise, they're horrible. I really hope we won't be left behind with 1080p when even really old smart TVs can handle Youtube 4k (which I think is usually VP9).

Thanks for answering and sorry if I got anything wrong, I'm not a developer.

[fgl27]

Is all the same and part of the same beta program.

If the device support you will be able to use it.

You can't on most tv devices use sw decoder due to the low performance.

You can test it today in the app you can disable the hw decoder and try it

Most devices struggle with h264 SW decoding so anything more complex is not a option for high resolution, anything over 640p is high.

[CristianGrecu]

I see, thank you! I guess we'll have to wait and see what happens when they enable it for everybody. Right now they don't even save VODs/clips from these tests, so everything is gone.

When I was watching on a PC, the bitrate was still low, seemed to be 10-12 mbps, but 4K. Codec was showing HEVC and it looked really nice.

Just one more question cause I don't want to open another thread. I noticed that recently drops started to work perfectly. I am just finding random stuff to claim all the time. Will it ever be possible for this app to properly detect viewing time and collect channel points/receive gifted subs etc.?

Edit: Just a side note because I just checked. My TVs SoC which is a very modest Mediatek shows in the spec list: Decoding: HEVC 8k 60fps, H. 264 8k 30FPS. So I hope most low-mid range TVs will do alright.

[fgl27]

On the av1 topic

There is a test hidden stream

If you search for videos "1440p120 av1" it shows. It will only play using avc 1080p on the app, but on twitch site it plays using av1.

https://www.twitch.tv/videos/637388605?t=18h36m31s

I enabled av1 and twitch only sends av1 links not av1 and avc, not a problem is just extra request that will be needed to show it all.

On the 2017 shield it can't play av1 it gets codec errors.

There is a lot of devices like this

If anyone find a h.265 test vod share the link so I can try it.

[fgl27]

Channel points probably not unless twitch changes things today isn't allowed.

Gifts works, as it is given for those logged in the chat, as long you are setup with yours user and chat logging is enabled must work.

I received a gift once and I only use this app for twitch so it must works for others.

The app can send a on screen warning if you get one.

[CristianGrecu]

My bad, I used the wrong wording. Getting a gift if you are targeted works, but you don't get random gifts as a loyalty reward. Back when I was watching from a PS4 in 2021 I used to have like 12 subs for free every month just because the system was seeing me as loyal to some streams. So I would have some sort of priority vs people that watch for a couple of minutes and leave. Nowdays 0 random gifts because it's as if I'm not even there.

[CristianGrecu]

I don't think there are any VODs for HEVC. Can you do anything so we can test the next time someone is live? Like a hidden setting or something?

Oh btw, only the 4K feed was HEVC, the rest were standard avc. Which is another problem in my opinion. I remember when twitch allowed 1440p streams and it would only be 1440p and then straight to 720. It's possible that in the future avc will only get 720p and the rest will be new codecs.

I still can't believe how well Youtube works and barely anyone streams there. I tested a VP9 1440p stream and it was flawless.

[CristianGrecu]

I think I found one. I'm not at the PC right now to check but live I am 100% sure it was 4k HEVC.

https://www.twitch.tv/burkeblack/v/2211849267?sr=a&t=0s

[fgl27]

My bad, I used the wrong wording. Getting a gift if you are targeted works, but you don't get random gifts as a loyalty reward. Back when I was watching from a PS4 in 2021 I used to have like 12 subs for free every month just because the system was seeing me as loyal to some streams. So I would have some sort of priority vs people that watch for a couple of minutes and leave. Nowdays 0 random gifts because it's as if I'm not even there.

I never used an official app, so I'm not aware of all things.

I assume this is also forbidden, as is similar and attached to the same way points are calculated.

Not that is forbidden for say forbidden, is just that isn't allowed an app that will generate points or anything related, as one can use that just to span and get things they don't deserve.

An app using such implementation can be easily banned no questions asked of how and why it does.

The "points" are more than just opening the stream, there are several API requests that need to be made to receive it, I assume the sub and anything related are the same.

If someday that changes we can revise it.

[fgl27]

I think I found one. I'm not at the PC right now to check but live I am 100% sure it was 4k HEVC.

https://www.twitch.tv/burkeblack/v/2211849267?sr=a&t=0s

On the av1 topic

There is a test hidden stream

If you search for videos "1440p120 av1" it shows. It will only play using avc 1080p on the app, but on twitch site it plays using av1.

https://www.twitch.tv/videos/637388605?t=18h36m31s

I will use this two VODs to setup something once all is done I share it with you.

thanks!

[CristianGrecu]

Thank you!!

[fgl27]

I have updated the app to better support those vods, they played just fine but not the best quality out of the box.

in this test apk they do.

app-debug.zip

install this after extract, this is a testing app it will add a second SmartTwitchTV with a yellow icon.

this app shares no context with the Play Store one you must setup add users etc.

If any issue let me know.

If you see any stream playing AV1 or HEVC let me know here.

Once I improve this version I release it.

[fgl27]

I have added this to the settings, technically needs to be set to yes to make it work

For those two testing VOD it does nothing, disable will not fall back to avc, I assume that isn't ready yet on the Twitch side.

Also, no guarantee enabling will make it work even if the streamer is using the codecs.

The day we find some live using those codecs we can better test.

[CristianGrecu]

I know for a fact that Maximilian_dood has been doing test streams for HEVC 4k and he usually has Twitch staff present. I'll keep an eye out. Other streamers mentioned they want to use 1440p 60 permanently in the future.

Honestly I don't think anyone is testing AV1 publicly so that might have to wait.

And thanks again for all the work.

[CristianGrecu]

That test app seems to work perfectly honestly. I'll test some more on the tv tomorrow because it's getting late. Live will be the true test though.

[fgl27]

I have made an updated version

app-debug.zip

will test this for a few days, test let me know the result.

extra codec and Codec capability & Blocked codecs are improved

the app allows to control the codec and let the user know if one is available

The final version will have a proper text on above settings, that is just for testing the functionality that isn't affected by the text

[fgl27]

This is the final or almost final version, improve some text and quality selection

app-debug.zip

Test this one and let me know

BTW this streamer has HEVC VODs, maybe they do some live in the future for testing

r0dn3y

[CristianGrecu]

I've done some testing. Here's what I found:

I tried a 4k60 HEVC vod, a 1440p60 Elden Ring vod and that AV1.

On the 60hz Sony TV with a gigabit usb network adapter :

4k60 HEVC: seems to work perfectly, 0 skipped frames 1440p60 HEVC: same as above. AV1: completely broken, blurry and it seems to be on fast forward. After a while graphical artifacts appean on 50% of the screen.

On a 144hz Snapdragon 870 tablet:

4k60 HEVC: almost perfect, like 1 skipped frame. This tablet has a 3k screen I think, so really good overall. 1440p60 HEVC: perfect, no skipped frames, really crisp image. AV1: no graphical artifacts, still broken. Blurry image and seems to be on fast forward even though the tablet is 144hz.

Honestly, I wouldn't bother with AV1 until they do public tests, because 120 FPS is too much for normal use anyway.

[fgl27]

Av1 is device issue most devices can't play av1 I have a phone with a 888 snapdragon plays fine but no audio.

Besides that av, hevc is working great

Will test for a few days then release it

[CristianGrecu]

But don't you think the 120 fps is the main issue? Because my TV runs 4k60 on youtube using AV1 without a problem.

[fgl27]

You can check the codec capability in settings to know what the tv can and can not

Just because is av1 doesn't mean it's the same thing

There several ways to encode that one stream isn't the standard.

[CristianGrecu]

We'll have to see when they start testing 60 fps AV1. That VOD is also quite old so who knows what changed.

It's very nice that HEVC works well, so thank you again. I'll keep looking for a live test but it should be fine.

[CristianGrecu]

Btw, unrelated, but have you considered adding an auto catch-up function? Sometimes when Twitch is having a bad day I have to manually get it back to live with the remote (shrinking the red bar as much as possible to the right) to reduce the delay as much as I can. It would be nice if let's say every 30 mins - 1 hour the stream would auto catch-up to live.

[fgl27]

there is a player setting that is disabled by default that does that, isn't exported to the user, I'll see to make it configurable

[CristianGrecu]

Ok, thank you!!

[fgl27]

I enabled the feature to keep the stream position even after buffers and related

app-debug.zip

It works automatically, there is no need to enable another setting, enable low latency on the player and it will do the work.

Normal is around 3 to 4 seconds Lowest 2 to 3 Disabled 14

It isn't possible to go lower as Twitch streams are segmented in 2s, anything lower will cause rebuffers

[CristianGrecu]

It's usually not a problem, I only mentioned it because yesterday I was having issues with literally 3 streams (out of like 50). In the right corner where it says net speed I usually see 200-300 and the same average. Sometimes just a few streams show 15-20 net speed even if the average is still 200. It is very strange but I think it's Twitch 's fault. Even now Kyedae' s stream does the same: net speed shows 17 and average 200 which means the stream is on the edge of buffering all the time. You can check it if you want, but it's probably an Eu problem.

Also, a stream that was doing it yesterday fixed itself today. I really don't get it.

[fgl27]

OK, but that feature is good, I disabled it in the past because was causing issues as it didn't allow configurations but now is working as expected.

So at the start or after a buffer, the player will always try to correct and find the target position.

[CristianGrecu]

Yes, of course, I meant I didn't want to bother you with more things!

[CristianGrecu]

So I've been using the new app, the latest one you posted, and I think I found a problem. Right after I join a stream I can hear some audio crackles/pops and static. It seems to go away after a while.

I kept going back and forth and the released app doesn't do this. Was there anything changed related to audio? Or could it be the auto catch-up thing misbehaving? (streams don't buffer so why would it)

[fgl27]

I didn't notice it, maybe is device-related...

The catch-up happens every time that the latency is off.

At the beginning, you can see the latency value (top right) going down or up depending on how the playback started.

I made two testing app tests to see if any diff, test let me know.

I slowed down the catch-up to see if was it now you may not notice it.

the only diff of this apk is that one will catch up the other will not.

app-debugs.zip

[CristianGrecu]

I tested it on the tablet and also using some Beyerdynamic dt770 and an amp on the TV, so I don't think it's device related because the old one doesn't do it.

I'll test with this one you posted and I'll be back asap.

[CristianGrecu]

So those 2 apps that you posted seem to be fine. I swear only the one above from 18 hours ago is the one that does it. Quite noticeable pops and crackles, I got it to do it on 7 different streams. Could it be a bug with that apk only?

It's quite random so it's hard to pinpoint the cause. I'm sure the current version never did it like that. Would it be possible to leave the catch-up function as a toggle just to be safe?

[fgl27]

I slowdown the catch up system that is the only diff

Just test that one if you notice any issue let me know.

Thank you!

[CristianGrecu]

So I tested the latest app with catch-up. Optical into amplifier and DT770 and it still happens. I tried multiple streams including plain music so I can hear the background. The old app sounds really clean but with the new one there's constant pops but lower volume than the content itself.

I'll admit that through TV speakers at medium-low volume it's probably not that pronounced. But I've been using this app for years so I'm more sensitive to changes I guess.

Also, and this is speculation only, it seems so stabilize after a few minutes, so if I change streams it happens but somehow it settles if I stay for some time. (not sure)

[fgl27]

You can look at the top right corner latency to broadcast it get lower and lower until reach the expected value

2.5 or 4.5 seconds is the expected.

Once the value gets to that value is when the issue stops?

The app without catch has the issue? I shared two apps on last post, which has the issue?

[CristianGrecu]

Yes I was comparing the current one with the latest catch-up, because I wanted to switch between them (can't have 2 new yellow ones installed). But my latency doesn't slowly go lower like you said. Just now I clicked on a stream and it just sits at 1.30. Others just start at 2.7.

Btw, I have the buffer size at 0.1 in the settings. This is how I've been using it for years and I had no problems.

But you're right, once it settles it holds the delay better than the old app.

[fgl27]

What I need to know is if the one that doesn't have catch on the name has the issue.

Because if it does the problems is something else

[CristianGrecu]

I'm trying the one without catch now and I can't get it to do it at all, sound is very good.

The one from 18 hours ago is the worst one, loud enough crackle to hear it even through speakers, especially when going from stream to stream fast.

Maybe just keep it tuned like the recent one, should be ok.

[fgl27]

OK I'll try another version later, one that allows to disable/enable the catch

At the same time slow down even more then we test again

Thank you for the help, great work, by myself I would not notice that issue.

[CristianGrecu]

No problem, I also kept looking for an HEVC live but I couldn't find any so that will have to wait. Other than that it seems to work really well.

[fgl27]

app-debug.zip

this has the option in setting making it easier to test for audio issues.

As it is tied to low latency, you can disable the player by simply disabling the low latency feature.

to disable all set low latency as below

[CristianGrecu]

Ok, so you can still independently have low latency with or without the catch up, right? But disabling just low latency is faster for testing if I understand correctly.

I'll check and I'll tell you if I find anything that doesn't work. Thank you.

[fgl27]

yes, you can leave the setting option 'catch-up low latency' disabled, and low latency will work.

and if you enabled 'catch-up low latency', and wanna disable it from the player just disable low latency and all will be off, for easier and faster testing

[CristianGrecu]

So I've been testing the app for a while. From what I can tell the audio issues are very minimal.

I was wondering if the catch up function helps with the natural delay drift. I thought it didn't but now I'm not sure anymore (in case you saw the first post).

So If I leave a stream on I usually move that white dot above the pause all the way to the right, to bring the delay as low as possible. After some time, the old app slowly gets more delay even with 0 buffers. So after 1-2 hours it's slighty behind a newly opened app. It's not a big deal because I just manually move the white dot again bringing it as close to actual live as possible. It's very subtle, sometimes it can be 0.5 seconds, sometimes a bit more.

I'm sure the catch up works with buffers, but I didn't get any so I can't test it fully.

But I'm sure this is nitpicking and most people don't care. It's just that obviously the Windows Chrome streams are still the absolute best when it comes to delay. Your app is probably a close second. The TV and console apps are terrible. At that point it's not even "live" anymore.

[CristianGrecu]

I've testing some more and it seems to maintain the delay better than the old one, so I think I was wrong. And sorry for the wall of text above.

[fgl27]

don't worry

latency depends on so many variables, a simple thing as your device clock being a little off in relation to the server clock or streamer can make it go up/down

Android OS and clock sync is something that is very often not perfect.

[CristianGrecu]

Oh, I see. There's still that one problem left, but it's also in the old app. Just a few streams every single day are always on the edge of buffering.

Sometimes it's a single stream out of many. I can tell because up in the right corner the net speed shows 15-17 instead of 200+ and those streams are very sensitive. After a bit the net speed dips to 6, it buffers and then goes back to 15-17.

I tried an unlimited 4G sim card so I know it's not my internet/DNS. It also doesn't happen on PC. I still don't think there's anything you can do but I wanted to let you know.

Unless it gets worse, it is what it is. Those streams are still watchable on source, but they are on the edge at all times.

[fgl27]

the app will use all available bandwidth, doesn't matter what your ISP is the limit is most of the time is on the Twitch server.

So using auto-quality playback is the best option to avoid buffers as it will always lower the quality buffer, if that doesn't work is because the internet stops not a bandwidth issue but a connection issue