search

iceeburr / roblox-video-codec

STILL IN DEVELOPMENT, NOT READY FOR USE! Efficient encoding/decoding, and display of video files within Roblox. Seamless integration for any game.
https://devforum.roblox.com/t/archive-video-player-v11-play-your-own-video-inside-roblox-full-resolution-fps-and-audio/2423381
MIT License
3 stars 1 forks source link

Will you add a external audio codec? #1

Open OfficiallyMelon opened 5 months ago

OfficiallyMelon commented 5 months ago

Would you somehow add support for audio (possibly via Base64 encoding & decoding) or something :P

Thanks

iceeburr commented 5 months ago

This project has been paused due to life complications, but I am planning to continue working on it soon.

It is completely possible to implement custom audio inside of Roblox, but I don't think you understand how complex that would be. This project is here as a proof of a concept, not to be used in practical solutions. Already, this completely bypasses any kind of moderation. I might try to do something with audio, but I doubt it would be possible. A lot of people underestimate the complexity of the stuff we use in our daily lives. Computer science is hard, and I am definitely not the one to reinvent the wheel. Feel free to try it on your own. The specification document for the HEVC (High Efficiency Video Coding) standard is let alone over 900 pages. You can read more about it here.

If you wanted to make a game without moderation and restrictions, you shouldn't be trying Roblox in the first place. This is not a real platform and is not used for real game development. You can learn quite a lot about programming concepts and actual game development, but that is only due to the fact that Roblox as an engine is so bad that it forces people to invent stuff that should be built in, for example, occlusion culling or the most simple shaders (rendering effects). Sorry, but you "ask for the impossible."

The latest commit on this repository is the most progress I have made with it. If I had to describe what happens, in short, it will read all the frames first and do preparation before the playback. This way, it stores them in memory and can actually reduce consumption. There is a lot of useless data in JPEG images that can be cut off. By doing this, I can leave the initial parsing of the JPEG headers. When it's time to decode the pixel data, it won't have any issues doing that. It is already aware of where to look and won't need to run loops through the whole frame to find anything. I haven't tested this theory enough, but I hope it will work in the future. Again, I am actively trying to get back into this project after having stopped working on it for quite a while.

This has been a really good way for me to learn how computers work. This is why I started this in the first place. The demo version was already good enough to prove it's possible. I want to continue grinding my skills further and finish this once and for all. Soon enough, I will rewrite the project structure again and make sure to optimize every last bit of code.

Also, as a last note, I want to explain why audio might be even harder to implement. There is simply no API that allows us to play audio as we want. Before, there were no editable images or meshes. We could still decode the video itself, but how do you then play it back? Right now, video decoding is harder than playback, as we have a very good way of displaying the pixels on screen with editable images. With audio, it's the opposite; it would be much easier to decode but more complex to play back. You would probably need to create thousands of audio files, link all of their IDs together, and set their frequencies. Then the playback engine would look at the current audio frequency that should play, find the nearest one we have in the data, or mix several together, then play them for a split second. This is the only way I can imagine this being possible. Maybe we can save some uploading by using the built-in audio effects to provide a "range" that certain audio can play. We find the closest frequency in our data, then try our best to calculate values for sound effects so they can make it sound as close as possible. Audio is just sine waves. It's still math, and while I haven't actually researched how audio files work, I do know it will be much easier to decode than a video file. I hope my reply helped you understand how complex this is.

I am leaving this issue open for the future, as videos are nothing without audio. Again, I am not saying I don't want to add it; I'm saying it might be too complex. As this is an actual issue, I will let people discuss the possibilities and ways to implement it here. If I'm wrong, please correct me.

OfficiallyMelon commented 5 months ago

This project has been paused due to life complications, but I am planning to continue working on it soon.

It is completely possible to implement custom audio inside of Roblox, but I don't think you understand how complex that would be. This project is here as a proof of a concept, not to be used in practical solutions. Already, this completely bypasses any kind of moderation. I might try to do something with audio, but I doubt it would be possible. A lot of people underestimate the complexity of the stuff we use in our daily lives. Computer science is hard, and I am definitely not the one to reinvent the wheel. Feel free to try it on your own. The specification document for the HEVC (High Efficiency Video Coding) standard is let alone over 900 pages. You can read more about it here.

If you wanted to make a game without moderation and restrictions, you shouldn't be trying Roblox in the first place. This is not a real platform and is not used for real game development. You can learn quite a lot about programming concepts and actual game development, but that is only due to the fact that Roblox as an engine is so bad that it forces people to invent stuff that should be built in, for example, occlusion culling or the most simple shaders (rendering effects). Sorry, but you "ask for the impossible."

The latest commit on this repository is the most progress I have made with it. If I had to describe what happens, in short, it will read all the frames first and do preparation before the playback. This way, it stores them in memory and can actually reduce consumption. There is a lot of useless data in JPEG images that can be cut off. By doing this, I can leave the initial parsing of the JPEG headers. When it's time to decode the pixel data, it won't have any issues doing that. It is already aware of where to look and won't need to run loops through the whole frame to find anything. I haven't tested this theory enough, but I hope it will work in the future. Again, I am actively trying to get back into this project after having stopped working on it for quite a while.

This has been a really good way for me to learn how computers work. This is why I started this in the first place. The demo version was already good enough to prove it's possible. I want to continue grinding my skills further and finish this once and for all. Soon enough, I will rewrite the project structure again and make sure to optimize every last bit of code.

Also, as a last note, I want to explain why audio might be even harder to implement. There is simply no API that allows us to play audio as we want. Before, there were no editable images or meshes. We could still decode the video itself, but how do you then play it back? Right now, video decoding is harder than playback, as we have a very good way of displaying the pixels on screen with editable images. With audio, it's the opposite; it would be much easier to decode but more complex to play back. You would probably need to create thousands of audio files, link all of their IDs together, and set their frequencies. Then the playback engine would look at the current audio frequency that should play, find the nearest one we have in the data, or mix several together, then play them for a split second. This is the only way I can imagine this being possible. Maybe we can save some uploading by using the built-in audio effects to provide a "range" that certain audio can play. We find the closest frequency in our data, then try our best to calculate values for sound effects so they can make it sound as close as possible. Audio is just sine waves. It's still math, and while I haven't actually researched how audio files work, I do know it will be much easier to decode than a video file. I hope my reply helped you understand how complex this is.

I am leaving this issue open for the future, as videos are nothing without audio. Again, I am not saying I don't want to add it; I'm saying it might be too complex. As this is an actual issue, I will let people discuss the possibilities and ways to implement it here. If I'm wrong, please correct me.

alright! thanks for the reply. I knew it would be super advanced... I guess I know my next project to work on!