search

ittiam-systems / libmpegh

MPEG-H 3D Audio Low Complexity Profile Decoder. Encoder: https://github.com/ittiam-systems/libmpeghe
http://www.ittiam.com/
BSD 3-Clause Clear License
87 stars 20 forks source link

Extract individual objects? #19

Closed sclsj closed 1 year ago

sclsj commented 1 year ago

From my reading of other issues it seems like 360 Reality audio downloaded from Tidal is object-based, and during decode it is mapped to multichannel output. Is it possible to extract the individual objects? I really like the track and want to figure out what each instrument does in this as well as recreate parts on piano based on listening, and having objects (I assume one or a group of instruments?) isolated is very helpful for that.

For an example audio, please use this track. If this is not possible, if convenient please let me know of anything you can think of that might help me with this process. Thank you so much for your help!

SakethSathuvalli commented 1 year ago

Hi,

Please refer to our https://github.com/ittiam-systems/libmpegh/blob/main/docs/IA-MPEG-H-3D-Audio-Dec-GSG.doc and https://github.com/ittiam-systems/libmpegh/blob/main/docs/IA-MPEG-H-3D-Audio-Dec-API.doc if any information is available for your use case.

Thanks!

sclsj commented 1 year ago

I'm sorry, I'm not very literate in this topic. I tried to read the guide but can't really understand it and so I can't find revevant information. Am I correct in understanding that this is not something I can do with the command-line program but something that might be possible by using the library?

SakethSathuvalli commented 1 year ago

Hi @sclsj,

Please refer to the entries in tables 3-1 and 3-2 of the API document. These tables have details about the buffers that carry the object audio information when external rendering is enabled. The interfaces are provided as per the specification for MPEG-H and expects the user of this data to have prior knowledge of programming and digital audio formats(PCM data of multichannel).

Thanks!

sclsj commented 1 year ago

Unfortunately I do not know any programming language. Is there a demo or sample code that allows extraction of object data (i.e. MPEG-H 3d mp4 --> adm bwf) anywhere?

SakethSathuvalli commented 1 year ago

Hi @sclsj,

We have recently added a wiki page on external rendering interfaces. Can You please check if it helps you with the issue you are facing?

Thanks!

sclsj commented 1 year ago

Hi @SakethSathuvalli

It definitely helped. However, it seems like I'm getting a lot of popping noises, perhaps due to some problems with the limiter, for example limiter not applied and resulting in heavy clipping. How can I set a lower global gain (for example, multiple every gain for object by 0.5x so it won't clip)? Or maybe disable DRC for objects completely? I can't find an option for either.

sclsj commented 1 year ago

Below is an example. See the first 2 channels that are heavily clipped/limited.

截屏2023-01-31 11 21 26
sclsj commented 1 year ago

I think these weird samples are why there are constant popping sounds. It seems like the clipped samples become negative? I'm not sure what's happening but it's definitely because the signal clipped.

截屏2023-01-31 11 21 22
SakethSathuvalli commented 1 year ago

I think these weird samples are why there are constant popping sounds. It seems like the clipped samples become negative? I'm not sure what's happening but it's definitely because the signal clipped.

截屏2023-01-31 11 21 22

@sclsj, Can You please let know if the waveform shown above is of the audio data file generated using -ext_ren:1 flag ?

SakethSathuvalli commented 1 year ago

Hi @sclsj,

Try once with our latest commit - 3b23f65

Thanks!

sclsj commented 1 year ago

Yes, it’s from the raw pcm generated by extren flag. I will try the latest commit.

获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: Saketh Sathuvalli @.> 发送时间: Wednesday, February 1, 2023 2:33:17 AM 收件人: ittiam-systems/libmpegh @.> 抄送: Jin, Joy @.>; Mention @.> 主题: Re: [ittiam-systems/libmpegh] Extract individual objects? (Issue #19)

This email originated from outside Davidson College. Use caution, especially with links and attachments.

Hi @sclsjhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsclsj&data=05%7C01%7Cjojin%40davidson.edu%7C1df4472e39b24813b1a508db04269704%7C35d8763cd2b14213b629f5df0af9e3c3%7C1%7C0%7C638108336088116109%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=5CrfXNLwJXWeYVTFsY3N01oJ4cGQn3CXSfBZycDCe8Q%3D&reserved=0,

Try once with our latest commit - 3b23f65https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fittiam-systems%2Flibmpegh%2Fcommit%2F3b23f65bf7750847e2a46c3092fda9b75ac60a26&data=05%7C01%7Cjojin%40davidson.edu%7C1df4472e39b24813b1a508db04269704%7C35d8763cd2b14213b629f5df0af9e3c3%7C1%7C0%7C638108336088116109%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Gyala4GxIocFodNUFPJR4SkUO4pRI2HDJzZVZQaMGc8%3D&reserved=0

Thanks!

― Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fittiam-systems%2Flibmpegh%2Fissues%2F19%23issuecomment-1411587286&data=05%7C01%7Cjojin%40davidson.edu%7C1df4472e39b24813b1a508db04269704%7C35d8763cd2b14213b629f5df0af9e3c3%7C1%7C0%7C638108336088273194%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=L%2FgMExHUlk7htN7eu%2BbWnl3HR6ZP%2FnS52ZCQ4TUsCn4%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAJG5FLDXJXHZGH7LDTAAGZDWVIGT3ANCNFSM6AAAAAARE2I234&data=05%7C01%7Cjojin%40davidson.edu%7C1df4472e39b24813b1a508db04269704%7C35d8763cd2b14213b629f5df0af9e3c3%7C1%7C0%7C638108336088273194%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lQBrN08AgcGR%2B%2FHMrVK3nckk5APMqgzHEGq54nqtqYI%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

sclsj commented 1 year ago

Hi @sclsj,

Try once with our latest commit - 3b23f65

Thanks!

It definitely worked.

However, the objects are still peak limited. How can I set a lower gain for objects?

For the channel output, I messed around with files containing "gain" and lowered the gain (therefore disabling DRC) by modifying all gain in impeghd_peak_limiter.c to 0.1. What would be the corresponding edit that would set a lower object gains (perhaps times a factor of 0.1)?

SakethSathuvalli commented 1 year ago

Hi @sclsj,

The DRC effect will not be applied on the external renderer interfaces PCM output. It appears that the original content itself is peak limited in nature.

Thanks!

SakethSathuvalli commented 1 year ago

Hi @sclsj,

We would like to close this issue if You can confirm that you are able to extract individual objects with the -ext_ren: switch.

Regarding the gain, the decoder does not have a provision to apply desired gain on decoded objects.

Thanks!

sclsj commented 1 year ago

Thank you for all your help! Yes, I was able to use the ext_ren switch. Just want to confirm again that no processing is done to the decoded objects for extren, and all the drc metadata and such are included in the bitstream file?

sclsj commented 1 year ago

Hi @sclsj,

Try once with our latest commit - 3b23f65

Thanks!

I'm still curious why that happened. From my experience you can exceed 0 dB in 24 bits? E.g. if I record in QuickTime the peak usually > 0dB and I have to apply like a -10dB gain. Even if 16 bit (e.g. convert from 24 bit to 16 bit in-place in Audition on file with >0dB peak), shouldn't be audio just be clipped? I thought that's the normal behavior?

And also why would that happen in the first place? Shouldn't the encoded objects be peak normalized/limited in the first place (by the encoder)?

And could you explain how did you fix this (e.g. lower the gain?)?

sclsj commented 1 year ago

Also, it seems like the new version of libmpegh fails to decode 360 RA files from Deezer.

ia_mpeghd_testbench -ext_ren:1 -ifile:/Users/jin/Music/deemix\ Music/YOASOBI\ -\ THE\ BOOK\ 2/07\ -\ もしも命が描けたら.mp4 -ofile:mingdeemix.wav
-ifile:/Users/jin/Music/deemix Music/YOASOBI - THE BOOK 2/07 - もしも命が描けたら.mp4 -ofile:mingdeemix.wav 
                       ITTIAM SYSTEMS PVT LTD, BANGALORE
                             http:\\www.ittiam.com
                      IA_MPEG_H_3D_AUD_DEC_ARM $Rev: 1.2 $

fatal error: Ittiam mpegh_dec  core coder module :Initialization: : Initialization fatal error
SakethSathuvalli commented 1 year ago

Also, it seems like the new version of libmpegh fails to decode 360 RA files from Deezer.

ia_mpeghd_testbench -ext_ren:1 -ifile:/Users/jin/Music/deemix\ Music/YOASOBI\ -\ THE\ BOOK\ 2/07\ -\ もしも命が描けたら.mp4 -ofile:mingdeemix.wav
-ifile:/Users/jin/Music/deemix Music/YOASOBI - THE BOOK 2/07 - もしも命が描けたら.mp4 -ofile:mingdeemix.wav 
                       ITTIAM SYSTEMS PVT LTD, BANGALORE
                             http:\\www.ittiam.com
                      IA_MPEG_H_3D_AUD_DEC_ARM $Rev: 1.2 $

fatal error: Ittiam mpegh_dec  core coder module :Initialization: : Initialization fatal error

The failure could be because the stream configurations may not be supported by our decoder. Example the stream could belong to a higher profile level.

SakethSathuvalli commented 1 year ago

Hi @sclsj, Try once with our latest commit - 3b23f65 Thanks!

I'm still curious why that happened. From my experience you can exceed 0 dB in 24 bits? E.g. if I record in QuickTime the peak usually > 0dB and I have to apply like a -10dB gain. Even if 16 bit (e.g. convert from 24 bit to 16 bit in-place in Audition on file with >0dB peak), shouldn't be audio just be clipped? I thought that's the normal behavior?

And also why would that happen in the first place? Shouldn't the encoded objects be peak normalized/limited in the first place (by the encoder)?

And could you explain how did you fix this (e.g. lower the gain?)?

The fix is related to limiting the upper and lower limits of a 24-bit data. These are referred to as saturation fixes.

sclsj commented 1 year ago

Hi @sclsj, Try once with our latest commit - 3b23f65 Thanks!

I'm still curious why that happened. From my experience you can exceed 0 dB in 24 bits? E.g. if I record in QuickTime the peak usually > 0dB and I have to apply like a -10dB gain. Even if 16 bit (e.g. convert from 24 bit to 16 bit in-place in Audition on file with >0dB peak), shouldn't be audio just be clipped? I thought that's the normal behavior? And also why would that happen in the first place? Shouldn't the encoded objects be peak normalized/limited in the first place (by the encoder)? And could you explain how did you fix this (e.g. lower the gain?)?

The fix is related to limiting the upper and lower limits of a 24-bit data. These are referred to as saturation fixes.

Can you tell me a bit more? I failed to get more information with basic search. Did you just clip the samples?

SakethSathuvalli commented 1 year ago

Hi @sclsj, Try once with our latest commit - 3b23f65 Thanks!

I'm still curious why that happened. From my experience you can exceed 0 dB in 24 bits? E.g. if I record in QuickTime the peak usually > 0dB and I have to apply like a -10dB gain. Even if 16 bit (e.g. convert from 24 bit to 16 bit in-place in Audition on file with >0dB peak), shouldn't be audio just be clipped? I thought that's the normal behavior? And also why would that happen in the first place? Shouldn't the encoded objects be peak normalized/limited in the first place (by the encoder)? And could you explain how did you fix this (e.g. lower the gain?)?

The fix is related to limiting the upper and lower limits of a 24-bit data. These are referred to as saturation fixes.

Can you tell me a bit more? I failed to get more information with basic search. Did you just clip the samples?

Yes, the samples will be clipped to maximum / minimum value if they are found to exceed.

sclsj commented 1 year ago

Got it. What should I do if I don't want it to clip? Is it possible to reduce the gain or to output as 32-bit float, which wouldn't have this problem?

SakethSathuvalli commented 1 year ago

For the individual objects it is not possible to apply gain.

SakethSathuvalli commented 1 year ago

Even when float output is given the values exceeding [-1.0, 1.0] will be clipped. However, support for float output is currently not available

sclsj commented 1 year ago

But is it possible to solve this one way or another? Clipping is still bad, although not nearly as bad as inverted samples.

sclsj commented 1 year ago

Even when float output is given the values exceeding [-1.0, 1.0] will be clipped. However, support for float output is currently not available

I'm asking about float since QuickTime Player on macOS records in 24-bit float and every time I record most of the samples are > 0dB. I just do a -12dB volume reduction and it's ready to be used.

SakethSathuvalli commented 1 year ago

A not so clean way is to tap the output before clipping. This has to be done in the code. Please check the fix given for saturation overflow. It can help you with the buffer containing audio data that is not clipped.

Thanks!

sclsj commented 1 year ago

Emm... I just replaced that with pcm_sample = pcm_sample * 0.5f and it seemed to work... Other than increased noise floor, what problems might that cause?

SakethSathuvalli commented 1 year ago

When You try to apply gain on the output again the clipping behavior will be seen again when the data is stored as 16/24 bit PCM. Other than this I don't see any other implications.

Thanks!

sclsj commented 1 year ago

I don't think I understood you. Could you please explain more? Are you saying that if this 0.5 gain is not low enough for certain files, the ext_ren pcm will still be clipped?

Also, why would the encoded objects be greater than 0dB? Shouldn't the encoder clip any values exceeding [-1.0, 1.0]? I don't understand why, if the encoded objects can be greater than 1.0, the decoded output can't be with a significantly higher bitrate. I'm sorry if this question seems silly.

Thank you for all your help!

SakethSathuvalli commented 1 year ago

All I am saying is when You apply a gain of 2.0 or more on this output(pcm_sample = pcm_sample * 0.5), the samples will get clipped.

The sample values exceed the range [-1.0, 1.0] by a very marginal value(< 1e-2). This is due to the implementation and is expected. Also this can happen even if the encoder takes care of clipping.

Thanks!

sclsj commented 1 year ago

Hello again! You locked #38 due to off-topic, so I'm unable to delete or hide my comment. You have my consent to delete or hide my comment.