Sound is tinny and cuts in and out

[jkhartshorne]

What I do:

Record in-app

What I expect

To have reasonable quality sound, and for it to not cut in and out.

What happens

It's hard to hear anything. It's alternately tinny and muffled. And it cuts in and out.

Machine

Samsung Galaxy S20, in Chrome

Notes

This was a problem we had early on and it was solved. I don't know when the problem came back, because I haven't done much in-app recording in the meantime. I usually record using the built-in video app and upload, but lately I've been making longer recordings, which can't be done that way (because of the upload limits).

@lorenachipstudio @SecondStreetDevelopment

[jkhartshorne]

PS I just recently recorded a very long recording (946.7 seconds), so should be easy to find and see for yourself.

[lorenachipstudio]

@jkhartshorne We're taking a look at this now! Thank you for reporting it!

[jkhartshorne]

Please get this fixed as soon as possible. We were in the midst of a lot of advertising, and now I've had to put it on hold -- again -- because I don't want to shop around a broken app.

[SecondStreetDevelopment]

As we discussed on Monday, this really could only be occurring in one of two areas, either the recording step or the encoding step. I know others are looking into this but I'm going to take a quick look at the waveform and encoding of the 946.7 second file and see if that sheds any light on where the issue is occurring. I'll also create a new 15 min test (benchmark) clip. The existing white noise clip that we've been using isn't very effective for evaluating sound quality issues. I'll report back later today.

[SecondStreetDevelopment]

Here are the details for the recording in question. It is a mono channel file with bit rate of 64 kb/s and sample rate of 44,100 Hz which is exactly what we would expect based on the settings in the encoding cloud function. I created a 15 minute test audio file and ran it through the production site. The output mp3 file properties were exactly as we would expect as well. I did not detect any audio issues with the 15 minute test file. The audio issues don't appear to be related to the encoding stage. The cloud function logs for the recording in question also are clean (see screen capture). I wonder if this issue could be related to playback?

Here is a link to the 15 minute test file which was processed through the production site: https://www.dropbox.com/s/6miriqz9j2id7xm/1600844400220_905.787188_ignore_ignore_output.mp3?dl=0

[SecondStreetDevelopment]

Here is the waveform for the recording in question. Besides some extended silent portions later in the file, I don't see any obvious issues:

[jkhartshorne]

If it's playback, that's interesting. Because KidTalk doesn't have any problem playing back audio that was recorded outside of the app and then uploaded. That sounds great. Keeping in mind that this is recorded and played on the same device. I also don't have any problem with recording and playing back using other apps on the same device. Also, the playback issue isn't device-specific: I have the same experience replaying that same audio on other devices.

On Thu, Sep 24, 2020 at 1:39 AM Mitchell notifications@github.com wrote:

Here is the waveform for the recording in question. Besides some extended silent portions later in the file, I don't see any obvious issues:

[image: image] https://user-images.githubusercontent.com/49379014/94105189-99f0e800-fded-11ea-82cc-c0cad83113d8.png

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jkhartshorne/KidTalkIssues/issues/56#issuecomment-698124327, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOVVYZNDNR7M543MFRANZ3SHLLR5ANCNFSM4RVBMOWQ .

[SecondStreetDevelopment]

@jkhartshorne, thank you for that info. Can you download your recording and take a quick listen for me? Do you hear the "tinny" sound issue when playing this MP3 directly?

Download link: https://www.dropbox.com/s/k68sb25z4ywbrxa/1600724705291_947.10_ignore_ignore_output.mp3?dl=0

[alexchippendale]

@jkhartshorne We've taken a look at the actual saved recording and observed that the recording itself is silent in some regions and plays correctly in others. After looking at the database records, the actual saved audio file, and many tests within the app, this doesn't appear to be a direct problem with playback, processing, or otherwise a software bug in the KidTalk code.

While it is hard to know with absolute certainty, it does look a lot like your phone went to sleep at times while recording which is why there are periods of silence. The recording becomes silent after 2 minutes, sound then comes in again at 3:05 which could be from tapping the screen or if the phone got any accelerometer input from movement or vibrations, and then goes silent again at 5:05, exactly 2 minutes after.

Web browsers, unlike mobile or desktop apps, may not signal their OS to keep the mic active after the screen dims or locks (or prevent them from locking in the first place). This depends on the device used and it's settings. That is why you would have no problem recording and playing back on mobile apps on the same device. The playback issue isn't device specific, because rather than being a playback issue it is the actual audio file which doesn't include audio in the regions where the phone went asleep and the mic was not active.

The easy solution to prevent this for your device is to adjust the screen lock or timeout settings (every device is a little different and sometimes have multiple settings related to this). KidTalk is not currently encountering errors processing or playing audio. We are now looking into techniques to force the browser to keep the device awake regardless of device or settings for cases like this where device settings are overriding the browser.

[jkhartshorne]

I just made two recordings in production. "Tinny 1" was made in app. "Tinny 2" was uploaded. The differences aren't as extreme as in the previous multi-person recording, but I think it's pretty clear.

I agree that the long silences are probably from the phone going to sleep. I guess the downloadable app will fix that. Though the cutting in and out i mentioned was actually during recording. Sometimes voices are so muffled as to be inaudible.

Josh

On Thu, Sep 24, 2020, 6:06 PM alexchippendale notifications@github.com wrote:

@jkhartshorne https://github.com/jkhartshorne We've taken a look at the actual saved recording and observed that the recording itself is silent in some regions and plays correctly in others. After looking at the database records, the actual saved audio file, and many tests within the app, this doesn't appear to be a direct problem with playback, processing, or otherwise a software bug in the KidTalk code.

While it is hard to know with absolute certainty, it does look a lot like your phone went to sleep at times while recording which is why there are periods of silence. The recording becomes silent after 2 minutes, sound then comes in again at 3:05 which could be from tapping the screen or if the phone got any accelerometer input from movement or vibrations, and then goes silent again at 5:05, exactly 2 minutes after.

Web browsers, unlike mobile or desktop apps, may not signal their OS to keep the mic active after the screen dims or locks (or prevent them from locking in the first place). This depends on the device used and it's settings. That is why you would have no problem recording and playing back on mobile apps on the same device. The playback issue isn't device specific, because rather than being a playback issue it is the actual audio file which doesn't include audio in the regions where the phone went asleep and the mic was not active.

The easy solution to prevent this for your device is to adjust the screen lock or timeout settings (every device is a little different and sometimes have multiple settings related to this). KidTalk is not currently encountering errors processing or playing audio. We are now looking into techniques to force the browser to keep the device awake regardless of device or settings for cases like this where device settings are overriding the browser.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jkhartshorne/KidTalkIssues/issues/56#issuecomment-698613612, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOVVY37YU4INRXUI443KO3SHO7IFANCNFSM4RVBMOWQ .

[jkhartshorne]

BTW Just to repeat: we did have the same problem with tinniness early in development. You fixed it then. I wonder if that fix just got accidentally reverted along the way.

[SecondStreetDevelopment]

Here is a link to the MP3 for "Tinny 1": https://www.dropbox.com/s/molvab8p93ip7qv/tinny%201.mp3?dl=0

Here is "Tinny 2": https://www.dropbox.com/s/lz9d5ku9by6a46i/tinny%202.mp3?dl=0

[jkhartshorne]

@SecondStreetDevelopment - did you want me to reply to that, or was that just for your records? I hope you can hear the difference, right?

[SecondStreetDevelopment]

@jkhartshorne, I can hear a slight difference in the two files but they both are pretty clear when played on my laptop. Do they sound tinny on your end?

[jkhartshorne]

The first one is quite muffled and tinny. That's on multiple devices.

On Tue, Sep 29, 2020 at 1:59 PM Mitchell notifications@github.com wrote:

@jkhartshorne https://github.com/jkhartshorne, I can hear a slight difference in the two files but they both are pretty clear when played on my laptop. Do they sound tinny on your end?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jkhartshorne/KidTalkIssues/issues/56#issuecomment-700882936, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOVVYZ6AVKTZG7Q4TM747LSIIOBNANCNFSM4RVBMOWQ .

[jkhartshorne]

@SecondStreetDevelopment - have you confirmed that the original fix hasn't been reverted?

[SecondStreetDevelopment]

Yes, we discussed this on Monday. This was before my time but David tells me that issue was caused by a polyfill in index.html which was commented out. See image, it has been uncommented ever since that original issue:

[SecondStreetDevelopment]

I'm admittedly at a loss on this issue. Both audio files sound roughly how I would expect them to from a consumer level or smartphone microphone. However, I'm certainly not an audiophile...so not the person you'd want to seek out to evaluate audio. The only knob I know of that we could turn to change audio quality would be the MP3 bitrate but I don't believe it has been changed, that's something David would know. Of course, increasing MP3 bitrate would come with it's own set of drawbacks.

I'll take a step back and ruminate on this issue today. Perhaps something will come to me.

[jkhartshorne]

I'm not sure why you can't hear the difference. Have you tried a different device? I had several people listen to the MP3s you posted, and they didn't have trouble hearing the difference.

The difference is much more pronounced when multiple people are talking. I have posted two new recordings (two of my three most recent) to production. "good night" was recorded in-app, whereas "good night 2" was uploaded. "Good night" is very tinny, and frequently the speech cuts out or is muffled so that you can't make out what was being said. There are no such problems with "good night 2".

Note that both recordings were made around the same time, with people in the same positions and with the device in the same place.

Josh

On Wed, Sep 30, 2020 at 10:59 AM Mitchell notifications@github.com wrote:

I'm admittedly at a loss on this issue. Both audio files sound roughly how I would expect them to from a consumer level or smartphone microphone. However, I'm certainly not an audiophile...so not the person you'd want to seek out to evaluate audio. The only knob I know of that we could turn to change audio quality would be the MP3 bitrate but I don't believe it has been changed, that's something David would know. Of course, increasing MP3 bitrate would come with it's own set of drawbacks.

I'll take a step back and ruminate on this issue today. Perhaps something will come to me.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jkhartshorne/KidTalkIssues/issues/56#issuecomment-701446630, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOVVYZ2SSHELHCHP5GGM3DSINBUPANCNFSM4RVBMOWQ .

[jkhartshorne]

@SecondStreetDevelopment - I wanted to make sure you got this.

[SecondStreetDevelopment]

I do notice a difference between these two recordings. It sounds like in "good night" the MP3 may be exhibiting signs of over compression. I'm not sure if this would be happening during the MP3 conversion step or not. I'll get with David and see if it's possible to increase the bitrate used during that step. I'm not certain it is the source of this difference in audio quality but perhaps we could make that change and see if it produces any noticeable results. (I'm a bit limited because I don't have an obvious way to reproduce this behavior on my own. Audio recorded in-app on my phone (Galaxy Note 9) has been quite clear thus far...at least to my ears.)

Here is a link to the MP3 for "good night": https://www.dropbox.com/s/69ahsfqvqocwap5/good%20night.mp3?dl=0

Here is a link to the MP3 for "going to bed 2": https://www.dropbox.com/s/kq8kvgyqt086gg9/going%20to%20bed%202.mp3?dl=0

[jkhartshorne]

The effect is much larger when you have multiple people talking, so maybe try that on your end to see if you can reproduce?

On Thu, Oct 1, 2020 at 3:06 AM Mitchell notifications@github.com wrote:

I do notice a difference between these two recordings. It sounds like in "good night" the MP3 may be exhibiting signs of over compression. I'm not sure if this would be happening during the MP3 conversion step or not. I'll get with David and see if it's possible to increase the bitrate used during that step. I'm not certain it is the source of this difference in audio quality but perhaps we could make that change and see if it produces any noticeable results. (I'm a bit limited because I don't have an obvious way to reproduce this behavior on my own. Audio recorded in-app on my phone (Galaxy Note 9) has been quite clear thus far...at least to my ears.)

Here is a link to the MP3 for "good night": https://www.dropbox.com/s/69ahsfqvqocwap5/good%20night.mp3?dl=0

Here is a link to the MP3 for "going to bed 2": https://www.dropbox.com/s/kq8kvgyqt086gg9/going%20to%20bed%202.mp3?dl=0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jkhartshorne/KidTalkIssues/issues/56#issuecomment-701934873, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOVVY2ZQVS4CXUTN4KBTOLSIQTAVANCNFSM4RVBMOWQ .

[SecondStreetDevelopment]

I'll give that a try.

[jkhartshorne]

Kathleen reports: I did not replicate this issue on my iPhone 7S, iOS 13.

[SecondStreetDevelopment]

Noted. I suppose it could have something to do with the unique audio compression that is being done on some smartphones and how it interacts with the audio compression we're using on Kidtalk. I'll look into that.

[SecondStreetDevelopment]

@jkhartshorne, I have increased the MP3 bitrate by a factor of four on the staging site. Can you do another set of tests on the staging site and let me know if you notice any improvement?

[jkhartshorne]

I'm not sure. It might be better, but still not as good as native recording. Is it possible to turn it up more, or is it already ridiculously high? Josh

On Fri, Oct 2, 2020 at 12:42 AM Mitchell notifications@github.com wrote:

@jkhartshorne https://github.com/jkhartshorne, I have increased the MP3 bitrate by a factor of four on the staging site. Can you do another set of tests on the staging site and let me know if you notice any improvement?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jkhartshorne/KidTalkIssues/issues/56#issuecomment-702521770, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOVVY57R5N4LCF26IA44J3SIVKZ5ANCNFSM4RVBMOWQ .

[SecondStreetDevelopment]

I just increased it to the maximum. Let me know if you notice any improvement.

Increasing the bitrate also increases the file size quite a bit. I'm wondering if this new issue could be caused by me increasing the bitrate: https://github.com/jkhartshorne/KidTalkIssues/issues/61

I'm not sure it will be possible to get it as good as native recording. Many of the premier smartphones (like the Samsung Galaxys) have very sophisticated multi-microphone arrays. The built-in recording app on these smartphones utilize custom APIs to unlock the full feature set of these mics. Those APIs won't be available to a JavaScript based web app.

One example of the type of microphone technology they're utilizing on flagship smartphones: https://www.samsung.com/global/galaxy/what-is/zoom-in-mic/

[jkhartshorne]

I don't know. I tried it and I'm not sure it sounds a lot better. But I also made a recording with production this morning under similar conditions and today it was actually decent for whatever reason. Maybe we should just focus on getting to the downloadable app, where you'll be using the native microphone capabilities, right?

[SecondStreetDevelopment]

We will have much more control in the mobile app. I can't speak to specifics as David and Alex have made the call on what platform is being used for the mobile app(s). But generally speaking, the mobile app will allow for greater microphone control.

[SecondStreetDevelopment]

I thought I might try a variable bitrate setting for the audio conversion. Theoretically, this should be the best of all worlds. I just deployed this change to the staging site. Feel free to run a test and let me know if you notice any difference.