search

muaz-khan / RecordRTC

RecordRTC is WebRTC JavaScript library for audio/video as well as screen activity recording. It supports Chrome, Firefox, Opera, Android, and Microsoft Edge. Platforms: Linux, Mac and Windows.
https://www.webrtc-experiment.com/RecordRTC/
MIT License
6.58k stars 1.76k forks source link

Audio record ios 11 no data #324

Open iamrototo opened 7 years ago

iamrototo commented 7 years ago

Since iOS 11 and the support of getusermedia, I would like to use the record feature for safari on iPad but is not working at all.

I checked this issue (https://github.com/muaz-khan/RecordRTC/issues/275). I have adapted my code to use StereoAudioRecorder. I get no error in the console but just that I record nothing (the data is 44 Bytes and is just empty) despite it works on chrome (PC).

Here is my dumb test index. I try to record the voice of the user for 1 second and then I stop and get the data. To easily debug I push in the html the base64 and I have a quick audio html5 to be able to play the sound.

<!DOCTYPE html>
<html>
<head>
<script src="https://webrtcexperiment-webrtc.netdna-ssl.com/RecordRTC.js"></script>
<script src="https://cdn.webrtc-experiment.com/gumadapter.js"></script>
</head>

<body>
<audio id="toto" controls></audio>
<div id="result" style="max-width:500px;     word-wrap: break-word; "></div>

<script>

function successCallback(audioStream) {
    // RecordRTC usage goes here

    var recordRTC = RecordRTC(audioStream, {recorderType: StereoAudioRecorder});
    recordRTC.startRecording();

    setTimeout(
        function() {recordRTC.stopRecording(function(audioURL) {
            var recordedBlob = recordRTC.getBlob();
            recordRTC.getDataURL(function(dataURL) { 
                var audio = document.querySelector('audio');
                audio.src = dataURL;
                document.getElementById("result").innerHTML = dataURL;
            });
        });}, 1000);
}

function errorCallback(error) {
    // maybe another application is using the device
}

var mediaConstraints = { audio: true };

navigator.mediaDevices.getUserMedia(mediaConstraints).then(successCallback).catch(errorCallback);

</script>
</body>
</html>

Thanks for your help.

dan24678 commented 6 years ago

@kaliatech I tested your page on https://recordrtc-ios-demo.glitch.me and do not see any indication that you have solved the iOS bug by making a new audiostream. If you sleep your phone and then unlock it, recording on iOS becomes completely broken until you make a new tab (tested on an iPhone 6S on 11.2.6). Reloading the page won't fix it, which is indicative to me that the problem is very low level and possibly not something we can resolve or prevent through code. (After reporting the bug to Safari's bug tracker Apple finally responded asking me to retest in the latest iOS beta, which I can't immediately do. Can you confirm the bug is still present there?). Also, note that your demo appears broken in Chrome and FF on OS-X, where it captured only silence.

For this reason, I am making iOS behave such that a new tab is always opened as an integral part of the recording process. I'm very thankful that this is even an option because without it, I would not have the confidence that iOS Safari recording can be done reliably enough to release it to my users.

Speaking of which, I am closing in on actually releasing this to my company's customer base and the implementation is unfortunately and by necessity complex:

To reiterate my requirements, I want a recording volume meter that captures audio in the browser and then uploads a wav file to our servers.

iOS uses the implementation laid out by @GersonRosales, except instead of sending the output through Lame mp3 encoder, I'm sending it through https://github.com/brianjlowry/Recorderjs to get my wav file. I'm also using window.open(), window.close() and window.opener.document to construct a flow where a new pop-up/authorization flow is created for every single recording. Uploading is done via FileReader().readAsDataURL() since "new FormData().append()" doesn't seem to work for user-created audio in Safari. I have not yet been able to break this implementation. It seems rock solid. knocks wood

Desktop Safari also uses the above approach, but without opening popups. When I tried using RecordRTC with desktop Safari, there was very bad distortion in the resulting audio file so I think any Safari support in RecordRTC is effectively too unreliable for Production use at this point.

For all other browsers (including Android phones) I am using RecordRTC. I was also able to leverage the volume meter code from GersonRosales' demo to work alongside RecordRTC.

I'd prefer to be able to use RecordRTC for all browsers, since it is more heavily supported and more likely to keep on top of the changing/deprecating APIs, but it simply doesn't work in Safari, so I needed to adopt this weird, hybrid approach.

kaliatech commented 6 years ago

@DrLongGhost There are a multiple issues in earlier discussions. To be clear, the issue I set out to workaround in that demo was the ability to start/stop/save, start/stop/save multiple recordings in a single page without a reload or new tab.

That said, I tested again today to see if I could replicate the issue you are describing when device is locked, and I could not. The recording and playback on that demo page always works after waking/unlocking, without reloading or opening a new tab. A colleague was unable to replicate your issue as well. Similarly, the demo I posted seems to work for us on on all major os/browser/device combinations. I've personally tested mac(10.12.6-sierra)-safari/firefox/chrome, windows(10)-chrome/firefox/edge, ios/safari, & android/chrome. Photo as proof: https://www.dropbox.com/s/u62wej9anlekwr1/recordrtc-tests-20180307.jpg?dl=0 .

My colleague was also using an iPhone 6s with 11.2.6. So i'm at a loss as to how the demo could be working without issue for us and not for you, especially if audio recording does work for you with example_3 above and similar.

dan24678 commented 6 years ago

Well, I believe you that you aren't seeing it but I definitely am. Additionally, @shejazi reported the identical issue in the comments above with a 5S. I tested on my wife's 6S Plus on 11.2.6 and was able to see the bug present on her device as well. This is exactly what I'm doing:

1) Record audio on your page then play it back. Works fine. 2) Click home button 3) Click sleep 4) Unlock 5) Open Safari. 6) Page is already loaded so I click Record then play it back. Only silence was recorded.

kaliatech commented 6 years ago

Strange. I believe you as well, just would like to understand what it isn't working for you so I could feel confident others aren't going to run in to this too. I'm sure the same is true for you in the opposite direction. Fwiw, here's video of me stepping through those steps on an iPad Pro 11.2.6: https://youtu.be/mSA7gqrO-fE

rowntreerob commented 6 years ago

@kakiatech your links were fine for me.. iphone7 - 11.3beta fwiw - another thread's accepted answer may illuminate some of the confusing ios symptoms.

Those apps that just like to place a record button as a single prompt and a single UI clik still need to follow the link's advice and to both instantiate the context and to wire up the nodes and onAudio events in response to the single UI event. I think it can get a bit confusing when you move from the 2 button clik (one for the init and call to 'getUserMedia', one to startRecord).

dan24678 commented 6 years ago

@kaliatech -- I tested some more this morning and your implementation does seem to work a lot better than I initially thought. The unlock bug seems to be gone or at least harder to trigger than it was yesterday. I think I may have had mic misconfiguration issues yesterday on my desktop which would explain why I now see your page working in Chrome and FF. I'm not sure why it suddenly seems more stable on iOS though, unless it was some sort of user error on my part, or the code has been changed since what I was testing.

Unfortunately, I am still seeing 2 issues with your implementation (one minor, one major):

  1. Click through to make a recording successfully. Click the home button to minimize Safari.
  2. Open "Voice Memos" on iOS and make a recording there (while the red notification bar is still showing from Safari indicating Safari is also recording)
  3. Return to Safari and try another recording. It will capture only silence. Sometimes a simple reload will fix this but I was also still able to see the tab "poisoned" and require an entirely new tab to set it right.

Are you able to replicate the above?

kaliatech commented 6 years ago

RE: Red Safari Recording Bar I was able to get this to disappear by calling audioStream.getTracks[0].stop(). Calling stop also changed the red microphone button in safari url bar to show muted status. (The user can also click the icon directly). In order to record again after stopping/muting, a new audio stream must be created.

RE: Unable to Record again After Another App Records I see this issue as well. After using demo to record, then switching to any native app and recording, then switching back to demo page in Safari, results in silent recordings. Interestingly, the red icon in url bar is also missing when switching back. Clicking to get permission again results in the microphone prompt, but the red icon doesn't come back and the recorded audio is silent. In my testing, nothing seemed to fix this except force quitting Safari. After force quitting, demo recording worked again. This only happened if the demo page was open and red icon displayed (muted or not) when switching to the native app to record. If demo recordings were done, but navigated to different page before switching, then going to back to demo page to record worked fine. This seems like a lower level ios bug and I can't think of any workarounds to try at the moment. Should be a relatively uncommon scenario for most use cases, but would be frustrating for any that run in to it.

dan24678 commented 6 years ago

Opening a new tab fixes the issue without requiring a force-quit for Safari. I feel like the underlying bug that causes the "poisoned" tabs (as I call them) will still manifest itself randomly in various scenarios involving sleeping your phone and/or leaving a tab with audio context enabled backgrounded for extended periods. The bug(s) seems less likely with your approach but I suspect it is still there and will continue to trouble users.

I'm sticking with the idea of opening a new tab every time I want to initiate recording in iOS (then closing it when recording is complete). With this method, I have never seen it break. There are trade-offs, but for my use cases, I think it offers the best chance of ensuring the most users a successful recording.

dan24678 commented 6 years ago

@kaliatech -- You may get to say I told you so here...

I released iOS 11 recording into Production just now and immediately was able to break it. I think my testing wasn't thorough enough because I missed a scenario (or just got lucky in earlier testing) where if you sleep your phone on Tab A and then Tab A opens Tab B, Tab B will be broken. The whole thing is a mess and I'm increasingly worried there is just some fundamental wonkiness to iOS recording which we will be unable to resolve completely.

Anyway, I reverted my release and I'm going to take a closer look at your solution and see if it improves things.

dan24678 commented 6 years ago

I've made a lot of great progress this week, most notably I think I figured out what the main source of the instability in iOS is. Putting your phone into lock mode, opening another app that uses the microphone or backgrounding Safari all seem liable to breaking audio recording on a given tab in Safari if the audio steam is left in place and not removed off the page.

Almost all of the iOS issues I was seeing disappeared once I started using the following code to clean up in between my recordings:

stopRecording() {
  this.processor.disconnect();
  this.processor = null;

  this.audioContext.close();
  this.audioContext = null;

// The conditional lets me safely call the function if we have an audioContext but no stream yet
 if (window._audioStream) {
   window._audioStream.getTracks().forEach(track => track.stop());
   window._audioStream = null;
}

The key here seems to be explicitly setting the context and stream to null. If I comment out the lines where I set these items to null, I immediately see recording problems return, whereas with this code in place, there are no issues remaining, afaict.

I'm going to continue rambling on about some other stuff I'm doing and stuff I found in my testing, but I think my finding above was the big find (at least in my implementation). With my current code in place, I am only able to "poison" a tab if I open Voice Memos and record in there while Safari is also recording. Everything else, including sleeping the phone, no longer causes issues.

Despite this, I actually came up with a great way to handle potential microphone problems. I was able to leverage @GersonRosales getVolume() function to keep track of all 0 volume 10ms segments. If you get only 0 volume for the first 3+ seconds of the recording, then you can safely assume the user has audio misconfigured (or in the case of an iPhone that the tab is poisoned). I then immediately show an error message with steps to fix it.

For iOS, the poisoned tabs can be fixed by one of two methods: 1) Force-Quit Safari 2) The user manually opens a new tab via the Plus button in Safari

I find it intriguing that a manually opened new tab always gives you a working tab 100% of the time, but a new tab created via window.open() or <a target="_blank"> does not. I wish I understood why this was.

@kaliatech / @GersonRosales -- I'm curious to see if you can replicate my findings. I now have a solid, working implementation that doesn't depend on either of the fixes (opening a second audio stream and keeping the audio stream open in between recordings) that @kaliatech put in place. I'm now using something very close to Gerson's Example 3 with my new bug fix to the StopRecording() method and some extra logic around error handling and other things.

EDIT TO ADD: I just realized that you apparently can't leave ANY open references to objects directly associated to the audio stream when you sleep Safari or it breaks. I just now found another race condition in my code where I wasn't unsetting RecorderJS and I think that was causing issues as well. I'm not sure if RecordRTC needs to be unset or reset as well, since I'm not using it with Safari. I think the general idea is as soon as your recording is done, grab the wav file or mp3 or whatever you intend to save (which is fine to keep in memory), and unset everything else to ensure the tab doesn't break if it's put to sleep. Even then, I think it still might break every once in awhile, but seems far less likely.

ClOne20 commented 6 years ago

Great work @DrLongGhost Dan. I'm still waiting for perfect stability, but everything looks fine now for the most part?

dan24678 commented 6 years ago

@cloone1 -- I hope so. We have error reporting via Sentry on the site I'll be rolling this out to, so I can actually report back in a couple weeks and let you know how many errors versus successful recordings in iOS we end up seeing.

danielstorey commented 6 years ago

Has anyone tried addressing these issues by creating a new instance of the stream by doing something like window._audioStream = window._audioStream.clone()?

This worked for me when I was unable to record more than once on the original stream. I don't know whether this would fix issues of restoring after locking as well...

kaliatech commented 6 years ago

@danielstorey - Cloning the audiostream does seem to work as an alternative to calling getUserMedia again to get a new stream, and might be easier to use in some implementations. It doesn't seem to work if the stream's tracks are stopped though.

@DrLongGhost - ~I can not replicate your findings if the track is stopped and context closed. When track/context is closed after recording is complete, then switching apps seems to break things almost every time. If the track/context is left open, then switching apps tends to be okay.~ UPDATE: 20180410 - After removing dependencies to ensure everything is cleaned up when recording is finished, I can now replicate. I no longer have stability issues when switching apps or sleeping/locking, even if closing tracks.

I'm no longer user RecordRTC so won't continue here, but in case it helps, I built a cleaner demo showing everything I've learned.

octavn commented 6 years ago

This bug report on bugs.webkit.org tracks the MediaRecorder API implementation in WebKit and thus Safari.

I encourage everyone to show their interest and add a comment describing how a MediaRecorder implementation would help their project/development. Creating an account takes a second.

Chrome implemented MediaRecorder in Jan 2016. By that time the MediaRecorder feature request on bugs.chromium.org had gathered 2800+ stars.

Also, WebKit is an open source project so one can also contribute code towards implementing the MediaRecorder API.

dan24678 commented 6 years ago

Here's some data from having the approach to iOS recording we worked out in this thread in Production for awhile. Since July, we saw 26 successful recordings produced with iOS measured against 15 failures from 10 unique users. So, it seems to work less well than I'd hoped but it's definitely doing the job for some of our users.