afalls-twilio
commented
1 year ago @KamilSucharski Thank you for the bug report, have you tried with the camera2 capturer?
Closed KamilSucharski closed 1 year ago
afalls-twilio
commented
1 year ago @KamilSucharski Thank you for the bug report, have you tried with the camera2 capturer?
KamilSucharski
commented
1 year ago @afalls-twilio Yes we use the Camera2Capturer. Here is the full class that relates to the video call:
import android.content.Context
import androidx.lifecycle.MutableLiveData
import com.twilio.audioswitch.AudioDevice
import com.twilio.audioswitch.AudioSwitch
import com.twilio.video.*
import io.reactivex.rxjava3.subjects.PublishSubject
import timber.log.Timber
import tvi.webrtc.Camera2Enumerator
class RoomManagerImpl(
private val context: Context,
waitingRoomSettings: WaitingRoomSettings
) : RoomManager {
private var localVideoTrack: LocalVideoTrack? = null
private var localAudioTrack: LocalAudioTrack? = null
private var audioSwitch: AudioSwitch? = null
private var currentCameraId: String? = null
private var frontCameraId: String? = null
private var backCameraId: String? = null
private var cameraCapturer: Camera2Capturer? = null
override val isMicrophoneEnabled = waitingRoomSettings.isMicrophoneEnabled
override val isCameraEnabled = waitingRoomSettings.isCameraEnabled
override val isDoctorCameraEnabled = MutableLiveData(false)
override val doctorVideoTrack = MutableLiveData<RemoteVideoTrack>()
override val patientVideoTrack = MutableLiveData<LocalVideoTrack>()
override val isPatientVideoMirrored = MutableLiveData(false)
override val doctorJoinedEventTrigger = waitingRoomSettings.doctorJoinedEventTrigger
override val doctorConnectionChangedTrigger: PublishSubject<Boolean> = PublishSubject.create()
private var room: Room? = null
override fun setVideoCapture() {
val camera2Enumerator = Camera2Enumerator(context)
localAudioTrack = LocalAudioTrack.create(context, isMicrophoneEnabled.value ?: true)
for (cameraId in camera2Enumerator.deviceNames) {
if (camera2Enumerator.isFrontFacing(cameraId)) {
frontCameraId = cameraId
}
if (camera2Enumerator.isBackFacing(cameraId)) {
backCameraId = cameraId
}
}
val frontId = frontCameraId
val backId = backCameraId
when {
frontId != null -> {
cameraCapturer = Camera2Capturer(context, frontId)
localVideoTrack = LocalVideoTrack.create(
context,
isCameraEnabled.value ?: true,
cameraCapturer!!
)
currentCameraId = frontId
setAudio()
patientVideoTrack.value = localVideoTrack!!
isPatientVideoMirrored.value = true
}
backId != null -> {
cameraCapturer = Camera2Capturer(context, backId)
localVideoTrack = LocalVideoTrack.create(
context,
isCameraEnabled.value ?: true,
cameraCapturer!!
)
currentCameraId = backId
setAudio()
patientVideoTrack.value = localVideoTrack!!
isPatientVideoMirrored.value = false
}
else -> return
}
}
override fun connect(accessString: String, roomName: String) {
connectToRoom(accessString, roomName)
patientVideoTrack.value = localVideoTrack!!
}
override fun disconnect() {
room?.disconnect()
}
override fun stop() {
localAudioTrack?.release()
localVideoTrack?.release()
audioSwitch?.stop()
localAudioTrack = null
localVideoTrack = null
audioSwitch = null
currentCameraId = null
frontCameraId = null
backCameraId = null
cameraCapturer = null
}
override fun switchCamera() {
if (currentCameraId == frontCameraId) {
backCameraId?.let { cameraCapturer?.switchCamera(it) }
currentCameraId = backCameraId
isPatientVideoMirrored.value = false
} else {
frontCameraId?.let { cameraCapturer?.switchCamera(it) }
currentCameraId = frontCameraId
isPatientVideoMirrored.value = true
}
}
override fun enableCamera(isEnabled: Boolean) {
localVideoTrack?.enable(isEnabled)
}
override fun enableMicrophone(isEnabled: Boolean) {
if (isEnabled) {
audioSwitch?.activate()
} else {
audioSwitch?.deactivate()
}
localAudioTrack?.enable(isEnabled)
}
private fun setAudio() {
audioSwitch = AudioSwitch(context)
startAudio()
}
private fun startAudio() {
audioSwitch?.start { audioDevices, _ ->
audioDevices
.firstOrNull { it is AudioDevice.Speakerphone }
?.let { audioSwitch?.selectDevice(it) }
}
audioSwitch?.activate()
}
private fun connectToRoom(accessToken: String, roomName: String) {
val connectOptions = ConnectOptions.Builder(accessToken)
.roomName(roomName)
.audioTracks(listOf(localAudioTrack))
.videoTracks(listOf(localVideoTrack))
.build()
room = Video.connect(context, connectOptions, roomListener())
}
private fun remoteParticipantListener(): RemoteParticipant.Listener {
return object : RemoteParticipant.Listener {
override fun onAudioTrackPublished(
remoteParticipant: RemoteParticipant,
remoteAudioTrackPublication: RemoteAudioTrackPublication
) = Unit
override fun onAudioTrackUnpublished(
remoteParticipant: RemoteParticipant,
remoteAudioTrackPublication: RemoteAudioTrackPublication
) = Unit
override fun onAudioTrackSubscribed(
remoteParticipant: RemoteParticipant,
remoteAudioTrackPublication: RemoteAudioTrackPublication,
remoteAudioTrack: RemoteAudioTrack
) = Unit
override fun onAudioTrackSubscriptionFailed(
remoteParticipant: RemoteParticipant,
remoteAudioTrackPublication: RemoteAudioTrackPublication,
twilioException: TwilioException
) = Unit
override fun onAudioTrackUnsubscribed(
remoteParticipant: RemoteParticipant,
remoteAudioTrackPublication: RemoteAudioTrackPublication,
remoteAudioTrack: RemoteAudioTrack
) = Unit
override fun onVideoTrackPublished(
remoteParticipant: RemoteParticipant,
remoteVideoTrackPublication: RemoteVideoTrackPublication
) {
isDoctorCameraEnabled.value = true
}
override fun onVideoTrackUnpublished(
remoteParticipant: RemoteParticipant,
remoteVideoTrackPublication: RemoteVideoTrackPublication
) {
isDoctorCameraEnabled.value = false
}
override fun onVideoTrackSubscriptionFailed(
remoteParticipant: RemoteParticipant,
remoteVideoTrackPublication: RemoteVideoTrackPublication,
twilioException: TwilioException
) {
isDoctorCameraEnabled.value = false
}
override fun onVideoTrackUnsubscribed(
remoteParticipant: RemoteParticipant,
remoteVideoTrackPublication: RemoteVideoTrackPublication,
remoteVideoTrack: RemoteVideoTrack
) {
isDoctorCameraEnabled.value = false
}
override fun onDataTrackPublished(
remoteParticipant: RemoteParticipant,
remoteDataTrackPublication: RemoteDataTrackPublication
) = Unit
override fun onDataTrackUnpublished(
remoteParticipant: RemoteParticipant,
remoteDataTrackPublication: RemoteDataTrackPublication
) = Unit
override fun onDataTrackSubscribed(
remoteParticipant: RemoteParticipant,
remoteDataTrackPublication: RemoteDataTrackPublication,
remoteDataTrack: RemoteDataTrack
) = Unit
override fun onDataTrackSubscriptionFailed(
remoteParticipant: RemoteParticipant,
remoteDataTrackPublication: RemoteDataTrackPublication,
twilioException: TwilioException
) = Unit
override fun onDataTrackUnsubscribed(
remoteParticipant: RemoteParticipant,
remoteDataTrackPublication: RemoteDataTrackPublication,
remoteDataTrack: RemoteDataTrack
) = Unit
override fun onAudioTrackEnabled(
remoteParticipant: RemoteParticipant,
remoteAudioTrackPublication: RemoteAudioTrackPublication
) = Unit
override fun onAudioTrackDisabled(
remoteParticipant: RemoteParticipant,
remoteAudioTrackPublication: RemoteAudioTrackPublication
) = Unit
override fun onVideoTrackEnabled(
remoteParticipant: RemoteParticipant,
remoteVideoTrackPublication: RemoteVideoTrackPublication
) {
isDoctorCameraEnabled.value = true
}
override fun onVideoTrackDisabled(
remoteParticipant: RemoteParticipant,
remoteVideoTrackPublication: RemoteVideoTrackPublication
) {
isDoctorCameraEnabled.value = false
}
override fun onVideoTrackSubscribed(
participant: RemoteParticipant,
remoteVideoTrackPublication: RemoteVideoTrackPublication,
remoteVideoTrack: RemoteVideoTrack
) {
doctorVideoTrack.value = remoteVideoTrack
}
}
}
private fun roomListener(): Room.Listener {
return object : Room.Listener {
override fun onConnected(room: Room) {
Timber.e("Connected to ${room.name}")
if (room.remoteParticipants.isNotEmpty()) {
val remoteParticipant: RemoteParticipant = room.remoteParticipants[0]
remoteParticipant.setListener(remoteParticipantListener())
if (remoteParticipant.remoteVideoTracks.any()) {
isDoctorCameraEnabled.value = true
}
}
}
override fun onConnectFailure(room: Room, twilioException: TwilioException) {
Timber.e("connect failure: $twilioException")
}
override fun onReconnecting(room: Room, twilioException: TwilioException) {
Timber.e("connect onReconnecting: ${room.state}")
}
override fun onReconnected(room: Room) {
Timber.e("connect onReconnected")
}
override fun onDisconnected(room: Room, twilioException: TwilioException?) {
Timber.e("connect disconnected")
}
override fun onParticipantConnected(room: Room, remoteParticipant: RemoteParticipant) {
doctorConnectionChangedTrigger.onNext(true)
remoteParticipant.setListener(remoteParticipantListener())
if (remoteParticipant.remoteVideoTracks.any()) {
isDoctorCameraEnabled.value = true
}
}
override fun onParticipantDisconnected(
room: Room,
remoteParticipant: RemoteParticipant
) {
Timber.e("connect onParticipantDisconnected")
doctorConnectionChangedTrigger.onNext(false)
}
override fun onRecordingStarted(room: Room) = Unit
override fun onRecordingStopped(room: Room) = Unit
}
}
}
@KamilSucharski Thank you for reporting the issue. Unfortunately, we were not able to repo it. We have included a test application attempting to recreate the issue. If you can please modify it so that we can reproduce the issue using the test app, we can more easily fix the issue.
link to the test app is here
Also, just looking at what was posted, AudioSwitch doesn't need to be called to enable/disable an audiotrack. In fact, see what happens if AudioSwitch is enabled at app start up and disabled at app shutdown.
KamilSucharski
commented
1 year ago @afalls-twilio Here are the results:
I'm not 100% sure what was the cause here, but I have a suspicion. The ugly while (true) in old line 193 was probably too agressive on the Schedulers.io() thread, which i imagine Twilio also uses and maybe it created some friction? I had issues with some network calls not going out of the app until I changed them to use Schedulers.newThread() instead of Schedulers.io() - at the time I suspected Twilio of hogging the thread to itself, but now I suspect both it and that volume indicator that was here.
afalls-twilio
commented
1 year ago @KamilSucharski What are you attempting to do? It looks like the code you posted is creating an audio track just to calculate what the audio level in decibels. Would it be easier to make a custom audio device that calculated the decibels as it recorded the audio that is being sent to webrtc?
KamilSucharski
commented
1 year ago @afalls-twilio It shows the users their mic volume before they join the call.
Description
When connecting to the room without microphone enabled we get a crash. This only happened on our Google Pixel devices. The stacktrace is not very helpful, as the relevant part looks to be
but there are no further details.
Steps to Reproduce
Code
Expected Behavior
Successful connection, like on other brand devices.
Actual Behavior
Crash.
Reproduces how Often
Happens 100% of the time
Logs
Versions
All relevant version information for issue.
Video Android SDK
7.1.0, but tried the newest too and it did not help.
Android API
SDK 33 (Android 13)
Android Device
Google Pixel 4a, Google Pixel 6a