search

Genymobile / scrcpy

Display and control your Android device
Apache License 2.0
103.68k stars 10.09k forks source link

Forward audio #14

Closed benwa closed 1 year ago

benwa commented 6 years ago

Directing audio to the client would be pretty neat.

Currently you can accomplish the same thing by plugging a 3.5mm cable into the mic or line-in on the computer. Some manufacturers are dropping headphone ports though.

Suggestion by @rom1v: The best solution currently is to connect the device via bluetooth to the computer. It works for all apps, it does not suffer from additional limitations and works for basically all devices.

--> There is work in progress for Android >= 11. Please test #3757 :rocket:

MrLint commented 6 years ago

Adding this would really making streaming games would so much better. So far all the other existing solutions kinda suck.

AdoPi commented 6 years ago

Is it even possible without a custom rom?

rom1v commented 6 years ago

As suggested by Juan, one possibility to investigate is using AOA.

I already implemented something using HID over AOA in the past (I even have a publicly available sample), it should be quite similar.

rom1v commented 6 years ago

I just implemented a PoC (separated of scrcpy) to enable audio forwarding from the device to the computer: aoa-audio.

On Debian stable, it "works": the device sound is played on the computer.

This is quite manual though, and several issues must be solved before something similar could be implemented in scrcpy.

Any help is welcome :wink:

AdoPi commented 6 years ago

May it be possible to use AudioRecord api server side? Problem is that AudioRecord requires an Activity context, I don't know if we can spawn one by using app_process.

rom1v commented 6 years ago

AudioRecord may only record input sources (like microphone), not the device audio output.

AdoPi commented 6 years ago

I may be wrong but here is an example of what I thought:

https://github.com/tzutalin/ReadPCMData/blob/master/src/com/da/readpcmtest/MainActivity.java

Sorry I should have added this on my previous comment.

rom1v commented 6 years ago

Oh, you're right, I missed the REMOTE_SUBMIX audio source.

rom1v commented 6 years ago

Arf, just calling new AudioRecord(…) segfaults, probably because there is no Context (fault addr 0x0).

Note that AudioRecord itself does not require a Context explicitly.

AdoPi commented 6 years ago

Yes... :( That's why I was wondering if we could create one even if we are using app process.

Audio support is really tricky :/

rom1v commented 6 years ago

Even from an app, the source REMOTE_SUBMIX does not work:

AudioRecord: Could not get audio input for record source 8, sample rate 44100, format 0x1, channel mask 0xc, session 73, flags 0

(while it works with MIC for example).

rom1v commented 6 years ago

Hi,

I would like to share with you my work-in-progress branch: audio. This branch can be rebased/rewritten at any time.

I implemented it, and it works fine for me! :tada:

I would like early feedbacks so that any major issue can be fixed.

Requirements

How to

Install libusb:

sudo apt install libusb-1.0-0 libusb-1.0-0-dev

Checkout the audio branch:

git fetch
git checkout origin/audio

Use a debug build to report problems:

meson d
ninja -C d

Run with audio forwarding enabled:

./run d --forward-audio

The audio should be forwarded.

When closed, the audio forwarding may not be totally disabled. It may require to restart the current audio track (in VLC for example). If it is not sufficient, unplug the device. At worst (but it should not be necessary), disable/enable USB debugging.

How it works

Scrcpy connects to the device over USB to send AOA requests so that the computer becomes an "USB audio accessory" for the device.

This creates a new audio input source on the computer (which can be seen in PulseAudio for example).

Scrcpy records this input source and immediately plays it to the default output source.

Thank you for your feedbacks.

rom1v commented 6 years ago

In theory, it should now also work on Windows (in MSYS2/mingw64), with the libusb package:

pacman -S mingw-w64-x86_64-libusb

However, when it try to communicate to open the device over USB on Windows, libusb_open() returns -3, which is (according to libusb_strerror()):

Access denied (insufficient permissions)

I have no idea how to get the permissions granted. Any ideas?

eyal-lezmy commented 6 years ago

On macOS you can install libusb with the following command: brew install libusb

On a MBP 15, macOS 10.13.3 it is not working.

I start the app while a podcast is playing on the device. After the app starts I get some noise coming out from the computer's speakers, but nothing that I can recognise. At the same time the device's speakers stop emitting a sound. After about 10 seconds, the scrcpy app stops itself and the podcast sound comes back on the device's speaker.

This behaviour is 100% reproducible on my side.

Here are the command line output:


2018-03-24 23:01:14.865 scrcpy[5291:36915] DEBUG: Device serial is ENU7N15B03004574
2018-03-24 23:01:14.866 scrcpy[5291:36915] DEBUG: USB device with serial ENU7N15B03004574 found: 18d1:4ee7
2018-03-24 23:01:14.867 scrcpy[5291:36915] DEBUG: Device AOA version: 2
2018-03-24 23:01:14.867 scrcpy[5291:36915] INFO: Audio forwarding enabled
2018-03-24 23:01:14.883 scrcpy[5291:36915] INFO: Waiting 2s for USB reconfiguration...
2018-03-24 23:01:17.144 scrcpy[5291:36915] INFO: Selecting input audio source: Built-in Microphone
d/server/scrcpy-server.jar: 1 file pushed. 2.9 MB/s (23985 bytes in 0.008s)
2018-03-24 23:01:17.918 scrcpy[5291:36915] DEBUG: Starting decoder thread
2018-03-24 23:01:17.918 scrcpy[5291:36915] DEBUG: Starting controller thread
2018-03-24 23:01:18.845 scrcpy[5291:36915] INFO: OpenGL shaders: ENABLED
2018-03-24 23:01:18.845 scrcpy[5291:36915] INFO: Created renderer: opengl
2018-03-24 23:01:18.849 scrcpy[5291:36915] INFO: Initial texture: 1440x2560
2018-03-24 23:01:20.662 scrcpy[5291:36915] DEBUG: Unknown touch device id -841524992, cannot reset
2018-03-24 23:01:25.184 scrcpy[5291:37036] DEBUG: End of frames
2018-03-24 23:01:25.188 scrcpy[5291:36915] DEBUG: Video decoder stopped
2018-03-24 23:01:25.188 scrcpy[5291:36915] DEBUG: quit...
2018-03-24 23:01:25.200 scrcpy[5291:36915] DEBUG: Server terminated
2018-03-24 23:01:25.408 scrcpy[5291:36915] DEBUG: USB device with serial ENU7N15B03004574 found: 18d1:4ee7
2018-03-24 23:01:25.409 scrcpy[5291:36915] DEBUG: Device AOA version: 2
2018-03-24 23:01:25.409 scrcpy[5291:36915] INFO: Audio forwarding disabled```
rom1v commented 6 years ago

Thank you for the test :+1:

Selecting input audio source: Built-in Microphone

It did not select the right input source, either because none were created, either because for now it just selects the last one, assuming it's the most recent (so it should probably be the input source from the device).

I'll check that. Thank you.

eyal-lezmy commented 6 years ago

For your information I've done 2 tests one with a headset connected to the computer and another one without any headset plugged in.

Good luck

rom1v commented 6 years ago

@eyal-lezmy I investigated on Mac, and it appears that enabling "audio accessory" on the device does not create a new audio input source on the computer.

So unfortunately, I think that the feature will be Linux-only…

I updated the branch to reject the input source if its name does not contain the device model (so that it does not open the built-in microphone for example).

MrLint commented 6 years ago

Ok so heres a wholly stupid question. Would it work via the Linux environment in windows 10?

rom1v commented 6 years ago

Would it work via the Linux environment in windows 10?

You tell me :wink:

yxtj commented 5 years ago

What is the plan of merging this excellent feature into the main branch?

rom1v commented 5 years ago

Did you test it? Does it work correctly for you?

There are several problems:

So for now, I don't merge it.

Maybe I could, mentionning that it is very experimental...

yimuchen commented 5 years ago

Hi, thanks for the package! I've been enjoying it a lot I wanted to test the audio forwarding feature out on my machines, but I cannot work out why the program is failing to start with the forward audio enabled.

On my Archlinux machine, I have both libusb and libusb-compat installed besides the dependencies already listed in the AUR package for scrcpy. The testing scrcpy pacakge was compiled with the pre-built server.jar. When attempting to start scrcpy audio forwarding with my Nokia6.1 (Android 8.1, July 1st 2018 patch), the program exits with the following error message:

DEBUG: Device AOA version: 2
INFO: Audio accessory enabled
INFO: Waiting 2s for USB reconfiguration...
DEBUG: Audio input #0: Built-in Audio Analog Stereo
DEBUG: setpriority() failed
DEBUG: Device model is: Nokia 6.1
ERROR: Cannot find the Android accessory audio input source
DEBUG: Disabling audio accessory...
DEBUG: USB device with serial PL2GAMR832306185 found: 18d1:2d05
DEBUG: Device AOA version: 2

Without the --forward-audio flag, the program runs fine.

rom1v commented 5 years ago

@yimuchen Could you test manually with pactl and ffplay:

$ pactl list short sources
0   alsa_output.pci-0000_00_05.0.analog-stereo.monitor module-alsa-card.c  s16le 2ch 44100Hz   SUSPENDED
1   alsa_input.pci-0000_00_05.0.analog-stereo   module-alsa-card.c s16le 2ch 44100Hz   SUSPENDED

$ ffplay -vn -f pulse -i alsa_input.usb-LGE_Nexus_5_05f5e60a0ae518e5-01.analog-stereo
yimuchen commented 5 years ago

@rom1v I'm a bit new to writing this sort of code, Can you help me with how I should get the string for my device, the alsa_input.usb-LGE_Nexus_5_05f5e60a0ae518e5-01.analog-stereo for my phone?

The output of pactl is:

0       alsa_output.pci-0000_00_1b.0.analog-stereo.monitor      module-alsa-card.c      s16le 2ch 48000Hz       SUSPENDED
1       alsa_input.pci-0000_00_1b.0.analog-stereo       module-alsa-card.c      s16le 2ch 44100Hz       SUSPENDED

On a side note, as a test, I edited the get_matching_audio_device in app/src/audio.c, so that the program uses the found Audio input #0: Built-in Audio Analog Stereo as the audio device, it seems to be able to pass the microphone input of my phone to my PC (I can hear myself talking to the phone from hy PC's headphone), but not the internal audio.

rom1v commented 5 years ago

Can you help me with how I should get the string for my device, the alsa_input.usb-LGE_Nexus_5_05f5e60a0ae518e5-01.analog-stereo for my phone?

Sorry, I copied incomplete instructions I kept in some textfile.

Instead, follow the steps I wrote in the README of aoa-audio.

yimuchen commented 5 years ago

@rom1v Thanks for instructions! I think the issue that I have is that the udev rules for my phone isn't available yet. The vendor/product ID under lsusb is constantly in flux, and changes everytime I run the ./audio package. I guess I'll try again when I find and updated udev rule : /

ml1969 commented 5 years ago

@rom1v First, I want to thank you for this awesome application. It is exactly what I was looking for to mirror the screen of my Huawei Mediapad M5 on the PC and in very high quality. Neither google cast nor miracast can keep up with the image quality Better even as an airplay from the Ipad Air 2 to the PC. I use it with adb over wlan with this cmd command: scrcpy.exe -s 192.168.10.74:5555 -b 32M -m 1735

So far, I transmit the sound either via cable, which is awkward because the M5 has only a USB-C and no headphone jack, or I use bluetooth but that is not so clever.

Since I work exclusively with Windows and unfortunately with linux little on the hat, I would be glad to test the addressed by you "experimental audio inclusive version". With the knowledge that it does not work 100% yet and that there can be problems.

Thanks again for this great application. Regards Michael

rom1v commented 5 years ago

Since I work exclusively with Windows and unfortunately with linux little on the hat, I would be glad to test the addressed by you "experimental audio inclusive version". With the knowledge that it does not work 100% yet and that there can be problems.

On Windows, it does not work at all. And anyway, when it works, it's only over USB.

ml1969 commented 5 years ago

Hello rom1v,

thanks for reply, if I can help testing on windows I can use USB that will no problem. What I would say is, that your great app works over WLAN better as the "big known players" and thats great..... RESPECT to your work !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

would it work in a VM???? Will adb work in a VM when USB is passed through?? I did not have any experience with adb on a linux system, I have some experience on Windows I root Samsung S5 S6 S8, Nexus 6 and the tablet Nexus 7, also Huawei P10 but the mediapad its just new so I dont will do it 😂 all using adb and fastboot.

Michael

rom1v commented 5 years ago

What I would say is, that your great app works over WLAN better as the "big known players" and thats great..... RESPECT to your work !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Glad you like it :wink:

would it work in a VM???? Will adb work in a VM when USB is passed through??

I didn't try.

ml1969 commented 5 years ago

@rom1v

For your notice, I set up a VM using Virtual Box and a Ubuntu 18 image and it works. I can mirror the screen from my tablet with forwarded USB to the VM. The quality is not so good as on my real Win PC but I think this comes frome the USB.

Now I cloned the VM, so that I can try the audio stuff. 👍

Michael

ml1969 commented 5 years ago

Hello again,

First: I have managed that the quality in the VM is quiet the same as on my real Windows PC. I have to enable USB-3.0 mode in the VM.

Second: as you described I created a debug build. If I run .\run d it works like the normal build with scrcpy, but if I use the "-a" flag I got this error messages:

wot@wot-VirtualBox:~/scrcpy$ ./run d -s  NNF6R18525002604  -m 600 -a

DEBUG: Enabling audio accessory...
DEBUG: USB: cannot open device 1d6b:0003 (Access denied (insufficient permissions))
DEBUG: USB device with serial NNF6R18525002604 found: 12d1:107e
DEBUG: Device AOA version: 2
INFO: Audio accessory enabled
INFO: Waiting 2s for USB reconfiguration...
DEBUG: Audio input #0: Eingebautes Tongerät Analog Stereo
DEBUG: setpriority() failed
error: device 'NNF6R18525002604' not found
ERROR: "getprop model" returned with value 1
ERROR: Cannot read Android device model
ERROR: Cannot find the Android accessory audio input source
DEBUG: Disabling audio accessory...
DEBUG: USB: cannot open device 1d6b:0003 (Access denied (insufficient permissions))
DEBUG: USB: cannot open device 80ee:0021 (Access denied (insufficient permissions))
DEBUG: USB: cannot open device 1d6b:0002 (Access denied (insufficient permissions))
ERROR: Cannot find USB device having serial NNF6R18525002604
WARN: Cannot disable audio forwarding
wot@wot-VirtualBox:~/scrcpy$

Where can I look into???

MIchael

rom1v commented 5 years ago

@ml1969 you probably need to add the udev rules for Android device (just install the android-tools-adb package on Debian-based distribution).

likezjuisee commented 5 years ago

After I executed the command "./audio 18d1 4ee1 1", can I get the PCM data from the usb handle?

rom1v commented 5 years ago

@likezjuisee no, it just adds a new input audio source on your computer, that you can play or record.

likezjuisee commented 5 years ago

If I want to record the audio data from the android, could you give me some advice? The audio data's format is PCM?

nrfwmd commented 5 years ago

After adding udev rules, I got this:

DEBUG: No serial provided, request it to the device DEBUG: Device serial is ZY224P7LF8 DEBUG: Enabling audio accessory... DEBUG: USB device with serial ZY224P7LF8 found: 18d1:2d01 DEBUG: Device AOA version: 2 INFO: Audio accessory enabled INFO: Waiting 2s for USB reconfiguration... DEBUG: Audio input #0: Built-in Audio Analog Stereo DEBUG: setpriority() failed DEBUG: Device model is: moto x4 ERROR: Cannot find the Android accessory audio input source DEBUG: Disabling audio accessory... DEBUG: USB device with serial ZY224P7LF8 found: 18d1:2d05 DEBUG: Device AOA version: 2

rom1v commented 5 years ago

If I want to record the audio data from the android, could you give me some advice? The audio data's format is PCM?

Once AOA audio is enabled, you have a new input source on your computer (e.g. in pulseaudio). You can then play or record it as any input source. See

You can probably record it with audacity too.

rom1v commented 5 years ago

@nrfwmd Yes, it does not work with all devices or all computers. That's why I did not merge this branch.

nrfwmd commented 5 years ago

@rom1v will rooting my device help me with this?

rom1v commented 5 years ago

I don't think so.

nrfwmd commented 5 years ago

@rom1v I tried this on Xperia XA1 Ultra and it works(scrcpy with sound I mean). 👍

yxtj commented 5 years ago

What I really think important for this function is to have a compiled Windows version. Even it only works for few models. After I advertised this awesome tool to the Chinese community, especially the community of mobile game streamers, all them love it. And all of them asked how can they forward audio. Apparently, most common users are using Windows and this is not a unique phenomenon in some countries. It would be great if we can give them a touchable hope. In addition, spreading your company name to the millions of viewers of these streamers surely is a good and free adverting.

rom1v commented 5 years ago

(I have no idea how to make it work on Windows at all.)

IdrisDose commented 5 years ago

So question, if I run this app within an ubuntu VM on a windows host machine would it work O.o (all I want to do is watch this app that only streams to 'mobile devices')

rom1v commented 5 years ago

@IdrisDose See https://github.com/Genymobile/scrcpy/issues/14#issuecomment-410515037.

likezjuisee commented 5 years ago

If I want to record the audio data from the android, could you give me some advice? The audio data's format is PCM?

Once AOA audio is enabled, you have a new input source on your computer (e.g. in pulseaudio). You can then play or record it as any input source. See

You can probably record it with audacity too.

I will try, thanks.

rom1v commented 5 years ago

I rebased the audio on the current master. There were conflicts and things to rewrite, the previous audio branch was written above v1.1. The old one is still available in audio_old.

Nothing changes in the audio forwarding implementation, but it is now implemented above the last version of scrcpy.

ghost commented 5 years ago

I exclusively use Gnu/Linux (Manjaro) as I hate windows. Scrcpy is very good alternative to connect app on windows although it doesn't support audio forwarding like connect app so I use Bluetooth to forward my android audio Would it be possible to use it via wireless display(cast)?

amosbird commented 5 years ago

Can you help me with how I should get the string for my device, the alsa_input.usb-LGE_Nexus_5_05f5e60a0ae518e5-01.analog-stereo for my phone?

Sorry, I copied incomplete instructions I kept in some textfile.

Instead, follow the steps I wrote in the README of aoa-audio.

Hmm, I've tested with ffplay and it works as expected. However I still get this error

 ❯ ./run d --forward-audio
d/server/scrcpy-server.jar: 1 file pushed. 3.4 MB/s (69550 bytes in 0.019s)
DEBUG: No serial provided, request it to the device
DEBUG: Device serial is 851QFDSF2259C
DEBUG: Enabling audio accessory...
DEBUG: USB: cannot open device 8087:8001 (Access denied (insufficient permissions))
DEBUG: USB: cannot open device 1d6b:0002 (Access denied (insufficient permissions))
DEBUG: USB: cannot open device 17ef:1012 (Access denied (insufficient permissions))
DEBUG: USB: cannot open device 1d6b:0003 (Access denied (insufficient permissions))
DEBUG: USB: cannot open device 04ca:703c (Access denied (insufficient permissions))
DEBUG: USB: cannot open device 8087:0a2a (Access denied (insufficient permissions))
DEBUG: USB: cannot open device 138a:0017 (Access denied (insufficient permissions))
DEBUG: USB device with serial 851QFDSF2259C found: 18d1:2d01
DEBUG: Device AOA version: 2
INFO: Audio accessory enabled
INFO: Waiting 2s for USB reconfiguration...
DEBUG: Audio input #0: Built-in Audio Analog Stereo
DEBUG: Audio input #1: SDM636-MTP _SN:95743F4E Analog Stereo
DEBUG: Device model is: MEIZU E3
ERROR: Cannot find the Android accessory audio input source
DEBUG: Disabling audio accessory...
DEBUG: USB device with serial 851QFDSF2259C found: 18d1:2d05
DEBUG: Device AOA version: 2

pacman -S android-udev is installed.