search

jamulussoftware / jamulus

Jamulus enables musicians to perform real-time jam sessions over the internet.
https://jamulus.io
Other
997 stars 222 forks source link

Android support #83

Closed corrados closed 3 years ago

corrados commented 4 years ago

This issue has been migrated from Sourceforge. Created: 2015-12-13 Creator: Volker Fischer

Add support for the Android operating system. The software can already be compiled for Android but the audio interface does not work right now. So the first step is to finish the audio interface implementation. As a second step, all GUI functionality must be checked that it is possible to use it with a touch screen only.

corrados commented 4 years ago

Some progress reported here: https://sourceforge.net/p/llcon/discussion/533517/thread/cfd165e0/#452b/6e3b

corrados commented 4 years ago

See here: https://www.facebook.com/groups/507047599870191/permalink/521878088387142/

j-santander commented 4 years ago

Is there any instructions on how to build this? or any specific branch/forked repository I could refer to? From the previous links there seems to be some implementation working to some extent. I might be able to give a hand, at least testing Thanks

corrados commented 4 years ago

The code on master is from Simon. But it does not work correctly. Recently, I have started a branch where I do some testing. But also, nothing working yet.

j-santander commented 4 years ago

Ok, thanks. I got it to compile and started testing.

There's an mistake on android/sound.cpp: on CSound::onAudioRead() when converting from incoming float buffer to int16_t:

                pSound->vecsTmpAudioSndCrdStereo[frmNum * oboeStream->getChannelCount() + channelNum] =
                    (short) floatData[frmNum * oboeStream->getChannelCount() + channelNum] * _MAXSHORT;

should be

                pSound->vecsTmpAudioSndCrdStereo[frmNum * oboeStream->getChannelCount() + channelNum] =
                    (int16_t)(floatData[frmNum * oboeStream->getChannelCount() + channelNum] * _MAXSHORT);

otherwise the precedence of the casting operator will make the input always 0.

Changing that if I write to a file the content pSound->vecsTmpAudioSndCrdStereo vector I got a correct signal.

To continue progressing I might need to understand better how the whole things work (I'm not at all familiar with neither your code or even the oboe library). A few doubts:

In this last case, I'd go probably for a temporary circular buffer where the onAudioRead() processing of input data will write the recording data left in the buffer and the call of onAudioRead() for the output data will read. It probably would need to be mutex protected (unless we can be sure that the oboe callbacks are always run on the same thread)

corrados commented 4 years ago

There's an mistake on android/sound.cpp: on CSound::onAudioRead() when converting from incoming float buffer to int16_t

I'll merge from a branch soon, where the int 16 type is replaced by float.

Is it assumed that the sound card is always stereo?

Yes, the input/output audio vector for the audio interfaces is assumed to be always stereo. If you have only a mono stream available, just write the same audio data in both stereo channels.

On my branch https://github.com/corrados/jamulus/tree/android_testing I tried to use the full duplex oboe example class. But it is not yet finished. There is function onBothStreamsReady which should have the same size for input and output (which is what Jamulus expects).

j-santander commented 4 years ago

I've got some working code at https://github.com/j-santander/jamulus/commit/a3558aa612e4793d7597620293296964abcc5bc1 (Unfortunately I was mistakenly working from the master code, not from the android_testing branch.. and only when I was ready to submit I realized of my mistake).

In my tests I can hear other clients and they can hear me. Not sure of the overall quality of sound.

Key modification was to setup the callback only on the input stream and do explicit write to the output stream after processing the incoming data.

I'll try to have a look at the code in the android_testing branch and incorporate it.

Let me know what you think or how should I proceed.

Cheers

corrados commented 4 years ago

My code on the android_testing branch is not working. I was just experimenting a bit. You do not need to use any of that code. If you have something working, then you could create a pull request. I can then test it on my Android phone. But be aware that some minutes ago I applied a big change to the code on master.

j-santander commented 4 years ago

Created PR #661.

Thanks

ann0see commented 4 years ago

How’s the quality? I assume it would be best to try it with a LAN adapter

j-santander commented 4 years ago

How’s the quality? I assume it would be best to try it with a LAN adapter

I might be able to test with more people today.

dzpex commented 4 years ago

i'm interested to test the android code but i do not have experience to compile qt5 project for andoid can you help me?

corrados commented 4 years ago

@j-santander I tested your code and it crashed with "AudioStreamRecord: maybeConvertDeviceData() conversion size 256 too large for buffer 128". I had the same error message on my Android testing branch. This is a known bug in the Oboe library. We have to use a newer version. Just to let you know: I'll now update the Oboe Git submodule to the newest Oboe code now to avoid this issue.

corrados commented 4 years ago

I just updated the Oboe library version but unfortunately, it still crashes, now with a different error message: " InputEventReceiver: Attempted to finish an input event but the input event receiver has already been disposed."

j-santander commented 4 years ago

Oh, I'll test it more on my side. I didn't experience neither of the errors. I'm on Android 11, not sure if it has anything to do.

j-santander commented 4 years ago

i'm interested to test the android code but i do not have experience to compile qt5 project for andoid can you help me?

Well, I didn't even have experience on Qt. I used the Qt installer (https://www.qt.io/download-qt-installer) and made sure the Android support was marked. Using the Qt Creator, I had to set up paths to SDKs and NDKs and select the target for android.

dzpex commented 4 years ago

i installed all the requirements for the QT and i'm able to set the Andorid Target. But the compiler return error.....oboe/Oboe.h not found any idea?

j-santander commented 4 years ago

i installed all the requirements for the QT and i'm able to set the Andorid Target. But the compiler return error.....oboe/Oboe.h not found any idea?

The oboe library is stored as a git submodule, I believe you need to do:

git submodule update --init
dzpex commented 4 years ago

i updated the submodule but the error is the same i try to copy oboe include folder into jamulus main path or android folder but nothin

j-santander commented 3 years ago

I just updated the Oboe library version but unfortunately, it still crashes, now with a different error message: " InputEventReceiver: Attempted to finish an input event but the input event receiver has already been disposed."

I've been trying with different devices, and although there are problems with the sound quality, I haven't experienced any crashes like the ones you report. What Android version are you in? what terminal?

What I've tried so far:

Regarding quality:

j-santander commented 3 years ago

i updated the submodule but the error is the same i try to copy oboe include folder into jamulus main path or android folder but nothin

Ok, let's recap. What I did was:

Regarding the actual source. oboe is a git submodule which is located at libs/oboe. Make sure it has been checkout properly.

Hope this helps

corrados commented 3 years ago

I've been trying with different devices, and although there are problems with the sound quality

Have you been able to jam with other on any of these devices yet? Could you measure the overall latency you got?

What Android version are you in? what terminal?

I have a special setup: LineagesOS 16.0 (Android 9) on a Samsung Galaxy S5 (SM-G901F)

The concern here is that we're doing all sort of stuff that we shouldn't be doing in the oboe's callback thread

We actually do this for all OSs. It is not the preferred way of doing it but it works fine.

j-santander commented 3 years ago

I've been trying with different devices, and although there are problems with the sound quality

Have you been able to jam with other on any of these devices yet? Could you measure the overall latency you got?

I did try briefly, but I'm in the middle of trying to deploy this to my choir and the focus of sessions is to get the people onboarded.

Regarding the latency. Right now, I'm connected to a local server in my LAN:

From the Windows Client (desktop - ethernet connected) the "Configuration" reports Overall Delay 39 ms. (Ping Time 1 ms).

From my Google Pixel 3a, over WiFi, Overall Delay varies quite a lot from 55 to 80 and 90 ms. I guess it is due to the great variation on Ping Time from 2 to 33 ms.

Not sure if that's the measurement you're after.

What Android version are you in? what terminal?

I have a special setup: LineagesOS 16.0 (Android 9) on a Samsung Galaxy S5 (SM-G901F)

ok, I'm not sure if I could find a way of emulating that.

The concern here is that we're doing all sort of stuff that we shouldn't be doing in the oboe's callback thread

We actually do this for all OSs. It is not the preferred way of doing it but it works fine.

Ok, I look into that. I might need to implement a circular buffer because in some of the models it seems that I cannot write the whole batch of frames of output frames in one go.

I'll keep testing and see how far I can go.

corrados commented 3 years ago

I did try briefly, but I'm in the middle of trying to deploy this to my choir and the focus of sessions is to get the people onboarded.

So you intention is to bring your choir to jamulus using their Android phones? Very interesting. Please post your results here since I guess a lot of other choir members would love to use their phones for Jamulus, too.

ok, I'm not sure if I could find a way of emulating that.

No problem if not. My device is very old and I do not expect a good latency coming out of it. So just focus on the newer phones, that's fine.

j-santander commented 3 years ago

I did try briefly, but I'm in the middle of trying to deploy this to my choir and the focus of sessions is to get the people onboarded.

So you intention is to bring your choir to jamulus using their Android phones? Very interesting. Please post your results here since I guess a lot of other choir members would love to use their phones for Jamulus, too.

Well, my objective is enabling the choir to be able to rehearse under Jamulus. However, I'm finding that for people outside of the computer world, even owning a decently modern computer could be an unsurpassable barrier (not to talk of installing or tinkering with the asio4all), and this cannot be the tool of a few, need to reach to the whole choir. The possibility of using an Android phone will do marvels to lower the entry barriers and I believe we can pay the cost in quality if that means that everyone can use it.

ok, I'm not sure if I could find a way of emulating that.

No problem if not. My device is very old and I do not expect a good latency coming out of it. So just focus on the newer phones, that's fine.

Submitted a new pull request. I've written the details in the PR, but in summary I changed the tackle moving back to two callbacks and placing a ring buffer to hold the output samples until they're retrieved from the output callback. I found that on devices using the OpenSL backend (or perhaps less powerful devices?) the output stream was not always ready to pull the samples as they were received.

This I tried with an old device (BQ Aquaris X5 Plus on Android 7.1.1) [using OpenSL] and two more moderns Google Pixel 3XL on Android 10 and Google Pixel 3a on Android 11.

I tested with some people outside home, although we didn't sing, I could hear people conversation correctly and they reported that they could hear me fine. All this over WiFi. I did find that that if you were not close to the WiFi AP quality will degrade very quickly.

From my previous test I found that the variation in the ping values on the WiFi connection will be quite large.

I'll try to deploy the application to some additional devices and perhaps do some additional testing and gathering stats to see if there are some additional settings that I can tweak.

dzpex commented 3 years ago

I reinstalled all necessary package and when i try to build and execute the code the compiler returned me this error: Android resource linking failed Output: /home/build-Jamulus-Android_Qt_5_12_9_Clang_armeabi_v7a-Debug/android-build/build/intermediates/merged_manifests/debug/processDebugManifest/merged/AndroidManifest.xml:92: error: resource array/load_local_libs (aka com.github.corrados.jamulus:array/load_local_libs) not found. error: failed processing manifest.

Any idea?? Thanks...

danryu commented 3 years ago

This all sounds very encouraging for the prospect of mobile builds - and being able to support choirs and similar scenarios as @j-santander described!

I would love to give an Android build a test - is master the best branch to attempt a build from currently?

Also PS - should perhaps open a new issue for this: is there any particular reason that an iOS build has not been attempted yet? I've been scoping out the C++ iOS sound libs available which I believe would be able to integrate with a Qt-iOS build.

ann0see commented 3 years ago

I think there was an attempt for iOS, but the repo disappeared. See iOS version thread on sourceforge.

j-santander commented 3 years ago

This all sounds very encouraging for the prospect of mobile builds - and being able to support choirs and similar scenarios as @j-santander described!

I would love to give an Android build a test - is master the best branch to attempt a build from currently?

Sorry folks, I believe my disappearance requires a bit of explanation. As I described above, what I did over the existing code for Android that Volker and Simon had in place was just a bit of bug fixing and small adjustments. However, as I was thinking about further involvement I realized that this was uncomfortably close to my day job (not really a direct overlap, but there might be technologies and techniques in common), so before involving myself any further, I decided to clear it up with my management. They sent a request to legal department and we're still waiting for an answer.

Regarding building. All my modifications were accepted to master, but I haven't tried to build it recently. Not sure if recent changes might have broken the build.

danryu commented 3 years ago

Thanks @j-santander for the update. FWIW - I don't know your exact situation, but from my experience there should be zero conflict with working on open-source projects, unless you are using company time to do so. Most progressive companies welcome their employees working on diverse projects in their own time, as it inevitably leads to an enrichment of experience and ability within the workforce.

ann0see commented 3 years ago

How does the UI look like on Android? I compiled Jamulus for iOS and it doesn't show anything for me. I assume, we might need QML for both operating systems?

corrados commented 3 years ago

The UI on Android looks the same as on the PC. No need for QML.

ann0see commented 3 years ago

Ok. Also, there was no need to change the code, I assume? On iOS is that I don't see anything but a black screen, so it's probably an iOS only problem.

I hope, my problem can be fixed in some way without having to rewrite the UI for iOS

corrados commented 3 years ago

Qt should give you the same GUI on all platforms.

nefarius2001 commented 3 years ago

Hi,

I got the code compiled and the GUI running, but not Audio transmission. Will try a different device as next step...

nefarius2001 commented 3 years ago

I have the audio part well working on my tablet (X10605). Some UI issues are still open (mainly using the "back" button)

I am thinking about a source for "nightly" APKs, for those eager to have at least anything. @corrados Any plans or existing infrastructure to provide "official" builds, except the released installers?

ann0see commented 3 years ago

I am thinking about a source for "nightly" APKs, for those eager to have at least anything.

Any plans or existing infrastructure to provide "official" builds, except the released installers?

GH actions should be able to build it automatically. Once #828 is merged and tested, it should be quite easy to add an action which automatically builds apk files.

corrados commented 3 years ago

That would be creat. Since the Action searches for a "r*" tag, I could easily create beta versions by simply tagging the Git repo with something like r3.6.2beta1.

ann0see commented 3 years ago

Yes, that’s possible but will also move the latest tag.

I tried to setup an Android build environment on a clean Debian 10 VM but didn’t succeed unfortunately. If anybody succeeds to create a shell script which installs the dependencies and starts the build, it‘s easy to upload the apk. You will then just need to move the .apk file to /deploy in the Shell script and edit the GitHub Action.

nefarius2001 commented 3 years ago

I tried to setup an Android build environment on a clean Debian 10 VM but didn’t succeed unfortunately.

I tried with an Ubuntu, but also no success yet. Weird, setting it up on windows with QtDesigner was easy, now Ubuntu is quite hard...

nefarius2001 commented 3 years ago

Until an official build is available, the APK of my pull request can be found here: http://116.203.80.188:8000/jamulus-024ec6a8402048c13b03a31a13504feed1de3635.apk

ann0see commented 3 years ago

I've now created a project board on this topic, since I think it's a larger one. It would be great if we could close this issue and document needed changes in separate issues and in https://github.com/jamulussoftware/jamulus/projects/6

ann0see commented 3 years ago

Just a quick reminder here: Thanks to nefarius we have working .apk android builds here: https://github.com/jamulussoftware/jamulus/releases/tag/r3_7_0beta3

Once 3.7.0 is out, GitHub actions will automatically build apk files for every new release.

hoffie commented 3 years ago

Once Jamulus 3.7.0 is out, there will be official apk files for every new release (hopefully).

I'm wondering if we want to bless them as "official" already or rather mark them as experimental for now?

I gave them a quick try and while the UI worked, I did not manage to get usable sound out of it on a Moto G4 Play (distortion in short frequency, not the typical Wifi-related unreliability). If this is supposed to work, I should probably open a bug report about it. Until now I assumed that this was some kind of proof of concept...? :)

ann0see commented 3 years ago

Yes. You're right. I just edited my post. Some people have reported "usable" results: https://github.com/jamulussoftware/jamulus/discussions/1142

gilgongo commented 3 years ago

Moving this to discussion in the interests of keeping issues to actionable work items.