Launch SUSI on F-Droid

[batbrain7]

Currently susi is only on play store, we have to launch it on the F-Droid .

[batbrain7]

In the recent progress with the Fdroid issue I successfully generated the metadata file, passed all the lint tests and started the local build of fdroidserver with susi.ai added in it. Three errors were faced by me :

[batbrain7]

The first error is with the link preview library for which alternative can be easily found, second error is with the binary file that is put in the app so that the HotWordDetection can take place.

1. The first library can be easily found and replaced.
2. For the second and third file we need to find an alternative to provide hotword detection and replace these binary files or another approach would be download these files after the app is installed.

So as we can see above that build was successful but these errors came in way. So once these errors are fixed we can make a merge request to F-Droid

[batbrain7]

To check for Fdroid compatibility and to build the app using fdroid follow these blogs :

1. https://docs.google.com/document/d/1IRc4EILOb-LHXkErQCybSv2CIKKkzqC_-Sese0_i5C8/edit

2. https://docs.google.com/document/d/1ItltxTNFFNgvMcN41yw176HwC7ena-pqtbwIIsgLP8g/edit

3. https://docs.google.com/document/d/1npSHMfZEY_ac3rOFKANPq_M7dizgA974wbNoaOOu6To/edit

[mariobehling]

For the second and third file we need to find an alternative to provide hotword detection and replace these binary files or another approach would be download these files after the app is installed.

There are not many good options for the download. So, downloading after installation sounds good to me.

[iamareebjamal]

@batbrain7 I'm not sure if binary files can be downloaded after installation. Also, this may violate the terms of service of FDroid

Snowboy is open source, so look for how this can be integrated in the build process

[batbrain7]

1. Issue for the link preview library since the library is open source, it should be accepted.
2. Link for the issue on fdroid-forum for the binary files

[codedsun]

@batbrain7 @iamareebjamal These two ".so" files are created using SWIG. I tried finding out the C or C++ file that could build the .so files using NDK, but that also won't help as these shared object files will be again causing the issues.

• [ ] What I would like to propose, is that downloading as suggested by @batbrain7 or we can go with pocketsphinx android that was used earlier (cmusphinx.github.io/wiki/tutorialandroid/)

[iamareebjamal]

How is downloading going to work? Please link to a documentation or article which provides more info into that. I may be wrong but I don't think it is possible to download and dynamically link the binary files like that

[batbrain7]

@iamareebjamal I did research it on the internet, I thought just like the obb package files we could download these files too, but I too don't think it is possible. https://android.jlelse.eu/app-bundles-and-dynamic-feature-modules-a-silver-bullet-for-bloated-apks-d0bcac151ae6 This article shows dynamic bundles but then again it requires play core library so it can't be used.

[iamareebjamal]

Even that doesn't allow you to download what you want. It downloads stuff automatically, the resources, layouts and libs it needs

[codedsun]

@iamareebjamal One suggestion I recieved on the post is

The README you linked contains instructions how to build the native libraries. Put them into the build recipe.

Please guide me for the same, the instructions given are here: https://github.com/Kitt-AI/snowboy/blob/master/examples/Android/README.md

[codedsun]

@iamareebjamal I researched that it uses swig and compile all the native methods in two .so files, but the problem here is how can I include in the build process? so that it creates that files during the build and not before hand.

[iamareebjamal]

Can you do it in F-Droid build process and not the SUSI build?

I think they are referring to build recipe(F-Droid) and not the build process of SUSI because I don't know how that will change the matter.

In any way we can change the process is to maintain a fork of snowboy which is not recommended. Clarify please

[codedsun]

@iamareebjamal @batbrain7

1 actionable task: 1 executed
INFO: Scanning source for common problems...
ERROR: Found usual suspect 'youtube.*android.*player.*api' at app/libs/YouTubeAndroidPlayerApi.jar
WARNING: Found JAR file at app/libs/YouTubeAndroidPlayerApi.jar
WARNING: Found possible binary at app/src/main/assets/snowboy/alexa_02092017.umdl
WARNING: Found possible binary at app/src/main/assets/snowboy/common.res
ERROR: Found shared library at app/src/main/jniLibs/arm64-v8a/libsnowboy-detect-android.so
ERROR: Found shared library at app/src/main/jniLibs/armeabi-v7a/libsnowboy-detect-android.so
INFO: Removing gradle-wrapper.jar at gradle/wrapper/gradle-wrapper.jar
ERROR: Could not build app ai.susi: Can't build due to 3 errors while scanning
INFO: Finished
INFO: 1 build failed

[batbrain7]

it seems even after adding playStoreimplementation the youtube library is scanned

[iamareebjamal]

No, you should be building the fdroid variant and not a general build

[batbrain7]

The most recent problem that was faced is how to build a particular flavor in the fdroid build process, so I posted the issue on the forum : https://forum.f-droid.org/t/how-to-build-a-particular-app-flavor-on-fdroid/3346

[batbrain7]

INFO: Removing gradle-wrapper.jar at gradle/wrapper/gradle-wrapper.jar
INFO: Creating source tarball...
ERROR: Could not build app ai.susi due to unknown error: Traceback (most recent call last):
  File "/usr/local/Cellar/fdroidserver/1.0.9/libexec/lib/python3.7/site-packages/fdroidserver/build.py", line 1120, in main
    options.onserver, options.refresh):
  File "/usr/local/Cellar/fdroidserver/1.0.9/libexec/lib/python3.7/site-packages/fdroidserver/build.py", line 896, in trybuild
    build_local(app, build, vcs, build_dir, output_dir, log_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
  File "/usr/local/Cellar/fdroidserver/1.0.9/libexec/lib/python3.7/site-packages/fdroidserver/build.py", line 562, in build_local
    tarball.add(build_dir, tarname, exclude=tarexc)
TypeError: add() got an unexpected keyword argument 'exclude'

INFO: Finished
INFO: 1 build failed

Now I am able to build the app on fdroid after removing the files but right after that I get this error

[iamareebjamal]

Always share your metadata reference here with the build log

[iamareebjamal]

You must be on Python 3.7, should be on 3.6

You seem to have installed fdroidserver via apt, if you had downloaded the git repo, you could have changed the code, easiest way would be to install fdroidserver in a virtualenv if possible

exclude was removed in Python 3.7 and replaced with filter

[batbrain7]

Categories:
  - Internet
License: Apache-2.0
SourceCode: https://github.com/fossasia/susi_android
IssueTracker: https://github.com/fossasia/susi_android/issues

Summary: Susi AI is an intelligent personal assistant
Description: Susi AI is an intelligent Open Source personal assistant. It is fully
    customizable and developed by a community of developers. It is capable of chat
    and voice interaction by using APIS to perform actions such as music playback,
    making to-do lists, setting alarms, streaming podcasts, playing audiobooks, and
    providing weather, traffic, and other real-time information. Additional functionalities
    can be added as console services using external APIs. Susi AI is able to answer
    questions and depending on the context will ask for additional information in
    order to perform the desired outcome. The core of the assistant is the Susi AI
    server that holds the "intelligence" and "personality" of Susi AI. The Android
    and web applications make use of the APIs to access information from a hosted
    server.

RepoType: git
Repo: https://github.com/fossasia/susi_android

Builds:
  - versionName: 1.0.10-Fdroid
    versionCode: 11
    commit: 1ad2fd0e858b1256617e652c6c8ce1b8372473e6
    subdir: app
    gradle:
      - fdroid
    rm:
      - app/src/main/jniLibs/arm64-v8a/libsnowboy-detect-android.so 
      - app/src/main/jniLibs/armeabi-v7a/libsnowboy-detect-android.so
      - app/libs/YouTubeAndroidPlayerApi.jar

AutoUpdateMode: None
UpdateCheckMode: Tags

this is the metadata reference file used for the build

[batbrain7]

@iamareebjamal can I move down a version in python?

[batbrain7]

dyld: Library not loaded: @executable_path/../.Python
  Referenced from: /usr/local/Cellar/fdroidserver/1.0.9/libexec/bin/python3.7
  Reason: image not found
Abort trap: 6

on moving down to version 3.6.5 I get this error , @iamareebjamal unable to understand this

[iamareebjamal]

Your fdroid is linked to python 3.7

You have to reinstall it

[batbrain7]

brew install fdroidserver
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
vips

==> Installing dependencies for fdroidserver: python
==> Installing fdroidserver dependency: python
==> Downloading https://homebrew.bintray.com/bottles/python-3.7.0.sierra.bottle.
######################################################################## 100.0%
==> Pouring python-3.7.0.sierra.bottle.1.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3
Target /usr/local/bin/2to3
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3'

To force the link and overwrite all conflicting files:
  brew link --overwrite python

To list all files that would be deleted:
  brew link --overwrite --dry-run python

Possible conflicting files are:
/usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/3.6/bin/2to3
/usr/local/bin/idle3 -> /Library/Frameworks/Python.framework/Versions/3.6/bin/idle3
/usr/local/bin/pydoc3 -> /Library/Frameworks/Python.framework/Versions/3.6/bin/pydoc3
/usr/local/bin/python3 -> /Library/Frameworks/Python.framework/Versions/3.6/bin/python3
/usr/local/bin/python3-config -> /Library/Frameworks/Python.framework/Versions/3.6/bin/python3-config
/usr/local/bin/pyvenv -> /Library/Frameworks/Python.framework/Versions/3.6/bin/pyvenv
==> /usr/local/Cellar/python/3.7.0/bin/python3 -s setup.py --no-user-cfg install
==> /usr/local/Cellar/python/3.7.0/bin/python3 -s setup.py --no-user-cfg install
==> /usr/local/Cellar/python/3.7.0/bin/python3 -s setup.py --no-user-cfg install
==> Caveats
Python has been installed as
  /usr/local/bin/python3

Unversioned symlinks `python`, `python-config`, `pip` etc. pointing to
`python3`, `python3-config`, `pip3` etc., respectively, have been installed into
  /usr/local/opt/python/libexec/bin

If you need Homebrew's Python 2.7 run 
  brew install python@2

Pip, setuptools, and wheel have been installed. To update them run
  pip3 install --upgrade pip setuptools wheel

You can install Python packages with
  pip3 install <package>
They will install into the site-package directory
  /usr/local/lib/python3.7/site-packages

See: https://docs.brew.sh/Homebrew-and-Python
==> Summary
🍺  /usr/local/Cellar/python/3.7.0: 4,787 files, 102MB

@iamareebjamal brew always installs fdroid linked with python 3.7

[iamareebjamal]

OK, then simply clone the git version, remove brew one. It's not gonna work

[iamareebjamal]

Also, remove the application ID suffix, it's not gonna work

Could not build app ai.susi: Wrong package ID - build ai.susi.fdroid but expected ai.susi

[batbrain7]

how to build the app after uninstalling the brew one as the "fdroid build" command is not available now

[iamareebjamal]

Did you install using git?

[batbrain7]

I have forked the fdroid repository into my own and then cloned the forked repository

[iamareebjamal]

Yeah, so there is fdroid binary in it, you must not have added it in your path

[iamareebjamal]

@batbrain7 It's been 2 days. I am able to build SUSI on my PC since we last spoke. What's the issue?

[batbrain7]

@iamareebjamal since the PR you made now if I install the fdroidserver with brew will it work ?

[iamareebjamal]

PR is not merged yet and even if it was, brew doesn't update libraries that fast

Why are you not cloning the repo, it's literally much much easier. Ask @nikit19 if you have doubts

[batbrain7]

@iamareebjamal I have cloned the respo but I don't understand where to add it in the path

[nikit19]

@batbrain7 I did the following to install the server git clone https://gitlab.com/fdroid/fdroidserver.git export PATH="$PATH:$PWD/fdroidserver"

credits: https://gitlab.com/fdroid/fdroiddata/blob/master/README.md#quickstart

[batbrain7]

this gives me error @nikit19 @iamareebjamal :

Traceback (most recent call last):
  File "/Users/mohitkumar/Documents/fdroidserver/fdroid", line 24, in <module>
    import fdroidserver.common
  File "/Users/mohitkumar/Documents/fdroidserver/fdroidserver/common.py", line 50, in <module>
    from pyasn1.codec.der import decoder, encoder
ModuleNotFoundError: No module named 'pyasn1'

[nikit19]

You have to install it

https://forum.f-droid.org/t/python-no-module-named-pyasn1-modules/2991/5

[batbrain7]

Found distributionUrl=https://services.gradle.org/distributions/gradle-4.7-all.zip via distributionUrl
/Users/mohitkumar/Documents/fdroidserver/gradlew-fdroid: line 59: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]
/Users/mohitkumar/Documents/fdroidserver/gradlew-fdroid: line 109: 1.4: syntax error: invalid arithmetic operator (error token is ".4")
No hash for gradle version distributionUrl=https://services.gradle.org/distributions/gradle-4.7-all.zip! Exiting...
ERROR: Could not build app ai.susi: Error cleaning ai.susi:1.0.10
==== detail begin ====
Found distributionUrl=https://services.gradle.org/distributions/gradle-4.7-all.zip via distributionUrl
/Users/mohitkumar/Documents/fdroidserver/gradlew-fdroid: line 59: declare: -A: invalid option
declare: usage: declare [-afFirtx] [-p] [name[=value] ...]
/Users/mohitkumar/Documents/fdroidserver/gradlew-fdroid: line 109: 1.4: syntax error: invalid arithmetic operator (error token is ".4")
No hash for gradle version distributionUrl=https://services.gradle.org/distributions/gradle-4.7-all.zip! Exiting...
==== detail end ====
INFO: Finished
INFO: 1 build failed

After building it, I get this error? @iamareebjamal @nikit19 any ideas?

[iamareebjamal]

This is because you are using mac AFAIK

Are you using bash on it? If not, install it

https://gist.github.com/samnang/1759336

[iamareebjamal]

This is why it is requested to contributors to stay at same environment, the declare command is different in bash and macOS

The script won't run properly

[batbrain7]

after installing bash too, I get the same error , are the steps different after installing bash than usual?

[iamareebjamal]

bash is a shell, did you run the instructions through bash after installing it? This is basic requirements setup

[batbrain7]

@iamareebjamal even after running all the steps through the bash shell, still I get the same error

[iamareebjamal]

What did you do?

[iamareebjamal]

@batbrain7 This is taking too long, what should we do?

[batbrain7]

@iamareebjamal I installed bash on my machine :
here is how it looks after opening it, now on reinstalling the fdroid server i.e cloning the fdroidserver again and following all the steps with this I get the above error again, is this not the bash to be used ?

[iamareebjamal]

Please setup TeamViewer and share your credentials in private Gitter channel ASAP so I can take a look into the issue, have the fdroiddata and server directory open with a terminal as well

We should tackle this within this day

[batbrain7]

Okay @iamareebjamal shared the credentials please check

[batbrain7]

@batbrain7 I did the following to install the server git clone https://gitlab.com/fdroid/fdroidserver.git export PATH="$PATH:$PWD/fdroidserver"

credits: https://gitlab.com/fdroid/fdroiddata/blob/master/README.md#quickstart

@nikit19 @iamareebjamal I have cloned repo from the original fdroid repo and not mine the forked, how to send MR now ?