Scripts and documentation for building the app are outdated

[JGreenlee]

Attempting to build e-mission-phone has a few additional steps now that we incorporated webpack.

We should revise the scripts so that we don't have to manually do these extra steps. We should also generally explain the build process more effectively in the README.

Cloning e-mission-phone and attempting to build it without doing anything extra yields the following output:

ERROR in ./www/index.js 46:0-42
Module not found: Error: Can't resolve './js/control/collect-settings.js' in '/Users/jgreenle/openpath/e-mission-phone_build/www'

ERROR in ./www/index.js 47:0-39
Module not found: Error: Can't resolve './js/control/sync-settings.js' in '/Users/jgreenle/openpath/e-mission-phone_build/www'

ERROR in ./www/js/i18nextInit.ts 152:0-51
Module not found: Error: Can't resolve '../../locales/lo/i18n/lo.json' in '/Users/jgreenle/openpath/e-mission-phone_build/www/js'
 @ ./www/js/app.js 20:0-47 21:17-35
 @ ./www/index.js 9:0-21

1st and 2nd Errors

Notice that collect-settings.js and sync-settings.js are not found when bundling. We can resolve this by having download_settings_controls.js be run by the 'setup serve' script. I will make this change and link to the PR here.

3rd Error

lo.json (the Lao language) was not present until recently. During setup, e-mission-phone, clones e-mission-translate into a subdirectory locales. By default, this instance of e-mission-translate points at the master branch.

The master branch doesn't have the Lao language (lo.json is not there).

This can be remedied by quickly going into the locales folder and running git checkout master_for_platform. (something like cd locales && git checkout master_for_platform && cd ..)

I could change download_translation.js to default to the master_for_platform branch. But master_for_platform should be getting merged to master soon anyway so I think we can just deal with it temporarily.

[JGreenlee]

• 1st and 2nd errors are fixed by https://github.com/e-mission/e-mission-phone/pull/1022
• 3rd error, we can temporarily workaround by running cd locales && git checkout master_for_platform && cd .. after the setup script

@niccolopaganini Can you verify that this works on your machine? Then, you can update the README to reflect the updated workflow:

• Instead of npx cordova ..., we now use the npm run ... scripts
• Add a note explaining the workaround if you use a branch of e-mission-phone that doesn't align with master on e-mission-translate.

[niccolopaganini]

To summarize my discussion with @JGreenlee, I cloned his branch but did not seem to work on my machine. Just to make sure, I made sure the local repo was up-to-date, but again, doesn't seem to build correctly.

Now that the branch is merged, @JGreenlee will try running it again in his computer to see if the build works and then I will go ahead and try.

[JGreenlee]

Here are the steps I followed:

• ensure I do not have a VPN connection active
• git clone https://github.com/e-mission/e-mission-phone.git
• cd e-mission-phone
• . setup/setup_android_native.sh
  • (this took a while)
  • there are warnings in the console about CocoaPods but I think this is ok because we're only building Android right now
  • [Process completed] - I needed to start a new terminal session
• npm run build-dev-android
  • Module not found: Error: Can't resolve './js/control/collect-settings.js' in '/Users/jgreenle/Documents/e-mission-phone/www'
  • Module not found: Error: Can't resolve './js/control/sync-settings.js' in '/Users/jgreenle/Documents/e-mission-phone/www'

At this point I realized https://github.com/e-mission/e-mission-phone/pull/1022 was only merged into label_dashboard_profile_sept_2023; I thought it had been merged into master.

So I start over:

• cd ..
• rm -rf e-mission-phone

And repeat the same steps, but using label_dashboard_profile_sept_2023:

• git clone https://github.com/e-mission/e-mission-phone.git
• cd e-mission-phone
• git checkout label_dashboard_profile_sept_2023
• . setup/setup_android_native.sh
  • (same as last time)

• npm run build-dev-android

  • FAILURE: Build failed with an exception.
    * Where:
    Script '/Users/jgreenle/Documents/e-mission-phone/platforms/android/CordovaLib/cordova.gradle' line: 75
    
    * What went wrong:
    A problem occurred evaluating script.
    > No usable Android build tools found. Highest 33.x installed version is 33.0.0; Recommended version is 33.0.2.
    
    * Try:
    > Run with --stacktrace option to get the stack trace.
    > Run with --info or --debug option to get more log output.
    > Run with --scan to get full insights.
    
    * Get more help at https://help.gradle.org
    
    Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.
    
    You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
    
    See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings
    
    BUILD FAILED in 2s
    Command failed with exit code 1: /Users/jgreenle/Documents/e-mission-phone/platforms/android/gradlew -b /Users/jgreenle/Documents/e-mission-phone/platforms/android/build.gradle cdvBuildDebug

[JGreenlee]

I tried re-running the prereq install:

• . setup/prereq_android_sdk_install.sh

But that failed too:

• Do you wish to continue? (Y/N)Y
  BEGIN: About to start android SDK download
  Warning: Failed to find package 'extras;intel;Hardware_Accelerated_Execution_Manager'
  END: Done with android SDK download, exiting scriptting updates...  

[shankari]

It looks like this may be a hardware issue aka your laptop might not support hardware acceleration. Would be interested in seeing the results from Nitish's personal laptop

We also then need to see why the CI is broken (doesn't appear to be the same issue) and fix it. The CI was working consistently until two weeks ago, but broke last week. https://github.com/e-mission/e-mission-phone/actions/workflows/android-automated-sdk-install.yml

EDIT: The CI is in fact broken due to the same issue check the "Install to a new SDK root" job in the broken run

This is a recent regression, so hopefully the SDK install upgrade should fix this.

We should also add a badge for the SDK install CI workflow to the README to increase the visibility of breakages.

[shankari]

can you check git branch and whether the changes to copy over the files are in package.json as opposed to only package.cordovabuild.json?

[niccolopaganini]

Reformatting everything for better clarity...

[niccolopaganini]

On my personal laptop & work laptop, running open jdk 11.0.19, the existing build process works. Personal MacBook has an ARM processor and the work computer has an Intel chip.

Commands that I ran for the tests:

$ git clone https://github.com/e-mission/e-mission-phone.git
$ cd e-mission-phone
$ git checkout label_dashboard_profile_sept_2023
$ . setup/setup_android_native.sh
$ npm run build-dev-android

Build was successful

Edit: MacBook Configurations

1. ARM

   • M1 Pro Chip
   • 16 GB RAM
   • MacOS Ventura 13.3

2. Intel

   • 2.9 Ghz Intel Core i9
   • 16 GB RAM
   • MacOS Monterey 12.6.7

[niccolopaganini]

On my same personal computer, I switched to the latest JDK through homebrew by

brew install openjdk@20

export JAVA_HOME=$(brew --prefix openjdk@20)

Then I ran the following commands:

$ git clone https://github.com/e-mission/e-mission-phone.git
$ cd e-mission-phone
$ git checkout label_dashboard_profile_sept_2023
$ . setup/setup_android_native.sh

This process went similarly... Then I ran

$ npm run build-dev-android

for which I got this output the first and only time:

(base) Nitishs-MacBook-Pro:e-mission-phone nvsr$ npm run build-dev-android
> edu.berkeley.eecs.emission@3.2.8 build-dev-android
> npx webpack --config webpack.dev.js && npx cordova build android

Need to install the following packages:
  webpack@5.88.2
Ok to proceed? (y) y
CLI for webpack must be installed.
  webpack-cli (https://github.com/webpack/webpack-cli)

We will use "npm" to install the CLI via "npm install -D webpack-cli".
Do you want to install 'webpack-cli' (yes/no): yes
Installing 'webpack-cli' (running 'npm install -D webpack-cli')...
npm ERR! code EACCES
npm ERR! syscall link
npm ERR! path /Users/nvsr/.npm/_cacache/tmp/d24d147e
npm ERR! dest /Users/nvsr/.npm/_cacache/content-v2/sha512/f0/5e/899f44ea4d30f33ccdf3cce40015f97354f5a5dbe99f91691538993b2b5c8390c74a69b0e7c12682226401cfe6f8d6920d05bfc1ec26b4698d53b3316784
npm ERR! errno EACCES
npm ERR! 
npm ERR! Your cache folder contains root-owned files, due to a bug in
npm ERR! previous versions of npm which has since been addressed.
npm ERR! 
npm ERR! To permanently fix this problem, please run:
npm ERR!   sudo chown -R 501:20 "/Users/nvsr/.npm"

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/nvsr/.npm/_logs/2023-09-08T18_32_08_096Z-debug-0.log

Errors:

ERROR in ./www/js/i18nextInit.ts 80:0-51
Module not found: Error: Can't resolve '../../locales/es/i18n/es.json' in '/Users/nvsr/e-mission-phone/www/js'
 @ ./www/js/app.js 20:0-47 21:17-35
 @ ./www/index.js 9:0-21

ERROR in ./www/js/i18nextInit.ts 81:0-51
Module not found: Error: Can't resolve '../../locales/fr/i18n/fr.json' in '/Users/nvsr/e-mission-phone/www/js'
 @ ./www/js/app.js 20:0-47 21:17-35
 @ ./www/index.js 9:0-21

ERROR in ./www/js/i18nextInit.ts 82:0-51
Module not found: Error: Can't resolve '../../locales/it/i18n/it.json' in '/Users/nvsr/e-mission-phone/www/js'
 @ ./www/js/app.js 20:0-47 21:17-35
 @ ./www/index.js 9:0-21

ERROR in ./www/js/i18nextInit.ts 83:0-51
Module not found: Error: Can't resolve '../../locales/lo/i18n/lo.json' in '/Users/nvsr/e-mission-phone/www/js'
 @ ./www/js/app.js 20:0-47 21:17-35
 @ ./www/index.js 9:0-21

ERROR in ./www/index.js 46:0-42
Module not found: Error: Can't resolve './js/control/collect-settings.js' in '/Users/nvsr/e-mission-phone/www'

ERROR in ./www/index.js 47:0-39
Module not found: Error: Can't resolve './js/control/sync-settings.js' in '/Users/nvsr/e-mission-phone/www'

6 errors have detailed information that is not shown.
Use 'stats.errorDetails: true' resp. '--stats-error-details' to show it.

webpack 5.88.2 compiled with 6 errors and 6 warnings in 6505 ms

which seems to be same/ similar as what @JGreenlee encountered

[niccolopaganini]

I tried re-running the prereq install:

• . setup/prereq_android_sdk_install.sh

But that failed too:

• Do you wish to continue? (Y/N)Y
  BEGIN: About to start android SDK download
  Warning: Failed to find package 'extras;intel;Hardware_Accelerated_Execution_Manager'
  END: Done with android SDK download, exiting scriptting updates...  

I am getting the same error for the above command.

However, the output for

npm run build-dev-android

is

FAILURE: Build failed with an exception.

* What went wrong:
Could not open settings generic class cache for settings file '/Users/nvsr/e-mission-phone/platforms/android/settings.gradle' (/Users/nvsr/.gradle/caches/7.6/scripts/dqjrgsw17c62wbpoagod92e1e).
> BUG! exception in phase 'semantic analysis' in source unit '_BuildScript_' Unsupported class file major version 64

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.6/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 311ms
Command failed with exit code 1: /Users/nvsr/.sdkman/candidates/gradle/7.6/bin/gradle -p /Users/nvsr/e-mission-phone/platforms/android wrapper -b /Users/nvsr/e-mission-phone/platforms/android/wrapper.gradle

Edit 1: This seems like a gradle issue. If I remember correctly, I encountered the same/ similar error during the time I was setting up my work computer.

[shankari]

@niccolopaganini are you going to update the errors that you encountered before last week to validate the findings from the CI (aka that this is a recent regression)?

[shankari]

wrt

ERROR in ./www/index.js 46:0-42
Module not found: Error: Can't resolve './js/control/collect-settings.js' in '/Users/nvsr/e-mission-phone/www'

@JGreenlee was able to fix them by switching to the correct branch (https://github.com/e-mission/e-mission-docs/issues/958#issuecomment-1708585344). Can you see why this didn't fix it for you? Are the changes to copy the files over in your package.json?

Also, these errors (around missing files) have nothing directly to do with the java version. Are you sure that there are not errors earlier related to the java version?

Also, are you using the same folder for build against the old and new java versions? If so, why do you need to re-clone? Or is there an rm -rf between them?

[shankari]

Given that @niccolopaganini is seeing this on his personal laptop (ARM) as well, this doesn't appear to be a hardware issue. It is also interesting that this is a regression in the past week https://github.com/e-mission/e-mission-phone/actions/workflows/android-automated-sdk-install.yml unless GitHub actions has changed its build hardware recently (https://github.com/actions/runner-images) looks like it has several changes recently, but seem to be primarily software related. There are some image changes in the past 4 days..

[shankari]

For the record, it is not unusual to include the HAXM in the list of modules. I see it in other repos as well, although potentially installed using different commands. We haven't changed any of this code https://github.com/search?q=Hardware_Accelerated_Execution_Manager+android&type=code

Passed Aug 26 Failed Sep 2

Note also that there are no changes to master between Aug 26th and Sep 2nd

[shankari]

I can confirm that the non-updated SDK install doesn't work on my laptop either although it worked when I was working on the upgrade https://github.com/e-mission/e-mission-phone/pull/1016#issuecomment-1685132683 So this is clearly a regression.

$ export ANDROID_SDK_ROOT=~/Library/Android/sdk

$ bash setup/prereq_android_sdk_install.sh
Downloading the android SDK. This will take a LONG time and will require you to agree to lots of licenses.
Do you wish to continue? (Y/N)Y
BEGIN: About to start android SDK download
Warning: Failed to find package 'extras;intel;Hardware_Accelerated_Execution_Manager'
END: Done with android SDK download, exiting scriptting updates...

Let's see whether upgrading the CLI will fix it.

[shankari]

Digging a bit deeper, HAXM has not been supported for OSX 10.x+ https://github.com/intel/haxm/issues/403#issuecomment-906052581

And it looks like the last successful run of the SDK install was on OSX 12.6.7 https://github.com/e-mission/e-mission-phone/actions/runs/5988843763/job/16244437744

  macOS 12.6.7 21G651

So here's what I think:

• HAXM support was removed a long time ago
• the SDK manager had the HAXM package still available (presumably as a NOP)
• sometime in the last two weeks, Google also removed the HAXM package from the SDK repository since it has been a NOP for a long time

If I am right, then the switch to most recent CLI tools will not help. We just have to remove the package 'extras;intel;Hardware_Accelerated_Execution_Manager' from the installation list.

[niccolopaganini]

Not sure if this addresses the situation but: https://developer.android.com/studio/run/emulator-acceleration

• sometime in the last two weeks, Google also removed the HAXM package from the SDK repository since it has been a NOP for a long time

<screenshot below for exact comment; article above from which I took the screenshot>

[shankari]

I actually don't think that quote is related. The context around it is

Choose a hypervisor on Windows Starting from emulator 33.x.x.x, HAXM is deprecated as Intel discontinues development of HAXM. The Android Emulator hypervisor driver (AEHD) replaces Intel HAXM on Intel Processors.

However, the same page has the following text about Mac

On MacOS, the Android Emulator uses the built-in Hypervisor.Framework, which requires the MacOS v10.10 (Yosemite) and higher. Note: It is not recommended to install and use Intel HAXM on the MacOS. The emulator doesn't support Intel HAXM on the MacOS any more.

I've been trying to find where the sdkmanager downloads its package repository from to see if we can check its change management but I think we're going to have to check the source code to figure it out. And I am not sure that is worth it given that HAXM is not supported on OSX since 10.10

I will upgrade to the most recent CLI tools anyway (since it is a good thing to stay up to date) but will also remove the haxm package from the install list

[shankari]

I can confirm that if I remove the HAXM emulator, even the current install prereq succeeds

$ diff --git a/setup/android_sdk_packages b/setup/android_sdk_packages
index 5d3f6737..a5d28811 100644
--- a/setup/android_sdk_packages
+++ b/setup/android_sdk_packages
@@ -3,7 +3,6 @@ build-tools;32.0.0
 build-tools;33.0.2
 emulator
 extras;google;google_play_services
-extras;intel;Hardware_Accelerated_Execution_Manager
 patcher;v4
 platform-tools
 platforms;android-30

$ bash setup/prereq_android_sdk_install.sh
Installing command line tools from commandlinetools-mac-9477386_latest.zip
ANDROID_SDK_ROOT=/Users/kshankar/Library/Android/sdk
Downloading the command line tools for mac
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  127M  100  127M    0     0  31.5M      0  0:00:04  0:00:04 --:--:-- 31.6M
Found downloaded file at /tmp/commandlinetools-mac-9477386_latest.zip
Installing the command line tools
Archive:  /tmp/commandlinetools-mac-9477386_latest.zip

---------------------------------------
Accept? (y/N): Y
[======================                 ] 57% Installing Google Play Intel x86 A
[========================================] 100% Unzipping... android-12/zipalign
[=======================================] 100% Unzipping... android-12/zipalign

END: Done with android SDK download, exiting script

[shankari]

After upgrading to the most recent version of OpenJDK (17)

$ java --version
openjdk 17.0.8.1 2023-08-24
OpenJDK Runtime Environment Temurin-17.0.8.1+1 (build 17.0.8.1+1)
OpenJDK 64-Bit Server VM Temurin-17.0.8.1+1 (build 17.0.8.1+1, mixed mode, sharing)

Android SDK install succeeds

$ bash setup/prereq_android_sdk_install.sh

---------------------------------------
Accept? (y/N): Y
[=======================================] 100% Unzipping... android-12/zipalign

END: Done with android SDK download, exiting script

And an android build in an existing checked-out repo succeeds

$ npm run build-dev-android
BUILD SUCCESSFUL in 33s
52 actionable tasks: 12 executed, 40 up-to-date
Built the following apk(s):
    /Users/kshankar/in-house/openpath-phone/platforms/android/app/build/outputs/apk/debug/app-debug.apk

Albeit with several deprecated APIs

w: /Users/kshankar/in-house/openpath-phone/platforms/android/app/src/main/java/com/adobe/phonegap/push/FCMService.kt: (169, 17): 'get(String!): Any?' is deprecated. Deprecated in Java
w: /Users/kshankar/in-house/openpath-phone/platforms/android/app/src/main/java/com/adobe/phonegap/push/FCMService.kt: (316, 20): 'get(String!): Any?' is deprecated. Deprecated in Java
w: /Users/kshankar/in-house/openpath-phone/platforms/android/app/src/main/java/com/adobe/phonegap/push/FCMService.kt: (627, 33): 'constructor Builder(Context)' is deprecated. Deprecated in Java
w: /Users/kshankar/in-house/openpath-phone/platforms/android/app/src/main/java/com/adobe/phonegap/push/FCMService.kt: (1190, 37): 'fromHtml(String!): Spanned!' is deprecated. Deprecated in Java
w: /Users/kshankar/in-house/openpath-phone/platforms/android/app/src/main/java/com/adobe/phonegap/push/PushPlugin.kt: (97, 25): 'get(String!): Any?' is deprecated. Deprecated in Java

[shankari]

For the record, it is very annoying that the Android SDK randomly removes packages from their download store without some form of release notes, but I guess that is what you get if you have a monopoly. And that's why we have CI, so that we can detect these regressions in the dependencies and resolve them in a timely fashion instead of waiting until people notice it.

We should restore the CI badge for the prereq install to the README.

• I know that it was there before - we should look at the history of the README to see when it was removed and restore it.
• We should also consider creating a github hook that automatically files an issue or puts a message into the teams channel if the build fails.

[shankari]

Fixed in https://github.com/e-mission/e-mission-phone/pull/1037