Dockerized build process

[divan]

User Story

As a developer, I want to regularly build an app for iOS and Android locally and test its functionality. However, I've never been able to build and run iOS version and had hours long (sometimes days) struggle with various build process issues for Android. In BKK, 4 core developers were sitting at my laptop for a few hours, trying to build iOS version for the device, and the best result was is a build for iOS simulator. I can't reproduce it now, though.

At the same time, I know that many ClojureScript developers here were successful with building and running app locally. I can see two reasons for that: 1) non-deterministic build process 2) additional knowledge required, which is not codified into instructions or build scripts

Obviously, the complexity of the build process for status-react is huge, but this is an essential complexity and we can't avoid it. We can, however, hide this complexity behind scripts and automatization.

For me, one of the biggest issues with building status-react is the fact, that build process heavily depends on the system-wide state – particular versions of the tools and frameworks (npm, gradle, re-natal, lein, SDKs etc) and the presence or absence of specific files and directories (npm caches, system-wide config files, etc). And knowledge about this is not codified in scripts or codified very sparsely.

For devs who want to occasionally sync the repo, and build the app, there is simply no way to know what is wrong in this complicated build pipeline and which versions of what they have to have.

I believe the build process should be deterministic - any developer should be able to clone the repo, run build process and be sure that the build process will use the same versions and frameworks, without learning all the build stack herself. If core developer or a hacker at the hackathon or any contributor wants simply build the app, we should not require her/him any additional knowledge.

For me, this issue was a major blocker for many ideas for more than half a year and the reason why we couldn't ask hackers to hack on our app during EthDenver hackathon.

Solution

Build process is just a function of a) input files and b) development tools/scripts/libs, which produces an output - the app. We need this function to be self-sufficient and have all inputs determined and codified, so the result is always the same.

I think the best solution for this sort of problems is to Dockerize the build tools, so the whole build process narrows down to passing current repository tree into docker container and executing it. This approach is common and hides the building complexity from developers.

The only program that developer need to install is a Docker. All correct versions of build tools are inside docker container, so there is no chance to have "wrong" version. Also, it's always executed with a clean slate, so no cache or system-wide global configuration files overlap problems.

---

The acceptance criteria for this issue will be a Dockerfile (or two, for iOS and Android, if it makes sense), that installs all needed tools and dependencies and build a docker container for local use. Further, developer can simply run docker <container-name> -v ... build ios/android and get the build artifact,

This approach might be difficult for debug/dev builds, where a few backgrounds need to be run, ports forwarder and REPL instantiated(?), but we can start with making non-debug builds reproducible.

[chadyj]

Awesome project! I'm happy to help test. Another benefit is making it trivial to integrate with CI systems like CircleCI.

[divan]

Ongoing work is in https://github.com/status-im/status-react/tree/feature/docker-build branch.

GitHub

status-im/status-react

status-react - a free (libre) open source, mobile OS for Ethereum

[divan]

So as described in the README in the docker-build/ folder in abovementioned branch, the process to use this is following:

1. Build the docker image with (presumably) all dependencies downloaded and included into the docker image:

   docker build -f docker-build/Dockerfile -t status-react-native-v2:latest .

   Note: The build part may require at some point manually enter "y" to accept Android NDK licenses (despite echo y in the pipe), but that should be solved.

1.5. Run the container from this image and try to build the app (it's failing for now):

docker run -t -i -v $(pwd):/workspace -w /workspace --rm --privileged --name build status-react-native-v2 bash

---

Commands I try to execute next (based on multiple sources of information about how to build status-react):

# copy node_modules/ and target/ from /build to /workspace
rm -rf /workspace/node_modules /workspace/target
cp -a /build/node_modules /workspace
cp -a /build/target /workspace

then

# this should be in the docker build step eventiually
cd /workspace
ln -sf './node_modules/re-natal/index.js' './re-natal'
# some magical incantation nobody understands if they needed or not
./re-natal deps && ./re-natal use-figwheel && lein re-frisk use-re-natal && ./re-natal enable-source-maps
mvn -f modules/react-native-status/ios/RCTStatus dependency:unpack

# this should be available in a makefile or entrypoint script or whatever
lein do clean, with-profile prod cljsbuild once android
./android/gradlew --daemon --parallel -q -b android/build.gradle assembleRelease

---

Currently it fails with two possible errors (it depends on how many times you try the universal solution of removing node_modules/ folder and reinstalling npm install again):

Starting process 'command 'node''. Working directory: /workspace Command: node --max-old-space-size=4096 node_modules/react-native/local-cli/cli.js bundle --platform android --dev false --reset-cache --entry-file index.android.js --bundle-output /workspace/android/app/build/intermediates/assets/release/index.android.bundle --assets-dest /workspace/android/app/build/intermediates/res/merged/release
Successfully started process 'command 'node''
Scanning folders for symlinks in /workspace/node_modules (500ms)
Scanning folders for symlinks in /workspace/node_modules (692ms)
Loading dependency graph, done.
warning: the transform cache was reset.

Unable to resolve module `crypto` from `/workspace/node_modules/eccjs/dist/0.3/ecc.js`: Module does not exist in the module map

This might be related to https://github.com/facebook/react-native/issues/4968
To resolve try the following:
  1. Clear watchman watches: `watchman watch-del-all`.
  2. Delete the `node_modules` folder: `rm -rf node_modules && npm install`.
  3. Reset Metro Bundler cache: `rm -rf $TMPDIR/react-*` or `npm start -- --reset-cache`.  4. Remove haste cache: `rm -rf $TMPDIR/haste-map-react-native-packager-*`.

:app:bundleReleaseJsAndAssets FAILED
:app:bundleReleaseJsAndAssets (Thread[Task worker for ':' Thread 2,5,main]) completed. Took 5 mins 57.13 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:bundleReleaseJsAndAssets'.
> Process 'command 'node'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --debug option to get more log output.

* Get more help at https://help.gradle.org

BUILD FAILED in 6m 19s
585 actionable tasks: 5 executed, 580 up-to-date

or

/workspace/android/app/src/main/java/com/statusim/MainApplication.java:19: error: cannot find symbol
      return BuildConfig.DEBUG;
                  ^
  symbol: variable BuildConfig

If you have any ideas what might be the reason, please feel free to suggest, and, of course, try to build the image yourself.

[status-github-bot[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[status-github-bot[bot]]

This issue has been automatically closed. Please re-open if this issue is important to you.

[mandrigin]

@jakubgs please close this one if it was fixed (I think it was, but want to double-check)

[jakubgs]

Hmmm, I guess it was for Linux, Windows, and Android. Though there is another step to do which is include use of docker containers in local development, but maybe we can open a separate issue for that.