Better support for monorepos

[yoshikischmitz]

Describe the solution you'd like I would like to be able to control how AppCenter clones my git repo. My app lives inside of a monorepo which is pretty big. It would be nice if AppCenter had a way to configure the git cloning process to use something like a partial fetch.

Describe alternatives you've considered Alternatives would be changing how our repo works. But I think there are enough monorepo users out there for AppCenter to consider this.

[yoshikischmitz]

Hi, would love to hear from the appcenter team if this is something they're interested in adding!

[patniko]

Definitely interested. Just a matter of getting to it. We do monitor this repo heavily to see where people are heavily requesting certain features.

[alexanderkrum]

Yeah +1

[lhrolim]

+1

[0xycvv]

I'm trying to use app center to build react-native. How can I set the entry file in monorepo build?

[nilofer]

@ericyip you cannot specify the file path for builds today. Noted as something that would be important as part of this feature request. Thanks!

[iamchathu]

We are using monorepo with Lerna and have structured the project with main App with sub packages with shared code and native dependancies. We need a way to specify with package should AppCenter build as main project and use Yarn workspaces or Lerna to install dependancies.

[yoshikischmitz]

As part of this, another issue I noticed is in the package dropdown, the current selected package shows up as just package.json. When you have many package.json files, this is not super helpful. To disambiguate between packages, it would be much more helpful to either show the path of the package file, or perhaps even better(as it shows you in the dropdown selection), or the name of the package(as specified in the name field).

[putuyoga]

as monorepo become mainstream, this feature would benefit a lot of people out there.

[dluc]

Hopefully the team will also fix the way a repository is scanned to detect React Native apps. As described in https://github.com/microsoft/appcenter/issues/928 if a RN app is in a subfolder, AppCenter doesn't detect it. Apparently also the "4 layers" search for package.json is not working as expected (see comments).

[brandonpearcy]

+1 for AppCenter to support monorepo projects. We were using AppCenter to deploy our react-native project, but have had to move away from it recently after switching over to a monorepo w/ yarn workspaces. We miss you AppCenter! 😢

Our monorepo project structure is as follows:

/projectRoot
    /node_modules (yarn installs most modules here)
    /packages
        /app
            /node_modules (appcenter build incorrectly assume RN modules will be here)
        /shared
        /web

[zeevl]

I was able to get this to work with our RN monorepo (using yarn workspaces) by doing the following:

• configure build to use package.json in RN sub project
• configure that RN package.json to nohoist it's packages:

  "workspaces": {
  "nohoist": [
    "**"
  ]
  }

• create a script in that sub project called appcenter-post-clone.sh that does a yarn install in the root project directory
• create an empty yarn.lock in the RN sub project to trick appcenter into using yarn

We also had to install a more recent version of node in our post-clone script so rn & others could install correctly. This is what appcenter-post-clone.sh looks like for us:

#!/usr/bin/env bash

set -ex

brew uninstall node@6
NODE_VERSION="12.13.0"
curl "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}.pkg" > "$HOME/Downloads/node-installer.pkg"
sudo installer -store -pkg "$HOME/Downloads/node-installer.pkg" -target "/"

# run yarn twice, ignoring errors from the first one
# https://github.com/yarnpkg/yarn/issues/6988
yarn --cwd ../../ install || true
yarn --cwd ../../ install 

[brandonpearcy]

Thank you! That got me closer, but I'm still seeing errors. I wonder what's different between our configurations.

I added nohoist to packages/app/package.json Moved react-native.config.js from project root to packages/app Confirmed that react-native config output all looks reasonable. Confirmed that react-native run-ios and run-android both build correctly locally. Added your appcenter-post-clone.sh script to packages/app, and verified that it is running successfully in AppCenter build

For my iOS build, I have AppCenter configured as follows:

• use packages/app/package.json
• XCode version 11.2.1
• Node version: 10.x
• Use legacy build system: off

Here is where the build fails. The build script is looking for index.js in the projects root directory, instead of packages/app/index.js

##[section]Starting: Generate source map
==============================================================================
Task         : Command line
Description  : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows
Version      : 2.151.2
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line
==============================================================================
Generating script.
========================== Starting Command Output ===========================
[command]/bin/bash --noprofile --norc /Users/runner/runners/2.160.1/work/_temp/c6837c87-5b14-443b-9a75-e1e5c8ac0a5b.sh
Found index.js for ReactNative index.
warn The following packages use deprecated "rnpm" config that will stop working from next release:
  - react-native-code-push: https://microsoft.github.io/code-push
  - rn-fetch-blob: https://npmjs.com/package/rn-fetch-blob
Please notify their maintainers about it. You can find more details at https://github.com/react-native-community/cli/blob/master/docs/configuration.md#migration-guide.
warning: the transform cache was reset.
error The resource `/Users/runner/runners/2.160.1/work/1/s/index.js` was not found. Run CLI with --verbose flag for more details.
Error: The resource `/Users/runner/runners/2.160.1/work/1/s/index.js` was not found.
    at fs.realpath.err (/Users/runner/runners/2.160.1/work/1/s/packages/app/node_modules/metro/src/IncrementalBundler.js:157:26)
    at gotStat (fs.js:1600:21)
    at FSReqWrap.oncomplete (fs.js:153:21)
##[error]Bash exited with code '1'.

[agusvazquez]

Hello!

I'm having the same issue as @brandonpearcy Did you guys find a workarround for this?

Thanks!

[vitalyiegorov]

Hello, also huge + for implementing monorepo support.

[agusvazquez]

Hello guys! Got it working by generating the main.js file manually and then pushing the file.

yarn workspace app-name build:ios|android to generate the bundle file

Remember to add this to the scripts part of package.json

"build:android": "react-native bundle --entry-file=./packages/mobile/index.js --bundle-output=./android/CodePush/index.android.bundle --assets-dest ./android/CodePush/ --dev=false --platform=android", "build:ios": "react-native bundle --entry-file=./packages/mobile/index.js --bundle-output=./ios/CodePush/main.jsbundle --assets-dest ./ios/CodePush/ --dev=false --platform=ios",

When the folder is created, just do a release

appcenter codepush release -c ./packages/mobile/ios/CodePush -t **VERSION** -a **PROJECT** -d **ENVIRONMENT**

or

appcenter codepush release -c ./packages/mobile/android/CodePush -t **VERSION** -a **PROJECT** -d **ENVIRONMENT**

[vitalyiegorov]

Using RN 0.61.5 tried @zeevl solution, it does not work with following error:

* Where:
Script '/Users/runner/runners/2.163.1/work/1/s/node_modules/@react-native-community/cli-platform-android/native_modules.gradle' line: 206

* What went wrong:
A problem occurred evaluating script.
> React Native CLI failed to determine Android project configuration. This is likely due to misconfiguration. Config output:
  [root:/Users/runner/runners/2.163.1/work/1/s, reactNativePath:/Users/runner/runners/2.163.1/work/1/s/node_modules/react-native, dependencies:[@react-native-community/masked-view:...

I guess the problem is that reactNativePath is resolved to the root node_modules folder

@zeevl Maybe you have a clue?

[agusvazquez]

Hey @vitalyiegorov try generating the bundle js manually and uploading it to code push like I explained on my previus post.

That should work and there is no need for black magic on the project.

[vitalyiegorov]

@agusvazquez Thanks for the solution! I will try to integrate it to speed up our builds.

Your solution is viable but it does not fully fit our development process, as our project is in the MVP stage we continue adding new native modules, thus we need to rebuild our APK and CodePush deployment that you have suggested will only partly solve our needs as we would love to dedicate build process for iOS/Android to Appcenter.

Anyway, we need someone help from @microsoft @patniko team to investigate and improve monorepo support as it is getting more and more popular, maybe some things need to be also fixed inside react-native-cli for running RNCLI inside the subfolders, latest RN 0.61.5 release states that monorepo support has been improved which means that the RN team dedicating its resources for it.

[agusvazquez]

@vitalyiegorov if your project has a lot of native modules, then Code Push makes no sense because you will always need to update the native code, therefore you will need to send another APK / IPA for approval to Google / iTunes.

Unless you want to do quick JS only fixes.

The solution I gave is for just doing a code push changing only javascript code. If you need to change native code of course you will need to recompile everything.

[vitalyiegorov]

@agusvazquez Again thank you very much for your advice, but this issue, in general, is not about publishing a new CodePush version, it is related to Appcenter monorepo support which is currently not working, we should be able to build monorepo projects.

[pontusab]

I guess we need to have a config for --entryFile for react native cli build command on AppCenter. Choosing package.json in the subdirectory of the monorepo is not enough.

[GoMino]

to fix @brandonpearcy issue and trick app-center when using a monorepo, a quick hack is it to create a index.js file at the root of the monorepo and require the index of your react-native project:

require("./packages/mobile");

[Maushundb]

@GoMino I tried that and it seemed to fix the index not found error folks have been describing, but now its telling me error Unable to resolve module lib/graphQl/ApolloClient from packages/mobile/src/App.js: lib/graphQl/ApolloClient could not be found within the project., which is the first non-node_modules import I have in my app.

I have a hunch it's trying to resolve imports relative to the root, rather than the package/mobile directory because of require("./packages/mobile");. Any other way to get appcenter to find the right index?

Setting the build setting to use packages/mobile/package.json and changes build:ios to --entry-file=./packages/mobile/index.js didn't seem to fix it.

[zeevl]

Hey all, just wanted to pipe in -- we're successfully building both iOS and android apps from a monorepo on appcenter, using the normal appcenter build tools. Keys for me were:

1. this comment,

2. Pinning the rn cli to 3.0.0 by adding the following to the root package.json:

  "resolutions": {
    "@react-native-community/cli-platform-android": "3.0.0"
  }

due to https://github.com/microsoft/appcenter/issues/1518 and https://github.com/react-native-community/cli/pull/852

I'm not sure I'll be able to provide any support other than that, as so many of the issues in this thread are not issues I encountered. But, I wanted to post this to let you know, it is possible.

[Maushundb]

@zeevl Did you change your build settings to use the root package.json or the sub-folder package.json?

[zeevl]

@Maushundb Build settings are configured to use the mobile app's subfolder's package.json (packages/mobile/package.json).

We also nohoist all of the RN project's modules:

  // in packages/mobile/package.json

  "workspaces": {
    "nohoist": [
      "**"
    ]
  }

[Maushundb]

Yeah I have the "nohoist": [ "**" ] in root/package.json, which seems to do the same thing as putting it in all your packages.

Last q @zeevl - where are your appcenter-pre-build.sh etc located? Root or package? Even if I set the build settings to use the subfolder, appcenter can't seem to find the scripts unless they're in the root.

[zeevl]

I only have appcenter-post-clone.sh, and that's in packages/mobile. It's definitely running with each build.

[Maushundb]

Ok got it working - for posterity, here's the steps we had to do -

1. Put nohoist: [**] in root/package.json.
2. Change the build settings on your branch to use the sub-folder package.json, then hit save and build. this was the key thing I missed:

When App Center detects build scripts for the first time, or whenever you make changes to the location of scripts or, for iOS projects, where you store CocoaPods, you must click the Save or Save & Build button in the build configuration to apply the changes. When you do this, App Center performs an analysis to index your repository tree and updates the build definition.

3. Put a dummy yarn.lock in the package dir

That's why it wasn't finding my appcenter-pre-whatever scripts when I moved them out of the root. Everything else (the require in index, changing the build:ios, the whole appcenter-post-clonescript etc) was unnecessary in our case.

[GoMino]

The problem for me is: since we migrated our react-native repo to a monorepo, we are not able to save or save and build anymore because we cannot select the buildVariant for Android or the build scheme for iOS. The dropdown selector is empty in both cases. How are you able to hit save and build @Maushundb

[Maushundb]

Idk how your appcenter is setup, but we have a master branches which all PR's are tested against, as well as feature branches. Hitting save and build worked on the feature branch since it had a new commit pushed with the new package structure. On master though, we had to merge the PR first (with failed tests) to master, then once the commit was pushed were we able to change the build settings on master.

[yannickvidal]

@GoMino I had the same issue and it seems like that App Center doesn't like to build.gradle too far down, even though the docs say otherwise. I solved the problem by moving by RN application from ./packages/mobile to ./mobile and modifying ./package.json accordingly

[fesaza]

Hey All, I got it worked for lerna monorepo

My folder structure

/packages
    /common
    /web
    /mobile

My mobile/package.json has something like:

   ...
  "dependencies": {
     "@codenull/common": "0.1.1", //my common package
  ....

I've added a pot-clone script with the following (Basically it installs dependencies first with lerna)

#!/usr/bin/env bash

# move to root folder
cd .. && cd ..

# Install dependencies using lerna
npm run bootstrap

# move to mobile and fix local dependencies (this is a custom js fn)
cd packages/mobile && node appCenterPostClone.js

# run cocoapods
cd ios && pod install

Without appCenterPostClone.js appcenter fails since npm i command during build process is not able to "download" my @codenull/common package.

The solution implemented in my appcenterPostClone file was just change my mobile/package.json. Change "@codenull/common": "0.1.1" to "@codenull/common": "file":"../common"

This is the code for appCenterPostClone.js in case this works for you

const fs = require('fs');

const appCenterPostClone = () => {
  console.log('running');
  fs.readFile('package.json', 'utf8', function(err, data) {
    if (err) {
      return console.log(err);
    }
    const result = data.replace(
      '"@codenull/common": "^0.1.1"',
      '"@codenull/common": "file:../common"',
    );

    fs.writeFile('package.json', result, 'utf8', function(errWritting) {
      if (errWritting) return console.log(errWritting);
    });
  });
};

appCenterPostClone();

[ainursharaev]

I made monorepo using Yarn workspaces. My folder structure:

/packages
    /common -- api's, models, etc. 
    /web -- React web app
    /native -- React Native mob apps

I encountered a few problems. This is how I've fixed them.

1. "We couldn’t find any Android/iOS projects in your branch" - silly me, I forgot to add the Pods folder in gitignore. Pods folder contains over 5000 files. For some reason, Appcenter couldn't handle my repo with the Pods folder.
2. Added postclone script. I think it is self-describing (thanks fesaza). Don't forget to hit Save and Build button when you add any build script for first time (thanks Brandon)

   
   #!/usr/bin/env bash

move to the root folder

cd .. && cd ..

install dependencies

yarn

compile js files in common folder

cd packages/common && yarn build

move to native folder and fix local dependencies

cd .. && cd native && node scripts/appcenter-postclone.js

install pods

cd ios && pod install

[younes200]

@nilofer any update on this issue ? monorepo is widely used and hacky script isn’t a long terme devops solutions.

[nilofer]

@elamalani to provide an update

[younes200]

Any update on this issue @elamalani please ?

Problem is due to this line which look for react-native-cli on local folder : https://github.com/microsoft/appcenter-cli/blob/v1.1.9/src/commands/codepush/lib/react-native-utils.ts#L212

@jokester This could just run @react-native-community/cli in the current directory instead of calling node_modules/react-native/local-cli/cli.js which run react-native/local-cli/cli.js anyway.

[jacob-israel-turner]

@elamalani Anything I can do to help with this? We use AppCenter and CodePush extensively - we are presently moving to use a mono repo and are running into issues building in AppCenter. I'm currently trying to hack something together, but it would be comforting to know if the AppCenter team is planning to improve the experience around mono repos.

[cheunjm]

@elamalani Would love to get some updates on this. As @younes200 suggests, we just need a way to call react-native-cli from the root of the folder rather than from the local one

[younes200]

The workaround I found is manually coping the cli.js from root to local folder using the appcenter-pre-build.sh. It's ugly but it works. Still waiting a response from Appcenter team : @elamalani @nileliao

  mkdir -p packages/mobile/node_modules/react-native/local-cli/
  cp node_modules/react-native/local-cli/cli.js packages/mobile/node_modules/react-native/local-cli/cli.js 

[elamalani]

Hey everyone, sorry for my late reply on this thread. App Center team has decided to prioritize improvements in reliability and performance for the service through mid-2021. This is mainly because we have accumulated technical debt in the product that needs to be addressed now. Adding new features in the product will be significantly reduced and unfortunately, we don't plan to support monorepos at the moment.

[jacob-israel-turner]

Bummer - sad to hear @elamalani. Will you let us know next year if this makes it into the roadmap?

@younes200 thanks for that pre-build script - it looks like that's getting me across the finish line to get this working in our monorepo.

@elamalani would it be possible to at least provide a small document or something around getting AppCenter working for monorepos? There have been some great suggestions in this issue. Even referencing this issue for those looking through the official docs would be helpful.

[edmofro]

Building off the very helpful answer by @fesaza, here's a tweaked script for anyone who needs to handle multiple local dependencies (note that all of our local deps are prefixed @tupaia/)

const fs = require('fs');

const fixLocalDepsForAppcenter = () => {
  console.log(
    'Fixing local dependencies for appcenter (see https://github.com/microsoft/appcenter/issues/278)',
  );
  fs.readFile('package.json', 'utf8', (err, data) => {
    if (err) {
      console.log(err);
      return;
    }

    const result = data.replace(/"@tupaia\/([^"]*)": "[^"]*"/g, '"@tupaia/$1": "file:../$1"');

    fs.writeFile('package.json', result, 'utf8', writeError => {
      if (writeError) {
        console.log(writeError);
      }
    });
  });
};

fixLocalDepsForAppcenter();

[wmaca]

It seems people are confusing the problems RN already has on monorepos (such as hoisting) with what App Center should do.

The main problem with App Center on our usage is that it checks for yarn.lock inside the app's package folder (which was configured on the Build Configuration UI).

As it cannot be found, it uses npm install and everything goes south.

But that is not the only problem. As it is being called inside your app's directory (e.g. packages/app), it may cause problems according to your setup.

The approach I used requires using Yarn2, and is comprised of two parts:

Part 1 - Install on the root and tricking App Center to use yarn

appcenter-post-clone.sh

#!/usr/bin/env bash
echo "Running post-clone script..."

cd .. && cd ..
yarn install 

cd packages/app

# Trick App Center into using yarn
touch yarn.lock

Part 2 - Avoid using the fake yarn.lock and unsupported arguments from App Center

When App Center runs its npm/yarn install script, now that we have a fake yarn.lock inside in packages/app, it will use yarn.

As we are using Yarn2, it will look for the .yarn/releases/yarn-berry.cjs when calling yarn on the command line.

This call must be hijacked, as App Center is mistakenly using --list and --network-timeout=600000 which are not supported by Yarn2.

Also, we created and empty yarn.lock, which yarn will see as a problem. We need to remove it before yarn actually tries to install anything.

Create a yarn-berry.cjs versions that calls the _yearn-berry.cjs (which is the original file that was renamed):

#!/usr/bin/env node
const fs = require("fs");

const isRunningOnAppCenter = process.env["APPCENTER_BUILD_ID"] !== undefined;

if (process.cwd().includes("/packages/app") && isRunningOnAppCenter) {
    console.log("Removing `yarn.lock` added to trick App center...");

    try {
        fs.unlinkSync("yarn.lock");
    } catch (err) {
        if (err.code !== "ENOENT") {
            throw err;
        }

        console.log(`File (${err.path}) was already removed.`);
    }
}

if (process.argv[2] === "list") {
    console.log("List not supported");
    process.exit(0);
}

if (process.argv[2] === "install" && process.argv[3] === "--network-timeout=600000") {
    console.log("Popping network timeout yarn flag since its not supported");
    process.argv.pop();
}

module.exports = require("./_yarn-berry.cjs");

This second part comes from here: https://github.com/microsoft/appcenter/issues/2134#issuecomment-790823974

[FranFara]

Any update on this issue @elamalani ?

[joshuat]

Since we're sharing hacks while waiting for this to be supported - here's our solution. We're using a preinstall and postinstall script to support Lerna w/ Yarn v1 (no yarn workspaces, no hoisting).

In the preinstall script we create backups of our package.json and yarn.lock before erasing all dependencies from package.json, so when the AppCenter forced yarn install runs there is nothing to do and it moves onto the postinstall script.

In the postinstall script we restore our original package.json and yarn.lock files and then run lerna bootstrap from the root directory which installs our dependencies as expected.

This should be adaptable for anyone using a monorepo setup - just let yarn install run, then place your own build steps in postinstall after restoring package.json and yarn.lock.

preinstall.sh

#!/bin/bash

# Only apply bridge when running yarn install from the project directory
if [[ "$npm_execpath" != *"lerna/cli"* ]]; then
  # Backup our package.json and yarn.lock
  cp package.json package.json.bridge_backup
  cp yarn.lock yarn.lock.bridge_backup

  # Erase all dependencies from package.json
  perl -i -p0e 's/"\w*dependencies":\s*\{[^\}]*\},?//gis' package.json
fi

postinstall.sh

#!/bin/bash

# Only apply bridge when running yarn install from the project directory
if [[ "$npm_execpath" != *"lerna/cli"* ]]; then
  # Restore our package.json and yarn.lock
  mv -f package.json.bridge_backup package.json
  mv -f yarn.lock.bridge_backup yarn.lock

  # Run lerna bootstrap from root directory
  yarn --cwd ../ install --non-interactive
  yarn --cwd ../ lerna bootstrap
fi

[airtonix]

what about when your monorepo actually looks like :

package.json
yarn.lock
brewfile
gemfile
pluginfile
.tool-versions
.eslintrc
tsconfig.json
tsconfig.spec.json
packages/
  design-system/
    button/
    drawer/
    userprofile/
    ...
  tools/
    changesets-formatter/
    fastlane/
    eslint/
  packages/
    logger/
    auth/
apps/
  appone/
    ios/
    android/
    fastlane/
    app/
    package.json
  apptwo/
    ios/
    android/
    fastlane/
    app/
    package.json
  appthree/
    ios/
    android/
    fastlane/
    app/
    package.json
  appfour/
    ios/
    android/
    fastlane/
    app/
    package.json

[alexisloiselle]

After a day of work I finally got it working. I'll share how I achieved it and maybe it'll help someone who comes across this thread.

I liked @joshuat's approach and tweaked it a little for my needs. My needs resemble @airtonix's where my app is in the apps/ directory and uses local packages from the packages/ directory.

On App Center, I configured the build to use the package.json from the sub project (in apps/my-app) with these two custom scripts.

appcenter-post-clone.sh

#!/bin/bash

# Backup the package.json
cp package.json package.json.backup

# Erase all dependencies from package.json
# Prevents App Center from installing dependencies from this directory
perl -i -p0e 's/(,\s*)?"\w*dependencies":\s*\{[^\}]*\},?//gis' package.json

appcenter-pre-build.sh

#!/bin/bash

# Restore backup package.json
mv package.json.backup package.json

# Install monorepo dependencies instead
yarn --cwd ../../

And that's all that needed to be done for my use case.

[younes200]

After a day of work I finally got it working. I'll share how I achieved it and maybe it'll help someone who comes across this thread.

I liked @joshuat's approach and tweaked it a little for my needs. My needs resemble @airtonix's where my app is in the apps/ directory and uses local packages from the packages/ directory.

On App Center, I configured the build to use the package.json from the sub project (in apps/my-app) with these two custom scripts.

appcenter-post-clone.sh

#!/bin/bash

# Backup the package.json
cp package.json package.json.backup

# Erase all dependencies from package.json
# Prevents App Center from installing dependencies from this directory
perl -i -p0e 's/(,\s*)?"\w*dependencies":\s*\{[^\}]*\},?//gis' package.json

appcenter-pre-build.sh

#!/bin/bash

# Restore backup package.json
mv package.json.backup package.json

# Install monorepo dependencies instead
yarn --cwd ../../

And that's all that needed to be done for my use case.

No need just copy reference of react-native-cli as suggested here : https://github.com/microsoft/appcenter/issues/278#issuecomment-703261762