Alternatives to Buddybuild?

rnystrom commented 6 years ago

Just as I was getting familiar w/ it....

Existing Free Starter plans and Android app development will be discontinued on March 1, 2018.

We will almost certainly lose support for this project. Time to find a new CI. Any suggestions?

I'm reading this article which offers:

I read about Buildkite and AppCenter on Hacker News.

I'm also considering open source, self-hosted solutions so that something like this doesn't happen again:

dkhamsing commented 6 years ago

https://docs.microsoft.com/en-us/appcenter/test-cloud/

_{Sent with GitHawk}

ElektrojungeAtWork commented 6 years ago

@dkhamsing App Center Test actually runs UI Tests on physical devices – we have a few thousand. Nope, you can't see pictures ;)

cbedoy commented 6 years ago

We're actually trying to identify a new CI solution.

AppCenter: similar to bb but doesn't provide PR support, I think is more focused to management people, also the logs doesn't provide a stack if any task fails.
Bitrise: Very configurable, offers many open "steps" like codecoverage, deploy, sign, unitTest, UITest, build, deliver, clean and custom, cuz you have the ability to configure it, just got a little confused with .yaml files, you can trigger steps given Push, PR, etc
Nevercode Very configurable, you can select between gradlew task per branch, build PR, no free plan.

I think at least Bitrise offers many features that we can exploit!

acristescu commented 6 years ago

From the link posted above about testing in App Center

Review the core concepts Understanding the core concepts of the Test Cloud experience improve ease of use, navigation, and communications with support. It is recommended to become familiar with these concepts before running your first tests.

What the hell... I don't want to review any concepts, core or otherwise, I just want it to work with 2 clicks like it did on BB :( I don't have 10 hours to sink into making this work, I'm a coder not a devops engineer...

dkhamsing commented 6 years ago

Yea the App Center documentation could be streamlined

_{Sent with GitHawk}

viktorbenei commented 6 years ago

@acristescu

After reading this thread it does seem to be a two-horse race: Bitrise and App Center. Yet nobody has touched on the subject of UI tests: I loved how in BB, with just a few clicks, you could run espresso tests on a virtual device. Does any of the two platforms under discussion support this?

Not (yet) a single click but also more powerful in a couple of ways: https://blog.bitrise.io/introducing-solid-and-snappy-virtual-device-testing-for-android

We're working on making the setup easier (that's why it's still "beta", not because of lacking functionality ;)).

ElektrojungeAtWork commented 6 years ago

@acristescu @dkhamsing We are aware! Keep the feedback coming.

acristescu commented 6 years ago

@viktorbenei I'm going to try but oh my god that art style is off-putting. I remembered why I stayed away from Bitrise in the past... smiling clouds hugging each other? Sharks with lasers strapped to them?!? Green buttons over a vibrant purple background? Perhaps I'm just showing my age here, but how could I recommend this tool to a client that is... say... a bank?

viktorbenei commented 6 years ago

No problem @acristescu , I definitely see your point, honest feedback is always welcome and a design update is already in the works ;)

acristescu commented 6 years ago

I decided to try them both with a simple repo ( https://github.com/acristescu/GreenfieldTemplate ) and see where I get to. So far I've tried App Center and hit a few snags:

solved (it couldn't find the gradle build tools!), you have to manually add the google() repository to your project's gradle
it restarted the build number from 1 whereas I already released on play store a 42, google is just going to reject the build!)
I couldn't find how to test on a virtual device for free, only on real devices with a 30 day free trial.

I'm not sure it ran the unit tests because

##[warning]No test result files matching /Users/vsts/agent/2.127.0/work/1/s/**/build/test-results/TEST-*.xml were found, so publishing JUnit test results is being skipped.

Not sure what's that about...

acristescu commented 6 years ago

Bitrise: On the plus side the setup was just as painless although I think that if I need to change anything I need to bring up an yml file and fiddle with it (update: found a thing called workflow editor. it looks scary but powerful). Snags:

it never asked me which variant to build and picked the wrong one. I wanted prodRelease but for whatever reason it decided to build exactly the other two mockDebug and prodDebug. Can't find where to change that, but I'm pretty sure there must be one.
build took longer, 4 mins instead of 2 mins 16 for app center. This is perhaps because of the above problem?
don't see any mention of junit tests anywhere in the logs. I doubt it ran them. Not obvious how to add them, perhaps somewhere in workflow editor? (update tinkered with the Workflow Editor for about 10 minutes and found it. Bonus points for allowing me the liberty to pick which test target to run)
not sure what build ID it used, how do I even see that?

dkhamsing commented 6 years ago

Thx for posting these

_{Sent with GitHawk}

acristescu commented 6 years ago

matthiaswenz commented 6 years ago

Thanks @acristescu for the detailed feedback, we greatly appreciate it. Especially on the warning for JUnit test report files in App Center, this is not affecting your actual test run and should be fixed soon. Keep it coming!

acristescu commented 6 years ago

It took me two hours but I managed to convince app center to upload to google play. However, I cannot seem to convince it to do this automatically, I had to download the signed APK from the app center and then upload it back up into the deploy/stores section (!) to make it work. Seems awefully convoluted, what am I doing wrong?

PS: To add insult to injury, BuddyBuild deployed several times in the same timespan since I forgot to disable it in the beginning and it just works automatically without any human intervention...

viktorbenei commented 6 years ago

Hi @acristescu ,

Re: https://github.com/rnystrom/GitHawk/issues/1330#issuecomment-368228417

it never asked me which variant to build and picked the wrong one. I wanted prodRelease but for whatever reason it decided to build exactly the other two mockDebug and prodDebug. Can't find where to change that, but I'm pretty sure there must be one.

Indeed our current scanner will add a Gradle Runner step, with assembleDebug configured for the base workflow. We realise that this might not be straightforward enough, but in short if you want to build prodRelease then the gradle task is assembleProdRelease. If you want to run lint then the gradle task is lint. You can do all of these with the Gradle Runner step, in fact gradle can handle multiple tasks so to run lint and then assembleProdRelease you can also specify this as the task: lint assembleProdRelease which will do both.

We're working on new steps & new scanner default configs which will make this easier, with more specific steps (e.g. a "Lint" step which runs the gradle lint task, instead of requiring you to set that task in the "Gradle Runner" step) 😉

build took longer, 4 mins instead of 2 mins 16 for app center. This is perhaps because of the above problem?

Indeed that seems to be the case, as assembleDebug most likely generates 2 separate APKs / variants in your case instead of the single "ProdRelease" one.

don't see any mention of junit tests anywhere in the logs. I doubt it ran them.

Specify test as the gradle task input of the Gradle Runner step, that will run your tests - or add the Gradle Unit Test step which is configured to run that gradle task by default.

not sure what build ID it used, how do I even see that?

If you mean whether we set the build number to the bitrise.io build number: by default we don't, you can do that by adding the Change Android versionCode and versionName step for example.

Thanks again for your feedback, we're listening & already scheduled to improve these points of the setup experience! ;)

jamesone commented 6 years ago

Awesome discussion. I've been finding it hard to find a BuddyBuild alternative that supports Carthage.

I looked into Nevercode, they only support cocoapods.

I believe the App center supports Carthage.

Any others?

cbedoy commented 6 years ago

@jamesone

I think the best option for you could be Bitrise, they provide a platform like pipelines bitbucket, also you can configure based on your needs using Steps.

Actually we moved from bb to bitrise, we're using Android and iOS and everything is perfect!

jamesone commented 6 years ago

Awesome @cbedoy What did you do about the installable builds that buddybuild provides for all your branches? Does Bitrise have an integration OR support for this?

cbedoy commented 6 years ago

You can trigger workflows (many steps) when you push, create a PR or tag.

Also you can schedule buils per branch.

You should check:

https://devcenter.bitrise.io/bitrise-cli/workflows/ https://devcenter.bitrise.io/bitrise-cli/steps/

When you undertand how bitrise works can create worksflows based of what you need, i.e I want a workflow where just running unitTesting if someone create a PR or I want a workflow where build and generate the .ipa when master was tagged.

Bitrise is something like docker images where you can chose a thirdparty steps for run unitTest, CodeCoverage, or Archive and deploy.

jamesone commented 6 years ago

Awesome man! Sounds really interesting. I'll look into it.

GitHawkApp / GitHawk

Alternatives to Buddybuild? #1330