Discussion Episode 6, Part2: Continuous Integration (CI) & Continuous Delivery (CD) with Fernando Cejas (& EVERYTHING & LIFE)

[artem-zinnatullin]

// I'll be on vacation (first ****\ time in my life, yay!) soon so I'm assigning discussion to you guys @sockeqwe @android10, deal with it :trollface:

[johnjohndoe]

You are missing the GitHub release for this part.

[sockeqwe]

Thanks for the hint, I'm uploading it right now.

Update: Now available in release section: https://github.com/artem-zinnatullin/TheContext-Podcast/releases/tag/Episode_6_Part2

[johnjohndoe]

Question: Jenkins building apps on library change

You mentioned in this episode that you use a shared library in two Android application projects. Then you said whenever you change the library you let Jenkins run all tests of the application projects to ensure the library changes do not break the applications. Can you please explain how you trigger this in Jenkins and which branch strategy you use for this?

[sockeqwe]

With jenkins you can specify dependencies to other projects so that jenkins can run them automatically after having build a certain project.

Unfortunately, this doesn't work out too good for our use case because we also have to increment the library version in build.gradle file for all depending projects (that include this library).

Maybe there is a better way (or a jenkins plugin) but our solution is basically based on shell scripts. The pipeline looks as follows:

1. Commit changes to shared library and push changes (after code review etc.)
2. jenkins will build the library and will automatically increment the version number of the library.
3. After jenkins has built the library, a shell script starts: This shell script does the following: for all apps that are depending on this library (hard coded in shell script) clone the apps repository, then create a new branch called shard-lib-new-version-number, then search the build.gradle script of that app for the library dependency definition compile my.company:shared.library:version and set the new version number (library is available through an internal maven repository). Then commit this change and push the new branch shard-lib-new-version-number.
4. Since jenkins is reacting on each push event, jenkins will build the app with the new library version on the new branch.
5. It depends if build succeeds of fails:
   • If that build fails --> Send a notification on HipChat (yes, we still use HipChat 😄 ) to android-dev channel that the build has failed. Then the library has broken this app and someone (ideally the author of the commit in library project which has broken this app) have to investigate ...
   • If build succeeds --> Create a pull request to merge branch shard-lib-new-version-number into development branch from the app. We could merged automatically to development without pull request, but we want that a real person take a look on the app on a real device because we don't have 100 % test coverage ...

[android10]

To build on @sockeqwe, in our case is very simple: @johnjohndoe when you bump the library version in your Android project, Jenkins will detect and pick up those changes and will run unit test, acceptance tests and static code analysis as it does with any change that happens in master.

[android10]

Regarding your question about branch strategy, we basically have master, release and feature branches. You send PRs that need 2 👍 in order to be merged to master unless you paired with someone, so in that case you only need 1 👍

[qazt482]

Hi,

I didn't find your criticism of Jenkins very compelling. I've been using Jenkins for many years. Its not perfect by any means but the browser refresh issue is not there any more. There is a button top right that says 'enable auto refresh'.

[artem-zinnatullin]

Which reloads the page periodically, which is not a solution :)

On 7 Sep 2016 3:07 pm, "Rakesh Patel" notifications@github.com wrote:

Hi,

I didn't find your criticism of Jenkins very compelling. I've been using Jenkins for many years. Its not perfect by any means but the browser refresh issue is not there any more. There is a button top right that says 'enable auto refresh'.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/artem-zinnatullin/TheContext-Podcast/issues/52#issuecomment-245274284, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3LkHSe33LzV5Bot7FL0ofx1fJzmcks5qnrbugaJpZM4Ju0Z5 .

[qazt482]

CI is not meant to be watched continuously. If you are, then you're doing it wrong :)

Its meant to be running in the background, automatically triggered by SCM commits and notifying you if anything fails (or passes if you want). Emails, slack whatever. When you get notification of a failure you go to the job instance in question and examine it further. Having an up to date UI is not important.

I realize you must know this, which is why I am baffled by your flippant attitude on this.

It would be a shame if beginner developers took what you said to heart and dismissed Jenkins for reasons that are not important in the real world, or based on your faulty preconceptions.

[artem-zinnatullin]

Well, I partially agree with your points, but I can't say that Jenkins works great without monitoring. For example we have a job that builds releases by watching for new git tags and we've noticed by message in Slack that Jenkins decided to build already existing tag just in the middle of the day without any changes from our side (tags are immutable in our code storage). Such things happens with Jenkins because it's a very huge and old piece of code + plugins, if I would trust it I wouldn't monitor it manually from time to time!

On 7 Sep 2016 7:32 pm, "Rakesh Patel" notifications@github.com wrote:

CI is not meant to be watched continuously. If you are, then you're doing it wrong :)

Its meant to be running in the background, automatically triggered by SCM commits and notifying you if anything fails (or passes if you want). Emails, slack whatever. When you get notification of a failure you go to the job instance in question and examine it further. Having an up to date UI is not important.

I realize you must know this, which is why I am baffled by your flippant attitude on this.

It would be a shame if beginner developers took what you said to heart and dismissed Jenkins for reasons that are not important in the real world, or based on your faulty preconceptions.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/artem-zinnatullin/TheContext-Podcast/issues/52#issuecomment-245357269, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3AL75EmcriUEMuhdp3zmEWRlBOxbks5qnvUtgaJpZM4Ju0Z5 .

[qazt482]

All software has bugs. I cant say I've ever hit any bugs with Jenkins in the years I've been using it (though I have had problems with 3rd party plugins).

There is a new version of Jenkins that perhaps you should check out (I believe the UI has been revamped) but I've not tried it myself since Jenkins I use works fine.

[aaronweihe]

Awesome podcast! Thanks! I have a few questions:

• Who owns the UI/acceptance testing? QA, developer, or both?
• How to deal with regression tests? That's the main problem blocking us from releasing faster because our regressions tests are 99% manual and would take roughly two weeks. We have somewhat automated UI testing but it's far less than enough to replace the entire regression tests.

[artem-zinnatullin]

@qazt482 I've been using Jenkins for about 5 years, I keep it as minimalistic as possible and carefully admin it and still it breaks from time to time and I have to spend time to fix problem.

Regarding new UI: I've tried it, but it requires you to use Jenkins Pipelines which we don't need to use, mainly because we don't have Pipelines and because we don't want our project to know about CI system we use (it requires you to store Jenkins config under version control of the project which we already have abstracted away from concrete CI system). And new UI is not ready yet, it's in alpha and was pretty unstable when I tried it about 3 weeks ago.

@aaronhe42 thanks man!

Who owns the UI/acceptance testing? QA, developer, or both?

On project I work at Juno both QAs and developers own, write and maintain UI tests, but at the moment most of them are written by our QA, ideally everyone should contribute them as needed.

How to deal with regression tests?

Automate them! You can test db upgrades separately, other cases can be covered with UI tests. We also write unit tests for regressions if it's possible.

@android10, @sockeqwe what about you?

[sockeqwe]

Regarding Jenkins, we use Jenkins a lot and it works for us good. We have no super fancy pipeline or tons of plugins, so Jenkins does the job for us. Moreover, it's open source and can be used for free. I, personally don't have high expectations or criterias regarding a CI server. As long as it builds stuff, I'm fine with it 😄

Who owns the UI/acceptance testing? QA, developer, or both?

Both, but in contrast to Artem and Fernando I'm not developing (most of the time) for our own product of our company, but rather develop apps for customers / partners (agency alike). We don't have a QA team, but our customers / partners have. So the final decision is made by their QA team.

How to deal with regression tests?

As Artem have already said: Automate them! It's worth the effort. I, personally (and I know that Artem will not agree with me on this), don't write too much UI tests, because our UI is super simple.

[qazt482]

@sockeqwe regarding Jenkins we are in agreement. I've used TeamCity many years ago and it was also a good choice but not free. I believe Circle CI is the new hotness but not tried it myself.

I'm a bit more ardent about UI tests. I don't think they are worth the effort (flaky, brittle, hard to maintain, etc). Unit test everything else and keep the UI dumb works for me.

[sockeqwe]

@qazt482 I guess, I'm just that kind of (lazy) guy, who dont really care too much about the underlying CI server. Jenkins was already there when I joined the company and it works (sure not perfect, but sufficient and who is actually perfect ... ). I could also use Travis, Circle CI, TeamCity, BuddyBuild or whatever else is available out there. I don't really care too much about it (as long as I don't have to spend weeks to setup a CI server) ...

In my opinion Artem made a fair point by saying Jenkins doesn't work to well for his needs, but he also clearly stated out that he don't want to discredit Jenkins and respects and admires all the hard work put into this open source project.

[android10]

@aaronhe42

Who owns the UI/acceptance testing? QA, developer, or both?

We as developers have to make sure of the quality of the code we write, so we basically write acceptance tests. We have no QA Team and our Testing Engineering Team takes care of all the tooling we use like testing frameworks, CI maintenance and so forth.

How to deal with regression tests? That's the main problem blocking us from releasing faster because our regressions tests are 99% manual and would take roughly two weeks. We have somewhat automated UI testing but it's far less than enough to replace the entire regression tests.

As pointed out above, we rely in our test battery but before each release, the entire team spends a couple of hours doing regression tests, basically by following up a regression test checklist we have.

[artem-zinnatullin]

Totally, definitely great that we can use Jenkins for free and contribute to it by plugins and PRs.

On 8 Sep 2016 12:42 pm, "Hannes Dorfmann" notifications@github.com wrote:

@qazt482 https://github.com/qazt482 I guess, I'm just that kind of (lazy) guy, who dont really care too much about the underlying CI server. Jenkins was already there when I joined the company and it works (sure not perfect, but sufficient and who is actually perfect ... ). I could als use Travis, Circle CI, TeamCity, BuddyBuild or whatever else is available out there. I don't really care too much about it (as long as I don't have to spend weeks to setup a CI server) ...

In my opinion Artem made a fair point by saying Jenkins doesn't work to well for his needs, but he also clearly stated out that he don't want to discredit Jenkins and respects and admires all the hard work put into this open source project.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/artem-zinnatullin/TheContext-Podcast/issues/52#issuecomment-245546853, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3PSNnpzqeCfUqRSX37vW4CxV0ooFks5qn9hwgaJpZM4Ju0Z5 .

[meoyawn]

can anyone help me choose between AVD, genymotion or a real device for running simple smoke tests on every PR on the CI (we're using jenkins)

what are the costs of maintenance of each choice? what about the reliability of the setup? in case of AVD/genymotion: do you spawn them for each build or do you keep them alive all the time?

[artem-zinnatullin]

We use Emulators 2.0 and spawn new with clean state for each build (takes about 40 seconds), costs only physical machine with KVM, ideally I'm planning to turn it into own SAAS — pool of Emulators with adb over network.

On 10 Sep 2016 1:18 am, "Nizamutdinov Adel" notifications@github.com wrote:

can anyone help me choose between AVD, genymotion or a real device for running simple smoke tests on every PR on the CI (we're using jenkins)

what are the costs of maintenance of each choice? what about the reliability of the setup? in case of AVD/genymotion: do you spawn them for each build or do you keep them alive all the time?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/artem-zinnatullin/TheContext-Podcast/issues/52#issuecomment-246056438, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3JsyBva-oxr97OZYr7sTXEQPlTR-ks5qodtLgaJpZM4Ju0Z5 .

[meoyawn]

@artem-zinnatullin out of, say, 100 builds, how many emulator failures do you get?

[artem-zinnatullin]

Haha, great question, 1-2. Sometimes emulator does not see network and our app checks for that and shows "No internet connection" dialog

On 10 Sep 2016 9:41 pm, "Nizamutdinov Adel" notifications@github.com wrote:

@artem-zinnatullin https://github.com/artem-zinnatullin out of, say, 100 builds, how many emulator failures do you get?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/artem-zinnatullin/TheContext-Podcast/issues/52#issuecomment-246128408, or mute the thread https://github.com/notifications/unsubscribe-auth/AA7B3CwiZ02dbLzBYuhqLqGCPbUT7cOIks5qovncgaJpZM4Ju0Z5 .

[PrashamTrivedi]

Hi, Some questions for all of you.

1. Do you have any code quality measurements in your CI? We use Jenkins here, and I am more interested to join Lint and Findbugs to my build process, want the code to pass some threshold, say the code is not mature if it has > 500 warnings and such code should not be merged. Do you have such practice? If you have, where does code analysis fit? And what do you do if the code does not pass a defined quality threshold?
2. Do you promote same apk from beta to production track in Google Play? or you change something before uploading apk to production? How do you identify if a user is running a Beta app or production app?
3. Do you follow an architecture from day 1 (when you start working on certain feature)? or you gradually refactor a code to follow that architecture? How difficult this is for the teammates to understand?

[sockeqwe]

Do you have any code quality measurements in your CI? We use Jenkins here, and I am more interested to join Lint and Findbugs to my build process

We also use findbugs and lint. if findbugs detect an error the whole build breaks. this happens for every pushed commit, which triggers a build. Basically our pipeline looks like this:

Developer writes code in his own feature branch 
     --> developer commits & pushes code to remote (origin) feature branch 
     --> CI builds this branch (triggered by every new commit). Building includes running tests and static code analysis via findbugs and lint 
     --> if build fails Developer will get a notification of the failed build (email and hipchat).
            if build succeeds (this means no error found by static code analysis and all tests passed ) then developer is allowed to create a pull request 
     --> Pull request will be reviewed by one other developer (randomly, whoever has time to review
           it, or can be assigned explicitly if you want that someone reviews your code explicitly. 
          We also use this as a indicator to say, if assigned explicitly to someone, 
          please review it as soon as possible ... we are a small team)
     --> Other developer reviews code and merge is into main branch (development branch) 
     --> CI server builds development branch 
     --> if fails, notify reviewer and feature branch developer, if passes everything is ok

That's basically it. At soundcloud as @android10 as far as I understood the development branch will be build with additional functional and integration and acceptence tests (these tests will not run on feature branch builds at soundcloud, but only on development branch builds). We don't have too many integration tests or end-to-end tests. 90% of our tests are small unit tests and therefore run on every branch during CI builds. As far as I remember @artem-zinnatullin also runs all tests on all branches (including integration tests and acceptance tests etc.) That's probably the main difference here between us 3:

Hannes: run all tests on each branch for each commit (as alreaidy said, 90% are simple unit tests running on jvm, the rest is UI and functional tests). Running those tests takes about 5 minutes Artem: run all tests on each branch similar to Hannes, but Artem has a lot more functional and integration tests, so running those tests takes longer (I don't know how long exactly, but I expect more than 5 minutes). Fernando: Run simple tests that don't take to long on every branch. However, run functional integration tests and acceptence tests only on development branch as it takes long (40+ minutes) to run this tests.

Obviously, when I say Hannes, Artem, Fernando I mean "and their team".

want the code to pass some threshold, say the code is not mature if it has > 500 warnings and such code should not be merged.

In our team: treshhold for fiindbugs is 0. One single detected bug by findbug will fail the build. Same was true for LINT, but sometime ago we decided to not be as strict with LINT. LINT won't fail any build anymore. BAD IDEA, don't do that. I was against this decision, but overall I support the decision since we made the decision as team. It all started like this: We had to deliver a test version of our app, but build failed because of lint, so we turned lint off "just for this build" and turn it on afterwards. Well, somehow we didn't enabled lint again. It print warnings, but doesn't fail the build anymore. We have now a huge number of warnings, but we use the common "no time for fix that" excuse to remove all lint warnings / errors. Again, I wouldn't suggest to do that, but being part of a team also means that there are decisions made by us as team that you personally dislike.

I also know that other developers run JaCoCo to generate code coverage reports and fail builds if code coverage is under a certain procentage number, but I, personally, don't believe in such numbers. Having a code coverage of 80% or 40% doesn't says anything about your code base imho. For instance, we don't have too many UI tests because our UI is very simple. So if JaCoCo says that our code has 60 % testing coverage there are a lot of UI related classes that we explicitly don't want to test. On the other hand, we really want to ensure that the "core" of our app is very well tested. We ensure that by code reviewing unit tests.

Do you promote same apk from beta to production track in Google Play? or you change something before uploading apk to production? How do you identify if a user is running a Beta app or production app?

We don't use google play beta. We manually upload apk's to google play store (happens once However, in Artem's and Fernando's CI pipeline at the end of each build of development branch a APK will be uploaded somewhere (like Crashlytics beta). Fernando also uses Google play beta. I expect that they mainly use the beta channel to get some early feedback but never promote the same apk to production. Hope that he can give us some insights on that.

Do you follow an architecture from day 1 (when you start working on certain feature)? or you gradually refactor a code to follow that architecture? How difficult this is for the teammates to understand?

That really depends on your team. On one hand I think consistency especially on architecture is important. On the other hand, architecture is always evolving over time and that is completely fine. The question is how big steps do you want / can to take between each feature. I our team, we basically have agreed on Clean Architecture and SOLID Principles. We also do MVP to separate the view from Model. So in our codebase you will always find a View interface and Presenter. If someone wants to add Databinding or transition MVVM, we are fine with that in general, but most likely there still will be a Presenter but that then offers a ViewModel (for databinding) to the View. But we also have Model-View-Intent inspired components, but again, here is a Presenter involved. So to us a Presenter is basically the input / output to our apps "core", but that is basically the only requirement we have regarding architecture. Underneath that layer, everybody is free to do what she / he wants. For example I mentioned Clean Architecture. However, we don't follow the books strictly. I.e. there is no Interactor or UseCase class explicitly forced by us to be there, nor do we force someone to use libraries like RxJava (although we use it extensively). Nevertheless, We have some Interactor or Repository alike components but we might don't have named the class Interactor. For us the idea behind the clean architecture is what drives our architecture and that is mainly SINGLE RESPONSIBILITY. We ensure that through code review. We also had some tools to analyse class complexity via static code analysis to detect classes that should be split into multiple smaller classes to follow the single responsibility principle, but at the end nothing beats code review (or pair programming) so we don't have this code complexity tools part of our CI pipeline anymore. So we are quite liberal on that architecural things.

In contrast, I have also worked with other teams and companies that are very strict on architectural things like, we use MVP, there must be a class FooPresenter talking to FooInteractor interacting with FooRepositoy and so on. That is also completely fine. Everybody then knows what class to search, what class to look into and what the class is good for just by looking at the name. It really depends on your team.

However, I think iterating on Architecture (and learning from architectural mistakes) is faster if you don't have very strict rules, but then one may argument that it is just a small step into (architectural) chaos.

[PrashamTrivedi]

In our team: treshhold for fiindbugs is 0. One single detected bug by findbug will fail the build. Same was true for LINT, but sometime ago we decided to not be as strict with LINT. LINT won't fail any build anymore. BAD IDEA, don't do that. I was against this decision, but overall I support the decision since we made the decision as team. It all started like this: We had to deliver a test version of our app, but build failed because of lint, so we turned lint off "just for this build" and turn it on afterwards. Well, somehow we didn't enabled lint again. It print warnings, but doesn't fail the build anymore. We have now a huge number of warnings, but we use the common "no time for fix that" excuse to remove all lint warnings / errors. Again, I wouldn't suggest to do that, but being part of a team also means that there are decisions made by us as team that you personally dislike

In our setup (we use jenkins), I have made 2-3 jobs for same repository. Where one job is specially for Static code analysis, one for regular build and after launch of Jenkins 2.0 I have made one to actually study pipeline feature(Which I am yet to finish BTW). Thus I have made a way that static analysis should never stop regular build, which in many cases requires anyway.

But I have set such a high threshold because of size and timeline of our code base. When we started said project, databinding is not yet announced and we are just learning RxJava. Now it's full of MVVM and RxJava. And right now we are slowly moving into covering as much as data - repository part in testing. I think keeping a high threshold will motivate developers to reduce it, and I can use that leverage because I have separated the jobs. Though I doubt this approach but I have decided to use it for couple of months more.

Same was true for LINT, but sometime ago we decided to not be as strict with LINT. LINT won't fail any build anymore.

I think we can use some lint flags in Gradle can help us to change severity or consider errors as warning. I am yet to try it though.

[artem-zinnatullin]

CI is not meant to be watched continuously. If you are, then you're doing it wrong :)

@qazt482

[user@ci ~ ]# service jenkins status jenkins dead but pid file exists [user@ci ~ ]#

Jenkins ¯(ツ)/¯