Post: How to use CocoaPods with Xcode CI Bots

[hons82]

@AliSoftware 1) actually in my Podfile I specified it without username password, so I had to do that. Maybe if I use keys as @joeblau suggested I could really avoid that

2) I wasn't avare of that... So maybe I could use update in any case

3) This is on a OSX server using the default CI user and I'm not commiting the Pods directory, so I need to setup the Project. However what I'm generally doing is specifying the version number I currently need in the Podfile

[joeblau]

@hons82 In your podfile reference your dependencies with a git@ prefix instead of the https:// prefix

pod 'PrivateRepository', :git => 'git@github.com:username/privateRepository.git'

In your bot's Before Integration run script add these lines. This script is running using the _xcsbuildd account on your server.

export LC_ALL="en_US.UTF-8"
cd <application>
pod install

The final thing you'll need to do is add the public key from your _xcsbuildd (Xcode Build Daemon). Xcode build server actually has two users that will connect to your repos. There is the current user of the system which is running, but your pre/post install scripts are actually run by _xcsbuildd. Both accounts needs access to your private repositories. I outlined the steps above to create a matching public/private key pair for that user.

[kdawgwilk]

Many reading this feed might benefit from learning how Xcode Server works behind the scenes.

Here is an article I recently found that helped me understand what was really happening:

http://honzadvorsky.com/blog/2015/5/4/under-the-hood-of-xcode-server

[AliSoftware]

3) This is on a OSX server using the default CI user and I'm not commiting the Pods directory, so I need to setup the Project. However what I'm generally doing is specifying the version number I currently need in the Podfile

be aware that even if you specify explicit version numbers in your Podfile, that isn't sufficient to guarantee that you will end up with the same versions of all the pods your project uses, because of dependencies.

For example, if your Podfile references that you use pod 'Foo', '1.2.3' then of course CocoaPods will download version 1.2.3 of the Foo pod, but if that Foo pod itself has a dependency to, say, 'AFNetworking', '~>1.0' in its own podspec, then you could get a different version of AFNetworking each time you run pod update.

That's why the Podfile.lock exists. Even if you don't commit the Pods directory, you are still expected to commit the Podfile.lock file (you should never gitignore this file), as this file contains the list of pods and versions that were actually installed the last time you installed your pods, and pod install will use this file to install the exact same versions of the Pods in case you didn't commit the Pods repository, installing the same versions of all the pods — including dependencies — as the last time you wrote and tested your code.

There is no problem not committing your Pods directory to your git repo. But if you don't commit Podfile.lock file or if you make your CI server execute pod update (which will discard the Podfile.lock and update everything to the latest version compatible with what you wrote in your Podfile) instead of making it execute pod install, then your build can pass one day, and even without you committing anything new into your codebase, fail the next day. Even if you didn't change a thing to your project. Just because some dependency (like AFNetworking in the above example) just released a new version that suddenly makes your test fail for some reason.

For more info, see this paragraph in the CocoaPods guides and my comment here explaining in more detail the differences in all the commands and their expected usage (I still need to transform that comment into a guides or blog article someday)

HTH

[jeffreyjackson]

Anyone have luck with getting private pod repos working?

[joeblau]

@jeffreyjackson I've gotten it working. I'm going to write a blog post on how to do it because all of the pieces necessary are in this thread, but it's a mess to go through. What part are you stuck on?

[jeffreyjackson]

@joeblau hey man, thanks for the quick reply. I have mostly everything working with the following exceptions:

For a couple of the projects we have here we are referencing a specs repo which points to public and private repos. We come across the error during pod install on the server side because it can't get to the private pod repo.

[joeblau]

@jeffreyjackson Yep. The reason that is happening is because the xcodebuild server has a separate daemon that is actually trying to connect to your private repositories and it doesn't have a public/private key pair to be able to pull from your private repos.

You need to run these steps on the build server. What you're doing is switching to the build server daemon user, creating a public/private key pair, and then pasting your public key into whatever git server host you're using.

sudo -u _xcsbuildd /bin/bash
ssh-keygen -b 4096
<enter>
<enter>
<enter>
cat /var/_xcsbuildd/.ssh/id_rsa.pub | pbcopy

Once the _xcsbuildd user has access to your repository, you should be able to pull from your private repositories. You can also always do a quick test when you're sudo'ed in as _xcsbuildd and try and clone your repo down into a temp directory before you run the full CI process on your Xcode Build Server.

[jeffreyjackson]

I'll try this and report back. Thanks @joeblau

[hons82]

@jeffreyjackson we finally got it working, but only using keys. Our specs repo is private and almost all the pods inside too, so we accessing all of them using ssh keys

[jeffreyjackson]

hey @joeblau - should that last line be cat /var/_xcsbuildd/.ssh/id_rsa.pub | pbcopy instead?

[jeffreyjackson]

@joeblau so after you sudo -u, if you do a cd ~ and pwd. you'll be in /var/_xcsbuildd/?

[jeffreyjackson]

Whats odd here is that I've changed the user to _xcsbuildd, but ~ is still /Users/jeffreyjackson and not /var/_xcsbuildd

I'm running 10.10.3 and newest Server.app. All are fresh installs on a clean disk couple days ago.

[joeblau]

@jeffreyjackson The public key should be in /var/_xcsbuildd/

[jeffreyjackson]

where does your key reside? in /var/_xcsbuildd ?

[jeffreyjackson]

thanks!

[joeblau]

I just checked on a local server: /var/_xcsbuildd/.ssh/id_rsa.pub | pbcopy is the correct command. I also went and updated all of my incorrect comments. Thanks.

[jeffreyjackson]

Right. So I believe we are aligned in our installations. All of my ssh-keys look like to be set up correctly. For example:

bash-3.2$ whoami
_xcsbuildd
bash-3.2$ ssh -T git@github.com
Hi jeffreyjackson! You've successfully authenticated, but GitHub does not provide shell access.
bash-3.2$

So it looks like as if ssh is working correctly. However, while logged in as _xcsbuildd and in the correct Source dir under Integrations/Cache on the server it still fails when i run pod install.

[jeffreyjackson]

Update: While sudoed as _xcsbuildd, i can clone the private pod repo using git without issue. pod install still doesn't want to work on the private pod repo however.

[joeblau]

What does your pre script look like?

[jeffreyjackson]

export LC_ALL="en_US.UTF-8"
cd <my_clients_repo>
pod install

[jeffreyjackson]

hey @joeblau - I noticed in your message to @hons82 on May 8th that the Podfile needs to specifically call out :git => 'git@github.com:username/privateRepository.git'

In my case, I'm just calling the pod by name only in the Podfile, and within the podspec I use :git => 'git@github.com:username/privateRepository.git'. This should be sufficient correct? Or do I need to update the Podfile to explicitly call this path?

I would think that podspec is the appropriately place for this, and that I don't need to make a change. (It's working right now locally, but not on the server when i pod install)

[joeblau]

The :git => needs to point to your private repository. That is just a generic example so for this repo (if this was a cocoa pods repo) it would be:

pod 'Blog', :git => 'git@github.com:CocoaPods/blog.cocoapods.org.git'

If you don't explicitly add the private repository, then CocoaPods is not going to know where to pull from.

[jeffreyjackson]

Right. I'm not specifying this in my Podfile though and it works fine locally. This Podfile calls pods from the master cocoapods specs repo, as well as private and public pods from my personal specs repo.

In this personal specs repo, I have a particular podspec that references a private github repo. In this specific podspec, I'm referencing the source just as you have suggested.

I know the specs repo is working with Xcode server because its able to reference my personal specs repo and pull down my personal public pods when I do a pod install. This works without issue for the projects that only use the public pods from my personal specs repo. The projects that do fail however, are the ones which pull private pods from the personal specs repo.

tl;dr

public pods - works correctly personal public pods - works correctly personal private pods - do not work

[joeblau]

Alright, I think I might know what the issue is. There are two users at work when you're dealing with the Xcode build server. There is your active user which is the user that you're running and there is the _xcsbuildd. You need both of the users public certificates to have access to your repositories. They are two separate public keys: One in /var/_xcsbuildd/.ssh/ and one in ~/.ssh/.

This is what my pub key upload looks like on GitHub. My machine user is called "miniboxx" and as you can see there are two separate public keys which are uploaded to my account to give me access to GitHub via ssh.

If you don't have both public keys uploaded, from what I've seen it won't work.

[jeffreyjackson]

Hey thanks that did the trick!

[joeblau]

@jeffreyjackson No problem, glad it's finally working. Oh FYI, if you re-install Xcode Server, I think it re-creates the _xcsbuildd user. I updated to Xcode Server 5 last weekend and my keys were all gone. Just something to keep in mind if you're writing documentation or anything.

[skydivedan]

quick question... when I do this:

sudo -u _xcsbuildd /bin/bash

I get this:

error retrieving current directory: getcwd: cannot access parent directories: Permission denied

Any idea why this would happen, or how to mitigate this?

[jeffreyjackson]

you have sudoed while on a directory that you don't have access to. After you sudo, try to cd ~

[skydivedan]

argh, of course. thx.

[rishigoelbr]

@joeblau @jeffreyjackson Hey guys, I need some help. Sorry if this has been discussed before but I am not yet clear how to get this to work. I am still stuck on how to use private pods with xcode bots. So let me explain the problem I am facing. In total we have 12 bitbucket repos for our project (imagine the names below), each one of them is an xcode project on its own: Repo_Main.git Repo_1.git Repo_2.git . . . Repo_11.git

Repo_main is the main project which builds the app for us but it needs the code from all the other 11 repos so we use cocoapods for that and we do "pod install" during the build. Our podfile has the following content

source 'https://github.com/CocoaPods/Specs.git' platform :ios, '7.0' xcodeproj './XYZ.xcodeproj' pod 'Repo_1', :path => '../Repo_1' pod 'Repo_2', :path => '../Repo_2' . . . pod 'Repo_11', :path => '../Repo_11' pod 'AFNetworking', '~> 2.0'

(As far as I know) One bot can checkout only one repo. In my case when I create a Bot for Repo_Main it checks-out this repo without any issues. But in order to run "pod install" in the Repo_Main as a part of the script in the Pre-Integration trigger, I should first of all have all the other 11 repos downloaded/checked-out at the same folder level as Repo_Main. How do I checkout these 11 repos?

[kdawgwilk]

@rishigoelbr To make things easier for the server you need to point your podfile at the git remotes instead of local directories, that way when you have all the ssh keys setup (discussed heavily in this feed) the server will look to the remotes and pull those pods down from bitbucket automatically for you (built into cocoapods gem).

eg.

source 'git@bitbucket.org:company/Repo_1.git'
source 'git@bitbucket.org:company/Repo_2.git'
.
.
.
source 'git@bitbucket.org:company/Repo_11.git'
source 'https://github.com/CocoaPods/Specs.git'
platform :ios, '7.0'
xcodeproj './XYZ.xcodeproj'
pod 'Repo_1'
pod 'Repo_2'
.
.
.
pod 'Repo_11'
pod 'AFNetworking', '~> 2.0'

[czechboy0]

Here's a tutorial on how to introduce a private Pod on Xcode Server: http://honzadvorsky.com/articles/2015-08-17-17-30-xcode_server_tutorials_3_prebuild__postbuild_scripts/

[kdawgwilk]

@czechboy0 :+1:

[skydivedan]

haven't tried it out yet, but @czechboy0 - well done. explained some things I've been missing.

I'm hoping someday that with all the knowledge that accumulated on this issue, we get a blessed "Here's how you do bots with cocoapods" section of the cocoapods guide. Don't get me wrong, your tutorial is most excellent, it would be nice to have it be part of the cocoapod docs themselves, so we can avoid tracking down various git-hub tickets on this.

[czechboy0]

@skydivedan FWIW, I can provide the markdown source to my article and someone can copy/paste anything from there into the guide.

[joeblau]

@czechboy0 Thanks for posting this. :+1: :100:

[orta]

The problem with providing something like this as a guide is that there's so much weird hackery going on. It's a great blog post, and it stands well on it's own (and within @czechboy0's series of digging in on Xcode bots) but I wouldn't want to try make this canon.

[czechboy0]

@orta my feelings as well. A link with a "hack solution" warning might be enough for people that are willing to go through the process. It's enough trouble that I understand that you don't want to make it a part of the official docs.

[skydivedan]

fair enough. I do understand that the solution isn't an easy one (and might possibly break in a future version of xcode server). It's just valuable information, and a real shame that there isn't an easier way to accomplish this.

[czechboy0]

@skydivedan Yeah not as far as I know. I don't think this solution will ever stop working, there might just be an easier way to do this at some point (fingers crossed).

[peterjenkins]

Apologies if this is not the right place to ask, but is it possible to use Xcode bots when you don't check in your generated xcworkspace?

Normally our steps are to clone the repo, run pod install, open the generated xcworkspace file, then finally try to run the app.

[czechboy0]

@pjenkins-cc I don't think so, because a valid shared xcscheme needs to be checked in for Xcode Server (or any CI for that matter) to know how to build the code. And those schemes can also be stored in the workspace (or the project).

I think the issue would be that Xcode Server would try to build the project (not the workspace) and not build the needed pods, thus fail. But I haven't actually tried it.

If I were you, I'd check in the workspace file, it's a tiny folder with one file pointing to the included project files, that's pretty much it (and that's what I do for all my projects).

[peterjenkins]

@czechboy0 Thanks for the response! We're currently using jenkins, but if we do decide to make the switch to bots, I will try just checking in the xcworkspace file.

[thebarndog]

@czechboy0 I followed your guide pretty closely and while it was awesome, I can't get the integration to build when my project depends on a private cocoapod.

I've done everything: created new ssh keys for _xcsbuildd and add them to github, verified _xcsbuildd can access github via ssh -T git@github.com, and all that jazz.

I have a pre-install script that looks like:

export LANG=en_US.UTF-8
export PATH="/usr/local/bin:$PATH"
cd "$XCS_SOURCE_DIR/myproject"

if [ -e "Pods" ]
then
pod update
else
pod install
fi

No matter what I do, I consistently get the output:

Update all pods
Updating local specs repositories
Cloning spec repo `myproject` from `https://github.com/myproject/CocoapodsPrivateSpecs.git`
[!] Unable to add a source with url `https://github.com/myproject/CocoapodsPrivateSpecs.git` named `myproject`.
You can try adding it manually in `~/.cocoapods/repos` or via `pod repo add`.

Can anyone help me out?

[peterjenkins]

@startupthekid Try referencing the specs repo using ssh instead of https. (e.g. git@github.com:myproject/CocoapodsPrivateSpecs.git)

[czechboy0]

Good point! :+1: On Thu, Oct 1, 2015 at 3:59 PM pjenkins-cc notifications@github.com wrote:

@startupthekid https://github.com/startupthekid Try referencing the specs repo using git instead of https. (e.g. git@github.com:myproject/CocoapodsPrivateSpecs.git)

— Reply to this email directly or view it on GitHub https://github.com/CocoaPods/blog.cocoapods.org/issues/21#issuecomment-144753790 .

[thebarndog]

For ssh, would it not be ssh://git@github.com:myproject/CocoapodsPrivateSpecs.git?

Also in my script I ran pod install --verbose and got this:

Update all pods
  Preparing

Cloning spec repo `myproject` from `https://github.com/myproject/CocoapodsPrivateSpecs.git`
  $ /usr/local/bin/git clone https://github.com/busycm/CocoapodsPrivateSpecs.git myproject
[!] Unable to add a source with url `https://github.com/myproject/CocoapodsPrivateSpecs.git` named `myproject`.
You can try adding it manually in `~/.cocoapods/repos` or via `pod repo add`.

/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/sources_manager.rb:48:in `rescue in find_or_create_source_with_url'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/sources_manager.rb:53:in `find_or_create_source_with_url'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer/analyzer.rb:623:in `block in sources'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer/analyzer.rb:622:in `map'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer/analyzer.rb:622:in `sources'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer.rb:129:in `resolve_dependencies'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer.rb:105:in `install!'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/command/project.rb:71:in `run_install_with_update'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/command/project.rb:156:in `run'
/Library/Ruby/Gems/2.0.0/gems/claide-0.9.1/lib/claide/command.rb:312:in `run'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/command.rb:48:in `run'
/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/bin/pod:44:in `<top (required)>'
/usr/local/bin/pod:23:in `load'
/usr/local/bin/pod:23:in `<main>'
  Cloning into 'myproject'...
  fatal: could not read Username for 'https://github.com': Device not configured

It might also be worth nothing that 1) I'm a contributor on this particular project, myproject is actually an organization and 2) None of the pods in my main Xcode project are private but rather I have a pod A that I made that relies on another pod B that I made and pod B is the only one sitting in a private spec repo.

[czechboy0]

I think the URL is just git@github.com:myproject/CocoapodsPrivateSpecs.git. Without the ssh. Try that maybe? (Also in the log it still seems to be adding the https URL) On Thu, Oct 1, 2015 at 7:07 PM Brendan Conron notifications@github.com wrote:

For ssh, would it not be ssh://git@github.com: myproject/CocoapodsPrivateSpecs.git?

Also in my script I ran pod install --verbose and got this:

Update all pods Preparing

Cloning spec repo myproject from https://github.com/myproject/CocoapodsPrivateSpecs.git https://github.com/myproject/CocoapodsPrivateSpecs.git $ /usr/local/bin/git clone https://github.com/busycm/CocoapodsPrivateSpecs.git myproject [!] Unable to add a source with url https://github.com/myproject/CocoapodsPrivateSpecs.git https://github.com/myproject/CocoapodsPrivateSpecs.git named myproject. You can try adding it manually in ~/.cocoapods/repos or via pod repo add.

/Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/sources_manager.rb:48:in rescue in find_or_create_source_with_url' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/sources_manager.rb:53:infind_or_create_source_with_url' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer/analyzer.rb:623:in block in sources' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer/analyzer.rb:622:inmap' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer/analyzer.rb:622:in sources' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer.rb:129:inresolve_dependencies' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/installer.rb:105:in install!' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/command/project.rb:71:inrun_install_with_update' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/command/project.rb:156:in run' /Library/Ruby/Gems/2.0.0/gems/claide-0.9.1/lib/claide/command.rb:312:inrun' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/lib/cocoapods/command.rb:48:in run' /Library/Ruby/Gems/2.0.0/gems/cocoapods-0.39.0.beta.4/bin/pod:44:in<top (required)>' /usr/local/bin/pod:23:in load' /usr/local/bin/pod:23:in

' Cloning into 'myproject'... fatal: could not read Username for 'https://github.com': Device not configured

It might also be worth nothing that 1) I'm a contributor on this particular project, myproject is actually an organization and 2) None of the pods in my main Xcode project are private but rather I have a pod A that I made that relies on another pod B that I made and pod B is the only one sitting in a private spec repo.

— Reply to this email directly or view it on GitHub https://github.com/CocoaPods/blog.cocoapods.org/issues/21#issuecomment-144803379 .

[thebarndog]

I think that's just the result of not cleaning the project because now, thankfully (?) I'm getting a different error, an SSH authentication error for my repo:

startupthekid/busycm/ios is not a valid repository name
Email support@github.com for help (-1)

startupthekid is my username, busyios is the organization, ios is the project