Feature request: add specs source in Podfile like in Gemfile

[netbe]

Hi,

Before going through with a pull request for this, I wanted to get some feedback on that (useless/ good idea, way to do it...).

I have been using cocoapods in several projects now,and had to deal with non-public libraries or dependencies that are hosted on our private git server.

NEED: In order to install specs for this private libraries I created a git repo listing like cocoapods/specs our private specs.

I have seen the pod repo add command and use it to add the private specs repo but it requires developers on the project to execute a setup.rb script which will add the repo as the first source to check. Two drawbacks:

• this solution is 'kind of hacky' and not very clean.
• the sources are checked by alphabetic order.

SUGGESTED SOLUTION: As in a Gemfile add a source attribute to specify the different sources to check

By default cocoapods/specs will be checked, but if a source is given only the source will be read. Also multiple source could be specified and will be checked by chronological order.

For example, at the top of the podfile

source "http://myspecsrepo.com"

we could also have blocks like:

source: "source1" do
pod "blabla"
...
end

Advantages

• Backup source in case github is down
• Extend cocoapods tool to integrate private libraries

Thanks.

[fabiopelosin]

Sounds good. However, two points are unclear to me:

1. This change would break exiting installations relying on the automatic pick up of other repos. Do you have any idea about how to mitigate this?
2. How do you plan to handle sources, would you fetch them from the remote to ~/.cocoapods if they are missing.

[alloy]

Sounds good to me too. My two cents:

• @netbe I would prefer to let the block variant be and just implement the source directive and make it global. Do you have a real use-case for the block variant yourself atm?
• @irrationalfab My understanding is that with the source directive CP will still create a checkout in ~/.cocoapods, but will only use that one (or maybe even an ordered list of sources?) when resolving. If there is no source directive it would fallback to the current behaviour and thus work with existing projects. @netbe did I understand that correct?

[fabiopelosin]

@alloy, sounds good! — Sent from my iPhone

On Thu, Apr 18, 2013 at 11:09 AM, Eloy Durán notifications@github.com wrote:

Sounds good to me too. My two cents:

• @netbe I would prefer to let the block variant be and just implement the source directive and make it global. Do you have a real use-case for the block variant yourself atm?

* @irrationalfab My understanding is that with the source directive CP will still create a checkout in ~/.cocoapods, but will only use that one (or maybe even an ordered list of sources?) when resolving. If there is no source directive it would fallback to the current behaviour and thus work with existing projects. @netbe did I understand that correct?

Reply to this email directly or view it on GitHub: https://github.com/CocoaPods/CocoaPods/issues/982#issuecomment-16565602

[netbe]

@alloy yes this is correct: ordered list of source and fallback to current source if no directive declared. For the block part, I can spare it. it could have been a clear way to see which podfile is located where. For example:

source :cocoapods do Pod 'HockeySDK' Pod '...' end

source "myprivaterepo" do Pod 'mylib1' Pod 'AFNetworking' #my private copy ... end

This will help not guessing what source is taken, but ordered list of source make it work too.

[fabiopelosin]

For the block part, I can spare it. it could have been a clear way to see which podfile is located where.

This would clash with the blocks used to group the Pods of the various targets. See the example of http://docs.cocoapods.org/podfile.html if you don't use them. So I agree with @alloy that it would be better not to implement this syntax.

@netbe If you need any pointer on where to start with this feature I would:

• add the appropriate DSL in the podfile (example).
• please be sure to add the key that you'll use to store that information here, as the Podfile supports also serialization to YAML.
• The this method should clone the repositories if needed.
  • The logic used to update the master repo if needed is here
• Then you need to only use the sources specified by the Podfile if that is the case. The SorucesManager should be a good class where to start.

[alloy]

Cool, sounds awesome :+1:

[netbe]

@irrationalfab thanks for the help I'll start this soon enough

[fabiopelosin]

Awesome. Let me know if you have any question about the codebase.

[82times]

I would like to clarify the feature a bit. I'll describe what I'd like to be able to do.

The developers on my team all keep a local copy of our internal podspec repo at ~/.cocoapods/foo_internal and we like to use a mix of private internal pods as well as pods in the public master repo. I'd like to be able to specify podspec repos on a dependency-by-dependency basis.

As an example, one project may have a Podfile that requires:

• pods from the public master podspec repo
• pods from my company's private internal podspec repo
• pods that are private internal forks of pods that are in the public master podspec repo

A syntax like this should suffice:

pod 'RestKit', '1.2.3', :repo => 'master'
pod 'FooUIBar', '1.2.3', :repo => 'foo_internal'
pod 'Reachability', '1.2.3', :repo => 'foo_internal'

1. How does that sound to everybody? I feel like it would be general enough to solve most people's needs.
2. Where is a good place to start in the codebase to make a change like this?

@netbe let me know if this doesn't cover the kind of problem you're trying to solve. We can certainly collaborate!

[netbe]

@82times yes that's also exactly my need. I started on something following @irrationalfab instructions. However, I have been stuck on the Podfile hash_key definition that needed to be add in the dsl.rb and podfile.rb. Compared to other keys located in `podfile.rb``

  HASH_KEYS = [
      'target_definitions',
      'workspace',
      'sources',
      'generate_bridge_support',
      'set_arc_compatibility_flag',
    ].freeze

I don't know i should declare source or sources in HASH_KEYS to get the source key word available in the Podfile. Another temporary solution could be just use sources keyword for now.

So any help appreciated in that matter.

[fabiopelosin]

@netbe I think that you added the sources key as currently is not there in master. The hash keys is there to check that there are no issues with the implementation (misspelling a key) and to be able to access programmatically all the keys stored by the Podfile. It describes the YAML representation of the Podfile which is optimized to be readable. So after adding the new key that you want to support, you need to add an implementation similar to this.

If you do any work and encounter any issue feel free to submit a pull request and ask for support.

Btw, I think that the keyword should be source.

[fabiopelosin]

Regarding @82times proposal I don't think that we need to be so granular, otherwise we'll end up with feature creep as we already have to many options for the pod DSL directive.

Would it work for you to change the name of the Pod to include a modifier (like NetworkingPod -> NetworkingPodCompany)? The reasoning is that in any case you need to copy the spec and at that point changing the name would be trivial and make more clear to a reader of the Podfile that the dependency is a fork of the Pod.

I've also considered to use change the version (like 1.2.0 -> 1.2.0-compay, which is valid semver) and then require that exact version in the Podfile.

Thoughts?

[82times]

@irrationalfab I like the solution of NetworkingPod renamed to NetworkingPodCompany in the Podfile. Very good idea--a trivial change that makes it clear what is going on.

Part of what I need to do is convince the rest my team (and other teams of developers internally) that CocoaPods is good for us and safe to use. The crux here is that I need to explain that when a developer executes pod install it isn't going out into the wilds of Github to grab source when we don't want it to.

• In its current state, the best I can do is say "there isn't a FooCompanyUIBar pod out there in the master repo, so it'll use the one it finds internally in foo_internal." Very ambiguous.
• With the addition @netbe is working on, my explanation would be "it always looks in the foo_internal podspec repo first. It only looks in the master podspec if FooCompanyUIBar isn't in our internal podspec repo." Having that global source directive would help.
• With the proposal I made, I could point at the individual pod declarations and explain it as "for FooCompanyUIBar, it only looks in the foo_internal podspec repo; for NetworkingPod, it only looks in the master repo." This a very clear explanation of what happens, but I definitely understand the reluctance to add another creeping feature to the pod directive.

I'll talk this stuff over with my team later today and see what people think.

[fabiopelosin]

With the addition @netbe is working on, my explanation would be "it always looks in the foo_internal podspec repo first. It only looks in the master podspec if FooCompanyUIBar isn't in our internal podspec repo." Having that global source directive would help.

To be clear, we definitely want to include the support for the specification of the sources (the spec repos) and their priority.

With the proposal I made, I could point at the individual pod declarations and explain it as "for FooCompanyUIBar, it only looks in the foo_internal podspec repo; for NetworkingPod, it only looks in the master repo."

The simply fact that the name of the pod (or the version of the Pod) includes the company modifier should be a pretty strong guarantee that the pod will only by fetched from you private repos.

Regarding my previous post I think that applying a modifier to the version is a better solution as pod outdated would still allow to check for updates. With this proposal the dependency would be:

source "company-repo"
pod 'UIBar', '= 1.2.0-company'

Note that with this scenario, as well, is clear that the dependency used was approved by the team.

[82times]

Ah, yes, that's great about 1.2.3-company style version numbers--I didn't realize those were good with semantic versioning. I like that a lot.

Though my colleagues and I would prefer a more explicit, granular approach, we can work with the ordering of sources that @netbe is working on. Thanks everybody for working on this great project!

[fabiopelosin]

Though my colleagues and I would prefer a more explicit, granular approach, we can work with the ordering of sources that @netbe is working on.

For the record, I don't consider the approach using a the namespaced versions a hack but I consider it a robust solution. With this approach it is clear that the pod is based on a given version with some custom modifications. On the other hand depending on UIBar 1.2.0 from a custom repo fails to disclose that some modifications where done (i.e. even with that feature I would recommend to change the version). This last point is important for the CocoaPods implementation as well because for the tool UIBar 1.2.0 is always the same pod regardless of the source (i.e. it would fail to reinstall it if the spec-repo would change).

[netbe]

Hi,

Just to let you know I am still working on that, I wish I had more time and could go faster.

Anyway I did the first step on the cocoapods-core side with the dsl and podfile description https://github.com/netbe/Core/compare/master...feature/specs-sources (should I already start the pull request even if not finished?). I might go back to it to make the source element an object with the following attribute:

• repo
• branch
• tag
• alias

I realized that the source could be on a specific branch or tag (but maybe I am over-thinking it). THe alias could help me with the next part when I need to checkout the repo under ~/cocoapods. Which folder should I use? I thought about a md5 of the url but won't be handy if some people want to mess in that folder.

Let me know what you think @irrationalfab

[fabiopelosin]

should I already start the pull request even if not finished?

Please do, so we can follow the development and provide feedback early on.

I might go back to it to make the source element an object with the following attribute [...]

I think that this would unnecessarily complicate the repos logic and thus I would prefer to wait to see if there is any demand for it.

Which folder should I use? I thought about a md5 of the url but won't be handy if some people want to mess in that folder.

Maybe the syntax should include the name of the source:

source 'master', 'https://github.com/CocoaPods/Specs.git'

Ideally we should be able to derive the name from the URL, but I don't see any easy way to do it.

[82times]

I like having the name of the source in there. In practice, I would prefer to use only the name and not bother with the URL (i.e., it would locate it alongside the master podspec_repo in ~/.cocoapods).

[fabiopelosin]

In practice, I would prefer to use only the name and not bother with the URL (i.e., it would locate it alongside the master podspec_repo in ~/.cocoapods).

I was thinking about this alternative as well. This approach would be much simpler to implement but it would be less automatic. For example if an user might want to use a private source named MyAwesomeSource it would need to run the following command before running an installation with a Podfile using it:

$ pod repo add MyAwesomeSource https://example.com/MyAwesomeSource.git

@alloy Any thought?

[alloy]

@irrationalfab I get the point about the setup, however, I don’t see how including the URL here makes any sense with how it currently works. I.e. what if this URL is different than the repo actually uses? Do we add logic to check that and raise? Etc etc.

[alloy]

In general, I think people should not re-use a name like ‘master’. If they work at company X, then the repo should just be named something with X. I don’t think we need more logic for that.

[fabiopelosin]

I don’t see how including the URL here makes any sense with how it currently works.

I think that @netbe was working on cloning the repo automatically if needed and for that feature to work the source is needed.

In general, I think people should not re-use a name like ‘master’.

:+1: Agreed. Btw, adding a quick check to the pod repo add command might prevent some headaches.

[netbe]

Please do, so we can follow the development and provide feedback early on.

@irrationalfab done

The this method should clone the repositories if needed. The logic used to update the master repo if needed is here

When I am updating the source repos in the [analyzer]((https://github.com/CocoaPods/CocoaPods/blob/master/lib/cocoapods/installer/analyzer.rb#L151-L157) there is the case the repo has not been cloned yet. How can I check that?

cc @alloy , For the cloning, can/should I directly call the command line like

$ pod repo add MyAwesomeSource https://example.com/MyAwesomeSource.git

Also running the tests with the command: COCOA_PODS_DEPENDENCIES=local bundle exec rake

Is it normal some specs are disabled?

Is thee a way to execute only a given spec like
COCOA_PODS_DEPENDENCIES=local bundle exec rake spec SPEC=spec/unit/installer/analyzer_spec.rb Did not work for me?

Thanks

[fabiopelosin]

When I am updating the source repos in the analyzer there is the case the repo has not been cloned yet. How can I check that?

This is the implementation of the naive approach which raises if unable to locate the source (i.e. not automatic clone)

source_names = SourcesManager.all.map(&:name)
missing_sources = declared_sources - source_names
unless missing_sources.empty?
  raise Informative, "Unable to locate the source `#{missing_sources.join("\n")}`"
end

To check for the url of a source we use this approach.

For the cloning, can/should I directly call the command line like

Please don't do that. For this I think that this logic could be moved to a dedicated method in the SourcesManager.

I think that @alloy was proposing to just add the name of the source in the Podfile and to let users manage the sources manually using the command pod repo add from the command line.

Is thee a way to execute only a given spec like

$ bundle exec bacon spec/unit/installer/analyzer_spec.rb

the COCOA_PODS_DEPENDENCIES=local flag is a left over of the previous approach, currently we use the local git repo feature of bundler:

$ bundle config local.cocoapods-core ~/SomePath/CP/Core

[netbe]

@irrationalfab regarding the issue #1158 there won't be any ambiguity after this feature, the order of repo source in podfile will prevail

[netbe]

@irrationalfab trying to find out where the fetching of a podspec source code occurs in order to just override the podspec location (i.e. not using master first but the first source specified in podfile).

You suggested to look into Sourcesmanager but don't really see where now

[fabiopelosin]

The logic is the following:

1. The resolver uses the aggregate of the sources as provided by the SourcesManager
   resolver.rb#L56
2. The specifications are searched using the #search method of the aggregate, which just forwards it to the single sources
   resolver.rb#L191

Currently the Source:: Aggregate is just initialized with a dir it might be a good solution to modify it to support initialization with a list of sources dirs (and move the initialization with all the children of a dir to a dedicated class helper). Another solution might be to handle this situation in the resolver. Note that the SourcesManager is not used only by the installer but also by other subcommands like search and list which need to use an aggregate of all the sources.

[yvbeek]

Interesting feature.

Few questions:

• Isn't it dangerous to automatically check other repositories than master for podspecs? Simply adding a repository could change the code in your project.
• Why would you want to define more than one repository (source) per pod in the Podfile?

[netbe]

@Zyphrax

Isn't it dangerous to automatically check other repositories than master for podspecs? Simply adding a repository could change the code in your project.

I don't really get your question here, sorry.

Why would you want to define more than one repository (source) per pod in the Podfile?

One usage is too have a backup source in case github is down. You can fetch the specs in the secondary repo specs.

[yvbeek]

@netbe

The first question is based on what is described in #1158: "if the same (resolved) version of a Pod is present in two repos the first one in alphabetical order (name of the repo - i.e. name of the dir) wins".

In that case adding a repo could change your code. Because the new repo could also have a Podspec for the specific Pod but with a different version or codebase.

If you don't specify a source, would it still check all repo's (in alphabetic order) or would it only check master?

I'm just saying that automatically scanning through repositories to find a Podspec makes the whole process a bit unpredictable.

For example:

• master contains AFNetworking 1.1, 1.2, 1.3
• myrepo contains AFNetworking 1.1, 1.2

My podfile contains: pod 'AFNetworking'

If the master specs repo is unavailable I would end up with AFNetworking version 1.2, otherwise it would be version 1.3

---

• If GitHub is down, CocoaPods wouldn't be able to update the master repo, but existing specs would still be there and you could add a remote to the master repo if you need to update it manually from another source.
• I understand the need for private pods and a private specs repository, for that purpose you should be able to specify a repo other than 'master', for example: Pod 'MyPod', repo => 'mycorp'

Why would you want to specify multiple repositories for 'MyPod'?

I hope this makes sense :) or perhaps I'm stuck with the wrong idea of what you want to accomplish.

[fabiopelosin]

In that case adding a repo could change your code. Because the new repo could also have a Podspec for the specific Pod but with a different version or codebase.

There are two factors to take into account:

• pod install doesn't affect existing Pods, so if the original installation used AFNetworking 1.2 CocoaPods will fetch that one.
• We consider immutable a given version of a Pod. So if a private repos wants to tweak AFNetworking 1.2 it should namespace either the name or the version of the Pod. AFNetworking should be always the same library (i.e. AFNetworking as resolved in the 1.2 tag). If the users don't respect this point all ours assumptions start to leak.

So the disambiguation using the alphabetical order is just a note and I don't recommend to rely on it.

If you don't specify a source, would it still check all repo's (in alphabetic order) or would it only check master?

I vote to just check master. Although that would be a behavior change which might affect existing installations.

[yvbeek]

Thanks for the explaination irrationalfab.

What happens in case of a pod update?

• If Pod X in my Podfile has sources A and B
• I do a pod install, CocoaPods installs X version 1.5
• Later on I do a pod update, but now source A is unavailable and it uses source B
• What if source B only has version 1.4, would it effectively downgrade my Pod?

[fabiopelosin]

@Zyphrax indeed it looks like that it would downgrade with the current implementation. However that is a matter of just adding a check which aborts the installation in that case proposing to the user a manual downgrade in Podfile with pod install.

[kcharwood]

Any further plans for this feature?

[netbe]

@irrationalfab back on that, the tests crashes when running Running examples I did bundle rake spec

Trace:

Installing Pods
rake aborted!
/Users/netbe/.rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/source/git/git_proxy.rb:88:in ``': Operation not permitted - git branch (Errno::EPERM)
    from /Users/netbe/.rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/bundler-1.3.5/lib/bundler/source/git/git_proxy.rb:88:in `git'
    from /Users/netbe/.rbenv/versions/1.9.3-p392/lib/ruby/gems/1.9.1/gems/bundler-
...

Any ideas? probably not running the right command right?

[peterjenkins]

Is this feature planned?

[bilby91]

@pjenkins-cc +1

[CocoaPodsBot]

Issue has been confirmed by @neonichu

[michaelpeternell]

what has happened to this feature? will it ever be implemented, or has it been abandoned?

How about a simpler solution... What if I write in the Podfile something like:

pod "AFNetworking",
{
    "repo" => "https://github.com/dilbert/AFNetworking",
    "podspec" => "AFNetworking.podspec",
    "branch": "HEAD"
}

?

=> super easy, super explicit, super awesome ;)

[orta]

This ^ already exists:

pod "AFNetworking", :git => "https://github.com/dilbert/AFNetworking", :branch => "custom"

[michaelpeternell]

@orta oh, thanks :) I couldn't find it...

[Ashton-W]

:+1: I have a need to be able to never source the master specs (without messing with my cocoapods install)

[peterjenkins]

@Ashton-W Yes, would be great if you could use CocoaPods without ever having the master repo on your machine.

So +1, but there's probably another issue that's more appropriate for that discussion.

[Ashton-W]

Sorry I should have been more specific; I want the master repo on my machine, but for certain podfiles I want to use explicit spec repositories only. The current work around is to remove the master repo and replace it with a fake master. But this is destructive and global.

[fabiopelosin]

I have been thinking about this and I'm not sure that a list of sources at the top of the Podfile is the best solution. Using a fall through approach (use the first source that has the Pod) might lead to unexpected results. I'm considering a syntax like

pod 'private_repo:PodA'
pod 'master:PodB'

I.e. REPO:POD. Where if the name of the repo is omitted it is assumed to be master.

[segiddins]

@irrationalfab I like the list of sources, combined with a fork_of attribute

[82times]

I like this repo:pod idea as well. Much more explicit to a human reading it.

Jonathan

On Jul 20, 2014, at 3:22 PM, Fabio Pelosin notifications@github.com wrote:

I have been thinking about this and I'm not sure that a list of sources at the top of the Podfile is the best solution. Using a fall through approach (use the first source that has the Pod) might lead to unexpected results. I'm considering a syntax like

pod 'private_repo:PodA' pod 'master:PodB' I.e. REPO:POD. Where if the name of the source is omitted the repos should be master

— Reply to this email directly or view it on GitHub.

[Ashton-W]

I prefer something like ruby gems 'source', just because my use cases would need less modification and repetition.. But :+1: to any solution, go with the simplest to implement.

[netbe]

@irrationalfab the first approach with block would also remove ambiguity

source: "source1" do
pod "blabla"
...
end