Support More Sources

[Queuecumber]

Sorry in advance if this has been brought up before, I did some searching and didnt find anything.

It would be really useful if the package downloader could support git or local file sources in addition to the conan server sources. This is similar to how npm handles dependencies.

For example maybe I could host my project on github and provide a link to clone the git repository, or I could host my project on a shared folder in my company and provide a path to that folder for conan to install from (location would still need the conanfile.py of course).

[requires] Boost/1.58.0@lasote/stable tinyxml2/3.0.0@lasote/vcpkg rapidjson/1.0.2-1@lasote/vcpkg git+https://github.com/someone/didntuploadtoconan.git#commit file:/home/myself/myprojectthatcantlegallybedeployedandITwontgivemeaserver

Something like that. If you are alright with this kind of thing I can look into implementing it

[memsharded]

Don't apologize, no problem, we are here to help, even if it were a duplicated issue!

Not sure if you are aware about how conan handle retrieval of source code, but basically, inside a package recipe you can implement the source() method, that can retrieve the source code of the package.

If you are talking about getting the recipes directly from other sources rather than conan remotes, there are several reasons we think it is not the best solution:

• Indexing and search. An important feature of a package manager is able to easily search and retrieve packages stored in it. With such direct download, search and indexing dissapears
• Package reference. Each package reference includes a username and a channel, that can be overriden and replaced. The "username" serves as a namespace, to avoid package name clashing (basically, allowing users to have their own package for lets say, Boost). Storing such username/channel inside the recipe itself breaks the possibility to reuse/copy/fork/rename recipes for you. It is very common that our users are getting a recipe from github, storing it under their own username in their own remote, without having to modify the recipe at all. These features are very important also for a security point of view, to avoid things as the one occurred to NPM a while ago.
• Package recipes are necessary to be able to store package binaries in the remotes. It is impossible to cache binaries not associated with a package recipe stored in a conan remote, as they basically belong to that recipe.
• There are many things in conan that depends on the registered or "exported" package recipe. The package recipe have manifests with the files and their checksum, which are checked against tampering and hacking. Hashes of package recipes are used to know if the stored binaries are out of date with the package recipe and thus have to be re-generated.
• Loading a package recipe in a conan remote is completely straightforward and done in seconds: git clone & conan export & conan upload, or just conan export && conan upload. Furthermore, if you don't have a remote server because IT wont give it (you have a big issue then, anyway), just do conan export, and it will work from local storage.

Everything in conan can operate locally, no need any remote at all. If for any reason you have a number of sources for conan package recipes that are not in a conan remote, you can always write a very simple script that will get each one of such package recipes and export them into your local. Of course you will have to provide a username/channel for them, in this case, I would say that local is a perfect name for the channel. This should be very simple, but please ask for help if you need it.

Getting philosophical, a package for C/C++ would be something very corporate, much more like Maven for Java than NPM for node/js. Maven doesn't fetch artifacts from github (I know that there is JitPack, but basically this is a remote that automates the creation of the artifacts and serves them as a Maven server). I don't like Java very much (even if I coded with if for a few years), and I might probably be wrong, but if they didn't implement it, they should also have good reasons against it.

Thanks very much for your feedback and offer to help!

[Queuecumber]

You have some good points, so within the bounds on what conan currently can do, how would you recommend I handle my current project:

find_package(INTERNAL_A REQUIRED)
find_package(OpenCV REQUIRED)
find_package(Boost REQUIRED COMPONENTS filesystem system regex)
find_package(TinyXML REQUIRED)
find_package(VLFeat REQUIRED)
find_package(INTERNAL_B REQUIRED)
find_package(MKL11 REQUIRED)
find_package(RapidJSON REQUIRED)

As far as I can tell I can only get Boost, tinyxml, rapidjson, and OpenCV publicly.

INTERNAL_A and INTERNAL_B are company proprietary.

VLFeat is open source but not packaged for conan, best case scenario I package and upload it. MKL is owned by Intel and I doubt it can be legally packaged, it requires a license per user anyway.

I would like over developers rolling on to my project to have an easy time of getting set up. One command to install the cross-platform dependencies would be great. What do you think my options are in this case (or what would you do in this situation)?

[memsharded]

Just a note of caution about public package recipes: they might not be perfect/complete or support all OS or compilers. Conan.io is not moderated, any user can upload any recipe at any time. If some recipe fails, it is typical to collaborate with the author to improve it. It is also typical that companies using conan are using their own conan_server, and hosting a copy of the packages, not depending to conan.io at all, so they are completely safe both from a security point but also from operations, so a network or conan.io outage won't affect them.

There is no need to get all dependencies from conan, you can both have conan dependencies and other (system package manager as apt-get, installed manually...) dependencies. So you can gradually adopt conan for those packages that are already available, and then step by step convert other dependencies to conan packages.

Of course, the best way would be to host a conan_server, it is very simple, and it has the obvious benefit that everybody in the team/company can use it for other packages/libraries, and also cache binaries, so you don't have to re-built them over and over, especially in Continuous Integration (of course Continuous Integration can and should be set-up to build package binaries from recipes. We do it all the time to generate the binaries for conan.io, in travis and appveyor)

There would be two strategies for the internal and other libraries:

• Add a conanfile.py and optionally, but convenient, a test_package to those projects. The conanfile will use the exports=* feature to copy the sources into the local cache. Getting them and building locally as packages is as simple as:

$ git clone <url>
$ cd dir
$ conan test_package

That would export the package recipe into the local conan cache, build the binary from it, and run a small test to ensure it finds headers and link correctly. Everything is local, nothing is uploaded to any remote. The package is ready to be used exactly as other packages retrieved from remotes.

• Have a folder elsewhere, in a separate repo, inside the consuming project... with subfolders for each library. Each subfolder would have a conanfile.py and a test_package. In this case, the conanfile will have a source() method that will git clone the source code from your local or private git server. The process would be very similar to the above:

$ # for each subfolder
$ cd subfolder
$ conan test_package

Then, I would write a "build.py" or "bootstrap.py" python script in the root of my project, that would basically automate the above steps for each of the dependencies that are not in any remote.

[memsharded]

Hi Max,

Did you have the chance to check the above? I think that if you agree there are some reasons for not supporting direct retrieval of recipes from other sources, we could close this issue. That doesn't mean that we cannot continue conversation in this same thread, or in any other channel if you prefer, just that this is not something to be fixed/implemented in the short term.

Thanks!