Add generators dynamically

[skizzay]

When writing the YCM generator, I figured it would've been cleaner to be able to register my generator dynamically (i.e. when my conanfile.py is loaded). I wasn't really sure the best way to do this, so I held off in favor of getting good functionality delivered. I think we should be able to do the following:

from conans import ConanFile, Generator

class MyConanFile(ConanFile):
    """
   yada yada yada and stuff
   """
   generators = "mine"

class LocalBuildGenerator(Generator):
   name = "mine"
   output_file = "awesome_build_system.out"

   @property
   def content(self):
      return "By far the coolest build system ever invented"

When we load the conanfile.py, we can scan for any classes inheriting from Generator and register it. This is similar to how we load the ConanFile implementation. This would allow users to create their own pluggable generators that can be reused without having to make a change to conan itself.

[memsharded]

+1. Good idea, generators should be able to be dynamically loaded. From https://github.com/conan-io/conan/pull/53 it is possible to import other files, so the generator can be imported from conanfile, and more easily reused among packages. I am concerned about the naming and reuse. If package A creates a generator called "make", and publishes it, it is the package end-users who must declare generators="make" to use such generator, and this might create a naming-collission issue.

It sounds to me like generators should be something decoupled from packages, maybe they could be added and dynamically loaded from other location as ~/.conan/generators, and use some user-prefixed-naming for them or the like to avoid name collisions. What do you think about this?

[skizzay]

I think the decoupling is a good idea. We basically have two approaches that can be taken. First, we could import anything found in a prescribed area (such as ~/.conan/generators). In this approach, we could either scan the entire directory for *.py files and load them dynamically or we could import it as a module and load any Generator classes exposed by the module. The benefit of this approach is the user just dumps a bunch of Generator classes (internals can be overwritten accordingly to remove name clashes) and conan does the rest. A nice side effect is that there is no real API to provide because conan is doing all of the work. The problem with this approach is that the set of generators are global for that user, which means "make" would have to have the same meaning for all projects.

The other approach is to simply have the conanfile.py register the generators through an api conan provides. The benefit of this approach is that by placing the logic in conanfile.py, we provide the user to pick and choose what is best for that project. The generators can be installed in any module that can be found in PYTHONPATH. This way, the user can have "make" mean one thing for one project and something completely different for another project. They could even have custom generators for that particular project. The problem with this approach is that we need to ensure that conan has a solid API to handle this (rarely does anyone create a brand new API and get it right on the first try). With conan still being rather young, I feel that the community would have some understanding and patience while it gets nailed down.

I favor the second because I feel that it's less opinionated and I usually prefer to form my own opinions on how things should work.

[memsharded]

Also the first approach has other problems too: a different workflow without a reference place to look for generators, no versioning scheme for generators. Yes, I think the second one is a better option, basically handle generators in a similar way to packages (but not exactly, so yes, that is why we'd need to define some new syntax/api). Also discovery/search, reference, upload/share mechanisms for generators are things that would be nice to have so lets think about this approach.

[lasote]

Yes, I prefer second option too. It could be hard to handle spread files in user home when a new conan version is released with new or updated generators. :+1: Great idea!

[skizzay]

I'll see if I can come up with something over the next couple of days. I think a quick prototype on a branch would be something to get a discussion going.

[bjoernpollex]

Since apparently installation using pip is coming at with 0.5, why not use setuptools' entry points for this?

They solve a lot of problems, including that of transitive dependencies (suppose I want to write a generator that uses other libraries). It would also allow users to manage sets of generators using virtual environments, and provide proper versioning.

[skizzay]

I'll have to train myself up setuptools a bit (I've used them as a client, never as a provider), but I agree that this is a better approach. From what I can tell, it gives us the flexibility to do everything as described above but in a more discoverable manner.

[memsharded]

While I like entry points, it is also true that not all people install it via pip. Actually no more than 40% of conan users, as there are also installers that do not even require python installed. All pure-consumers can just use conanfile.txt without seeing a line of python. I will try to come up with a branch with a proposal based on conan packages, so we can compare and discuss.

Conan packages already transitive, versioned, and discoverable in the web. It is true that they will not install all python dependencies lib for the generator itself in a native manner (you can always run a "pip install" inside system_requirements()), but this seems a bit premature, I prefer to work against strong real use cases, i.e. actually someone willing to write a generator that depends on an external python lib. Lets try to make a proposal in code and see how it looks.

[lasote]

Hi @skizzay! any progress with this? I know little time is available

[memsharded]

Yes, sorry on my side too. I have been extremely busy with other tasks, and I didn't have time to bring up any proposal to this issue.

[memsharded]

Hi, @skizzay!

I have resumed this task, because of the premake generator requested in #85, and I thought it would make a good example for this feature, so I will try that the premake generator is actually loaded dynamically. I am working in a proposal that includes managing the generators as packages, so they can be published, versioned, shared... I will come back soon with a PR.

[memsharded]

This has been released in 0.9, and so far works well. Closing the issue, but reopen or create new issue if any problem with the proposed solution. Thanks very much @skizzay for all your contribution. Best!