[WIP] multiproject via config objects

[svenevs]

Fixes #27.

• [x] Add new configuration values to allow for multiple projects to be specified
  • [x] Auto-generate exhale_projects when exhale_args is specified, framework will then just iterate exhale_projects
• [x] Create an actual config object rather than ridiculous globals() setting on the module
• [ ] (massive) give root / node objects access to project configs, updating module references to instance references
• [ ] at end, remove exclude for this branch from .travis.yml

[nikhilkalige]

@svenevs I am trying to use this tool with our internal codebase which has close to a 250 small libs and apps. So instead of asking doxygen to recursively generate xml's, I ran doxygen separately for every individual lib and app hoping to make it faster.

So, I used the monkey patch code linked in #27. During the generation process, the directives keeping trying to find the definition of an entry in the default_breathe_project doxygen folder.

Therefore I was trying to look into the generated rst files and found that :project: is not added to the breathe directives, which causes it to default to the default project.

I was looking at this pull request to see if any such changes were made here. Can you please help me understand how this is actually supposed to work.

I am a newbie in Exhale, so I am sorry if I misunderstood how things work.

[svenevs]

Hi @nikhilkalige!

I have not actually used the monkeypatch. It should definitely enable you to get Exhale to generate multiple projects as per the README instructions, I think the breathe_projects_source is crucial in how @mithro is able to get around me not having :project: calls.

However, if it is causing you problems, since you need to track that file locally here is what I suggest you do to further patch things to suit your needs.

1. Track the file exhale_multiproject_monkeypatch.py next to your conf.py and make sure to append to sys.path so you can import it. Making sure that you do this before declaring anything else.
2. At the bottom of his patch, after Exhale has generated everything but still inside the for project in breathe_projects loop, manually splice in your own :project: commands. It won't be pretty, but should at least enable you to move forward in the interim. It may look something like this

for project in breathe_projects:
    # leave everything at the beginning of the loop as is
    # this stuff is what actually gets past Exhale's obsessive
    # requirements enabling you to generate multiple APIs
    #  ....
    #
    # Generate the full API!
    try:
        exhale.deploy.explode()
    except:
        exhale.utils.fancyError("Exhale: could not generate reStructuredText documents :/")

    # This is where you come in
    containmentFolder = app.config.exhale_args["containmentFolder"]
    # traverse the *ENTIRE* containmentFolder subtree for '.rst' files
    # you need to open _EVERY_ file (ugh) and replace directives now

So use os.walk or whatever. But basically, you should be able to regular expression replace the contents of the files. Read the file in, now replace any line of the form ^(.. doxygen.*::.*)$. Now since you have the project variable, you can replace it. This is all very wishy-washy, but the idea is you want to turn something like

.. doxygenfunction:: namespace::foo

into

.. doxygenfunction:: namespace::foo
   :project: {insert project here}

Just be absolutely sure that there are exactly three spaces before your inserted :project: there since Exhale uses three spaces for other potential specifiers added and reStructuredText demands consistent whitespace.

I hope that is helpful in providing a bypass. Unfortunately this PR is not in a state to use in production, and won't be for a while :cry:

Please feel free to ask for clarifications if that is not clear enough. Replacing .. doxygenANYTHING by capturing it and inserting r"\1\n :project: {project}\n".format(project=project) is the name of the game. Doing this for every single file is unfortunate, but in reality the generation of the files by Exhale (and subsequent modification by you) is actually quite fast (tip file I/O is much faster in python 3). What takes a long time is Sphinx parsing these files / translating them into HTML.

[nikhilkalige]

Thank you for the explanation. I did not set up the breathe_projects_source variable. I did not understand how I am actually supposed to use it. Let me go over the breathe source again to see if I can understand it better.

[svenevs]

@nikhilkalige on a scale of 1-10 (lower means less comfortable) how comfortable are you with python? I've thought of a much better option for you that should be about as much effort as the above approach but won't require reopening and patching every file.

If you are willing I would like to describe how to do it so that you can implement it and ideally submit a pull request to the monkey patch repo. But if you are not very comfortable with python (particularly working with lists and dictionaries) then I should just do it myself, noting that I need to fix some other bugs and release 0.2.0 before I can do this for you.

[nikhilkalige]

I would say 10 :)

I tried by making the following changes, I added an extra argument called projectName in the configs, used that in the section where you write the directives, which I believe happens in two places. I added something like

write('   :project:{}'.format(self.project_name))

https://github.com/svenevs/exhale/blob/17ecbd9573efe70e59fabc73b66ca544fb24d310/exhale/graph.py#L2289 https://github.com/svenevs/exhale/blob/17ecbd9573efe70e59fabc73b66ca544fb24d310/exhale/graph.py#L2741

I you have a better way of doing it, I could make the changes for you..

[svenevs]

Hey @nikhilkalige sorry for the radio silence, #35 sucked the life out of me, Windows has such a stupid filesystem. It is 2018 right?...anyway.......

So yeah you've got the gist of what's going on here, but the monkeypatching for this can actually be much more straight-forward, and don't actually require any direct changes to exhale itself. Or at least I would really like to avoid injecting a bypass directly, since this PR is going to be murder to rebase as it is.

So there's this feature called customSpecificationsForKind explained here. All we need to do is get the ':project: {project}'.format(project=project) in there for every kind in utils.AVAILABLE_KINDS (feel free to write different code...that's pretty confusing looking xD), I don't know why I never thought of this before.

There is a potential snag, which is where things might get a little complicated / you responding with a 10 bodes well. The single biggest mistake I made with exhale was doing the configs stuff at module level. The reason this might become a problem is because it is used like this:

https://github.com/svenevs/exhale/blob/f93135fa02cba9db2d51697565bb7d8690efa4a8/exhale/graph.py#L2340

and the implementation of utils.specificationsForKind used there is

https://github.com/svenevs/exhale/blob/f93135fa02cba9db2d51697565bb7d8690efa4a8/exhale/utils.py#L405-L406

I am hopeful that this will be simple, since the function specifically does configs.customSpecificationsForKind (rather than at the top doing from configs import customSpecificationsForKind, which would mean we would have to reload the module...). So what you need to do in the monkey patch is basically just before doing exhale.apply_sphinx_configurations(app) you basically just need to do something like

default_exhale_args = dict(app.config.exhale_args)
if "customSpecificationsForKind" in default_exhale_args:
    orig_custom_specs = default_exhale_args["customSpecificationsForKind"]
else:
    orig_custom_specs = {}
for project in breathe_projects:
    # ... original stuff ...
    app.config.exhale_args["containmentFolder"] = os.path.realpath(app.config.exhale_args["containmentFolder"])

    # force the :project: in for every kind
    project_custom_specs = makeCustomSpecificationsForKind(lambda x: return [':project: {0}'.format(project)])
    custom_specs = deep_update(orig_custom_specs, project_custom_specs)
    app.config.exhale_args["customSpecificationsForKind"] = custom_specs

    # the rest e.g. exhale.configs.apply_sphinx_configurations(app)

Where you can just copy-paste deep_update from here:

https://github.com/svenevs/exhale/blob/f93135fa02cba9db2d51697565bb7d8690efa4a8/testing/utils.py#L16

This PR actually moves that into exhale core, but the testing/ folder is not installed so best to just paste it in the monkeypatch.

In short: by pure coincidence I thing my existing (ridiculous) customSpecificationsForKind will fit the bill. The deep_update might run into a snag on this:

https://github.com/svenevs/exhale/blob/f93135fa02cba9db2d51697565bb7d8690efa4a8/exhale/utils.py#L187

But for the most part, the overhead is really just respecting what users put in conf.py / making sure that the monkeypatch doesn't "overwrite". The last thing is to make sure that you deep_update for "class" and "struct" to include the default list here if the user did not provide them aka if orig_custom_specs is the empty dictionary:

https://github.com/svenevs/exhale/blob/f93135fa02cba9db2d51697565bb7d8690efa4a8/exhale/utils.py#L410

I hope this is not too scatter-brained, I wanted to send you the response but after the all out battle I just had with #35 I am kind of not lucid. Please let me know if I wasn't clear enough. I think that this will not take very much effort, but will require some testing. The good news is I believe it completely eliminates the need for breathe_projects_source, which would be great because like you I have no idea how it actually plays a role in this :no_mouth:

[svenevs]

@tkhyn simple question, the deep_update came from your arsenal, do you mind if it gets copied into the source of the monkeypatch? The license is dual ISC License or CC0 1.0 Universal. I don't think you would mind, but wanted to make sure since it's not really my code to endorse for those licenses.

[nikhilkalige]

@svenevs Let me take a look, and try to make the changes.

[tkhyn]

@svenevs regarding deep_update, no problem at all to include it anywhere you like under whatever license terms you like. Thanks for your work!

[svenevs]

no problem at all to include it anywhere you like under whatever license terms you like

WOOOOT! This PR already merged it into exhale core because it's such a useful method :) @nikhilkalige lemme know if you run into trouble / can't figure it out.

Thanks for your work!

You're welcome! And thank you again @tkhyn for all the help on the testing framework. I made a fixture all on my own the other day that generates a file / deletes it after the testing class is done, made me really happy.

Note: for the sake of your inbox(es), you may want to mute this thread since it's going to get messy over here.

[svenevs]

more stale than a chip you find under the couch