Modify httpd lens so it can parse httpd configs recursively?

[examon]

@lutter

Hi, it is possible to modify httpd lens so it can parse httpd configs recursively?

As far as I now understand, augeas parses httpd configs that are specified in the "filter". But if I have non-standard/weird apache config location that is not specified in the "filter"? I guess I'm out of luck.

It would be very nice if I could somehow specify that I want to load file recursively based on the particular directive.

Example: Lets say we have the following line in our main httpd.config:`

Include /opt/config.conf

Right now, I would need to add "/opt/config.conf" into "filter" so I can get this parsed. In my use case, I'm extracting information from the httpd configs. The problem is, that I don't know ahead of time what to add to the "filter". I only know the location of the main httpd.config and not necessary location of all httpd configs.

Could I tell augeas something like: If you find Include/IncludeOptional directive somewhere, take its argument and parse it ("add it to the filter")?

With this way, I could just tell augeas the location of main httpd.conf config. All other includes would be recursively added and there would be much lesser change of missing some config files.

[lutter]

You can achieve that by manipulating what's in /augeas/load which is what ultimately governs what gets loaded (the filters/transforms from the lenses are used to prepopulate that)

After loading your initial files, you can go through them, find Include directives and put them into /augeas/load, e.g., to also load /opt/config.conf, you'd do this in augtool:

> set /augeas/load/Httpd/incl[last()+1] /opt/config.conf
> load

This will not reprocess files that were loaded the first time around - Augeas automatically suppresses unnecessary parsing based on the file's mtime.

To take this one step further, to find all the files that were loaded by the Httpd lens, do this:

> match /augeas/files//path[../lens = "@Httpd" or ../lens = "Httpd.lns"]
/augeas/files/etc/httpd/conf/httpd.conf/path = /files/etc/httpd/conf/httpd.conf
...

Then use each of those paths to find include directives:

> match /files/etc/httpd/conf/httpd.conf/directive[. = "Include"]/arg
/files/etc/httpd/conf/httpd.conf/directive[3]/arg = conf.modules.d/*.conf

And then resolve them to absolute file names and append those to /augeas/load/Httpd/incl as described in the first snippet.

Does that work for your situation ?

[examon]

Thank you. I've tested your approach last week and I think this will work fine in my situation.

[lutter]

Closing this for now - feel free to reopen if it should turn out this doesn't work for you.

[examon]

@lutter (I would like to reopen this issue, but GitHub will not let me.)

I've stumbled upon this one thing.

If I open interactive session and modify /augeas/load like this:

> set /augeas/load/Httpd/incl[last()+1] /opt/config.conf > load

I can see in /augeas/load/Httpd this:

/augeas/load/Httpd/incl[12] = "/opt/config.conf"

which is what I want. But when I exit interactive mode and run it again, /augeas/load/Httpd is in its default state.

Is there a way how to save the changs I did in /augeas/load, so I can load them in another augtool session?

I would like to run augtool -f mystuff.txt, do bunch of changes to /auges/load/* and then after some time, run augtool again with the previous changes already present/remembered.

[lutter]

Unfortunately, no there is no way to save what you have set up in /augeas/load. The best option I can think of is to script the setup by using one of the language bindings, like the Python or Ruby bindings.

[examon]

That's unfortunate. Anyway, I've played with Python bindings today, API looks nice and I can do what I need to do using it. Thx

[raphink]

There's actually easier ways to do this. In interactive mode:

transform Http.lns incl /opt/config.conf

does the /augeas/load manipulation automatically. When starting augtool, you can use the --transform|-t flag. Coupling it with --noautoload|-A is a good way to make it much faster:

$ augtool -At "Http.lns incl /opt/config.conf"

This is useful for testing, but of course it doesn't work with the Python bindings. You can however use the aug_transform call in the python bindings.

[examon]

Thank you @raphink . Your solution works well for me.

[examon]

@lutter

I have written a tool that uses your proposed set + match workflow to achieve recursive processing. It works fine and does everything I need.

I want to ask if it is feasible to extend the lens language, so it would be possible to specify right in the lens code something like: when we find Include directive, take its argument and add it to the filter (so we can achieve the same result as with the set + match)

I think that having this capability natively would be great.

[lutter]

I have to think about that a bit how that should work internally as issuing a load then will need to interleave loading files with matching and adding to the filter. In the language, this could take the form of an additional thing you could add to a filter, maybe incl_match <path expression> to trigger this behavior.