Please add a --vassals-include option

[matthijskooijman]

Currently, the --vassals-inherit option can be used to include a default configuration for all vassals. However, this uses the --inherit option, in which placeholders are not expanded, making it impossible to use a generic default configuration file.

Implementing a --vassals-include option, which works exactly like --vassals-inherit but passes --include to the vassal instead of --inherit should fix this, without adding significant complexity to uwsgi, AFAICS?

[unbit]

Using vassals-inherit is not very common since we have a consistent parsing order: http://uwsgi-docs.readthedocs.org/en/latest/ParsingOrder.html so basically it is the vassal that includes its template

By the way, an eventual patch will be no more than 3 lines, can you send an example config so i can check that effectively --vassals-include could be a solution ?

[matthijskooijman]

Not sure what you mean with your first comment about the parsing order. I understand that the vassal includes the template.

This is what I'm trying to achieve:

/etc/uwsgi-emperor/emperor.ini

[uwsgi]
# place timestamps into log
log-date = true

# vassals directory
emperor = /etc/uwsgi-emperor/vassals
vassals-include = /etc/uwsgi-emperor/vassal-defaults.ini

/etc/uwsgi-emperor/vassal-defaults.ini:

[uwsgi]
logto = /var/log/uwsgi/vassals/%(domain)-%(app-name).log
socket = /run/uwsgi/%(domain)-%(app-name).socket

/etc/uwsgi-emperor/vassals/foo.ini:

[uwsgi]
domain = foo
app-name = bar

When I use the existing vassals-inherit option, things don't work as expected:

/run/uwsgi/:
srwxrwxrwx 1 root root 0 Sep 17 11:16 %(domain)-%(app-name).socket

/var/log/uwsgi/vassals/:
-rw-r----- 1 root root 1406 Sep 17 11:16 %(domain)-%(app-name).log

If I add ini = /etc/uwsgi-emperor/vassal-defaults.ini to the foo.ini file and remove the vassals-include option, things work as expected, but I would rather not specify this ini file for every vassal I define.

/run/uwsgi/:
srwxrwxrwx 1 root root 0 Sep 17 11:19 foo-bar.socket

/var/log/uwsgi/vassals/:
-rw-r----- 1 root root 1405 Sep 17 11:19 foo-bar.log

Does this provide enough info?

If you decide to implement this, I'm happy to send a pullrequest for the corresponding documentation.

[unbit]

Ok, now is clear, and yes adding ini = in the vassal is what i was suggesting.

By the way the problem is that the vassals-inherit is specified before --ini (in the vassal command line).

So the solution is not moving to "includes" but changing the position of the vassals-inherit to be after the --ini

so the option should be --vassals-inherit-append

I will work on it today

[matthijskooijman]

I'm pretty sure you got it wrong there (after spending quite some time going through the sources to figure out why things weren't working the way I expected them to). Let me quote some pieces of code.

Let me quote some pieces of code.

static void uwsgi_emperor_spawn_vassal(struct uwsgi_instance *n_ui) {

...

        if (!strcmp(n_ui->name + (strlen(n_ui->name) - 4), ".xml"))
            vassal_argv[1] = "--xml";
        if (!strcmp(n_ui->name + (strlen(n_ui->name) - 4), ".ini"))
            vassal_argv[1] = "--ini";
        if (!strcmp(n_ui->name + (strlen(n_ui->name) - 4), ".yml"))
            vassal_argv[1] = "--yaml";
        if (!strcmp(n_ui->name + (strlen(n_ui->name) - 5), ".yaml"))
            vassal_argv[1] = "--yaml";
        if (!strcmp(n_ui->name + (strlen(n_ui->name) - 3), ".js"))
            vassal_argv[1] = "--json";
        if (!strcmp(n_ui->name + (strlen(n_ui->name) - 5), ".json"))
            vassal_argv[1] = "--json";
...

        counter = 3;
        uct = uwsgi.vassals_templates;
        while (uct) {
            vassal_argv[counter] = "--inherit";
            vassal_argv[counter + 1] = uct->value;
            counter += 2;
            uct = uct->next;
        }
        vassal_argv[counter] = NULL;

This shows that --inherit is already given after --ini on the vassal commandline (I've verified this using ps output).

static struct uwsgi_option uwsgi_base_options[] = {
...
    {"inherit", required_argument, 0, "use the specified file as config template", uwsgi_opt_load, NULL, 0},
    {"include", required_argument, 0, "include the specified file as immediate configuration", uwsgi_opt_load, NULL, UWSGI_OPT_IMMEDIATE},

Here you can see that --include is expanded immmediately (UWSGI_OPT_IMMEDIATE) when it is found (just like --ini, --xml, etc.). --inherit, on the other hand, is not expanded until uwsgi_configure is called, which is after expanding placeholders etc.

In other words, the problem here is not caused by the order on the commandline, it really is that the inherit option is designed not to expand placeholders (as also stated here: http://lists.unbit.it/pipermail/uwsgi/2012-February/003623.html)

On a related note, I'll be submitting a documentation pullrequest later that documents this behaviour of --inherit since it's not currently noted anywhere.

[unbit]

yes, you are right, all of the ini/xml/yml options are IMMEDIATE so they will always win over inherit. It looks like includes is the best choice as it is more consistent with the other config parsers

[matthijskooijman]

Just tested latest master and this works as expected!

[matthijskooijman]

Hmm, one more thought: Now, the --include and --inherit specified by the emperor are put after the regular --ini for the vassal config. This allows specifying defaults, but does not actually allow the vassal config to override values (since the --included default config will be processed last, overriding the vassal config values). Perhaps it would make sense to swap these? AFAICS none of the other stuff (referring to placeholders or custom options) will break, since that doesn't depend on the actual order of things in the config file (because of the well-designed config parser!).

[mathiasertl]

+1.

If I specify e.g. a uid in both the ini-file specified via vassals-include and in the ini-file, the uid in the include-file takes precedence. This is contrary to the documentation, which gives "vassals-defaults.ini" as a config name for the included file.

Starting the uwsgi instance with the reverse order does solve the problem:

# uid in vassal-defaults.ini take precedence
uwsgi-core --ini pypiserver.ini --include /etc/uwsgi-emperor/vassal-defaults.ini
# uid in pypserver.ini takes precedence (desired behavior)
uwsgi-core --include /etc/uwsgi-emperor/vassal-defaults.ini --ini pypiserver.ini