Parameters in Pillar and in result file nginx.conf have different order

[pavel-z1]

When we configure Nginx configuration file order for parameters is important. When I use Nginx-formula version 2.3.1 I receive different parameters order in my nginx.conf file comparing with Pillar parameters.

For example, Pillar:

pillars:
  nginx:
      # Use the official's nginx repo binaries
      install_from_repo: false

      lookup:
        package: nginx
        service: nginx
        webuser: www-data
        conf_file: /etc/nginx/nginx.conf
        server_available: /etc/nginx/sites-available
        server_enabled: /etc/nginx/sites-enabled
        server_use_symlink: true

        ### prevents rendering SLS error nginx.server.config.pid undefined ###
        pid_file: /var/run/nginx.pid

      service:
        enable: true

      server:
        config:
          user: nginx
          worker_processes: 6
          worker_rlimit_nofile: 1000000
          pid: /var/run/nginx.pid
          error_log: /var/log/nginx/error.log
          events:
            worker_connections: 5120
          http:
            server_name_in_redirect: 'off'
            server_names_hash_max_size: 10240
            server_names_hash_bucket_size: 1024
            default_type: application/octet-stream
            server_tokens: 'off'
            sendfile: 'on'
            tcp_nopush: 'on'
            tcp_nodelay: 'on'
            keepalive_timeout: 30

            gzip: 'on'
            gzip_vary: 'on'
            gzip_disable: '"MSIE [1-6]\."'
            gzip_proxied: any
            gzip_http_version: 1.0
            gzip_min_length: 1000
            gzip_comp_level: 6
            gzip_buffers: '16 8k'
            gzip_types: 'text/plain text/xml text/css application/x-javascript application/xml application/javascript application/xml+rss text/javascript application/atom+xml'

            limit_req_zone:
              - '$limit_ip zone=request_limit:10m rate=100r/s'
              - 'limit_req_zone: $servername_key zone=perserver:10m rate=5r/s'
            limit_req_log_level: warn

Nginx configuration file nginx.conf:

# Default nginx server configuration
#
# **** DO NOT EDIT THIS FILE ****
#
# This file is managed by Salt.

http {
    types_hash_max_size 2048;
    server_names_hash_max_size 10240;
    gzip_vary on;
    gzip_types text/plain text/xml text/css application/x-javascript application/xml application/javascript application/xml+rss text/javascript application/atom+xml;
    server_names_hash_bucket_size 1024;
    gzip_min_length 1000;
    tcp_nopush on;
    keepalive_timeout 30;
    gzip_http_version 1.0;
    limit_req_log_level warn;

    limit_req_zone $limit_ip zone=request_limit:10m rate=100r/s;
    limit_req_zone limit_req_zone: $servername_key zone=perserver:10m rate=5r/s;
    sendfile on;
    server_tokens off;
    default_type application/octet-stream;
    gzip_buffers 16 8k;
    gzip_comp_level 6;
    access_log /var/log/nginx/access.log;
    gzip_proxied any;
    server_name_in_redirect off;
    tcp_nodelay on;
    error_log /var/log/nginx/error.log;
    gzip_disable "MSIE [1-6]\.";
    gzip on;

}
pid /var/run/nginx.pid;
error_log /var/log/nginx/error.log;
worker_processes 6;
user nginx;
events {
    worker_connections 5120;
}
worker_rlimit_nofile 1000000;

How can be configured nginx formula to get the same parameters order in result files as in Pillar?

Salt version:

# salt-minion --versions-report
/usr/lib/python2.7/site-packages/salt/scripts.py:198: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date.  Salt will drop support for Python 2.7 in the Sodium release or later.
Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: 1.10.6
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.31.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: 1.2.5
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  6 2019, 09:12:32)
   python-gnupg: 0.4.3
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.7 Valery Bykovsky
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-962.3.2.lve1.5.25.8.el7.x86_64
         system: Linux
        version: CloudLinux 7.7 Valery Bykovsky

[aboe76]

@pavel-z1 this is not possible because of how python interprets yaml. Can't find the documentation for this but it has something to do with orderd dicts

some issue on saltstack about this: https://github.com/saltstack/salt/issues/12161

[pavel-z1]

Such behaviour cause issues on practice. This formula can't be used if need to configure map rules with variables, that will be used in limit rules.

For example, I need configure map in http section. I am configure pillar:

---
states:
  - nginx

pillars:
  nginx:
      snippets:
        limit_requests_per_domain.conf:
          # Exclude IPs from $servername_key variable that used in limit_req_zone
          - map $ip_exclude $servername_key:
              - 0: '""'
              - 1: $servername_firstcheck

      server:
        config:
          http:
            include:
              - '/etc/nginx/mime.types'
              - '/etc/nginx/snippets/limit_requests_per_domain.conf'

            limit_req_zone:
              - '$limit_ip zone=request_limit:10m rate=100r/s'
              - 'limit_req_zone: $servername_key zone=perserver:10m rate=5r/s'
            limit_req_log_level: warn

As result I will receive this configuration in nginx.conf:

# cat /etc/nginx/nginx.conf
# Default nginx server configuration
#
# **** DO NOT EDIT THIS FILE ****
#
# This file is managed by Salt.
.........

http {
    limit_req_log_level warn;

    limit_req_zone $limit_ip zone=request_limit:10m rate=100r/s;
    limit_req_zone limit_req_zone: $servername_key zone=perserver:10m rate=5r/s;

    include /etc/nginx/mime.types;
    include /etc/nginx/snippets/limit_requests_per_domain.conf;
}

With nginx error:

# nginx -t
nginx: [emerg] invalid number of arguments in "limit_req_zone" directive in /etc/nginx/nginx.conf:21
nginx: configuration file /etc/nginx/nginx.conf test failed

Variables $servername_key started to use before being defined in map.

[aboe76]

@pavel-z1, there are I think two options:

• put those other limit_req stuff in a different snippet. and then let the include order it.
• use tofs to manage whole files TOFS