nginxinc / crossplane

Quick and reliable way to convert NGINX configurations into JSON and back.
Apache License 2.0
719 stars 87 forks source link

Crossplane is slow for big configurations #97

Open BobocIonutGabriel opened 3 years ago

BobocIonutGabriel commented 3 years ago

When creating(didn't checked loading yet) a big nginx configuration crossplane is slow Having a big nginx configuration with lots of files creating large ones can be very slow. Sometimes taking more than 30 seconds to create 4 files.

Cause After some profiling I discovered that what takes the longest is deciding if a string requires quoting. While this is a nice feature, to automatically quote it can get in the way if you know your configuration doesn't require that and you want it to run fast.

Solution The following patch adds a new parameter that toggles the quoting part if you know you don't need it. This makes the creation of the files almost 10x faster.

diff --git a/crossplane/builder.py b/crossplane/builder.py
index 049d224..8ffdc5e 100644
--- a/crossplane/builder.py
+++ b/crossplane/builder.py
@@ -50,59 +50,59 @@ def _needs_quotes(string):

     return char in ('\\', '$') or expanding

 def _replace_escape_sequences(match):
     return match.group(1).decode('string-escape')

-def _enquote(arg):
-    if not _needs_quotes(arg):
+def _enquote(arg, no_quote=False):
+    if no_quote or not _needs_quotes(arg):
         return arg

     if PY2:
         arg = codecs.encode(arg, 'utf-8') if isinstance(arg, unicode) else arg
         arg = codecs.decode(arg, 'raw-unicode-escape')
         arg = repr(arg).replace('\\\\', '\\').lstrip('u')
         arg = ESCAPE_SEQUENCES_RE.sub(_replace_escape_sequences, arg)
         arg = unicode(arg, 'utf-8')
     else:
         arg = repr(arg).replace('\\\\', '\\')

     return arg

-def build(payload, indent=4, tabs=False, header=False):
+def build(payload, indent=4, tabs=False, header=False, no_quote=False):
     padding = '\t' if tabs else ' ' * indent

     head = ''
     if header:
         head += '# This config was built from JSON using NGINX crossplane.\n'
         head += '# If you encounter any bugs please report them here:\n'
         head += '# https://github.com/nginxinc/crossplane/issues\n'
         head += '\n'

     def _build_block(output, block, depth, last_line):
         margin = padding * depth

         for stmt in block:
-            directive = _enquote(stmt['directive'])
+            directive = _enquote(stmt['directive'], no_quote)
             line = stmt.get('line', 0)

             if directive == '#' and line == last_line:
                 output += ' #' + stmt['comment']
                 continue
             elif directive == '#':
                 built = '#' + stmt['comment']
             elif directive in EXTERNAL_BUILDERS:
                 external_builder = EXTERNAL_BUILDERS[directive]
                 built = external_builder(stmt, padding, indent, tabs)
             else:
-                args = [_enquote(arg) for arg in stmt['args']]
+                args = [_enquote(arg, no_quote) for arg in stmt['args']]

                 if directive == 'if':
                     built = 'if (' + ' '.join(args) + ')'
                 elif args:
                     built = directive + ' ' + ' '.join(args)
                 else:
                     built = directive