Support parameter replacement in files

[bdbaddog]

This issue was originally created at: 2008-02-22 14:24:07. This issue was reported by: simpleton.

simpleton said at 2008-02-22 14:24:07

This might be useful by some people. I'm new to Scons, and I've been using autoconf. One features nice in autoconf is AC_CONFIG_FILES and AC_CONFIG_HEADERS, where a parameter can be declared, and then replaced. I've created a small class that does just this and perhaps if it is found useful by others it could be integrated into Scons. I find it useful for my project, for automatically generating the 'setup.nsi' scripts, keeping such information as the application name, version, etc, all in one place, instead of manually updating each item.

Well here it is what it looks like.

import os
import os.path
import re

class FileParameterReplacer:
    """File parameter substitution class
    With this class you can create substitution values like in autoconf
    in files, and apply them to the files.  The file parameter names
    are enclosed in '@' like in autoconf:

    Sample usage:

    fpr = FileParameterReplacer()
    fpr.Replace("name", "myProject")
    fpr.Replace("version", "0.2.1")
    fpr.ApplyFile("test.in", "test.out")
    fpr.ApplyHeader("config.in", "config.h")"""

    def __init__(self):
        self.values = { }

    def Replace(self, name, value):
        self.values[name] = value

    def FileHandler(self, mo):
        attr = mo.group(1)

        if attr in self.values:
            return self.values[attr]
        else:
            return ""

    def ApplyFile(self, sourceName, destName):
        sourceFile = None
        destFile = None

        try:
            # Create the directory if needed
            tdir = os.path.dirname(destName)
            if os.path.isdir(tdir) == False:
                os.makedirs(tdir)

            sourceFile = open(sourceName, "rU")
            destFile = open(destName, "w")
            subre = re.compile("@(.*?)@")

            # Read a line from the input, and replace any items matching
            # @NAME@ with the value of the item 'NAME" in the dictionary
            for line in sourceFile:
                destFile.write(subre.sub(self.FileHandler, line))
        finally:
            if sourceFile is not None:
                sourceFile.close()
            if destFile is not None:
                destFile.close()

    def HeaderHandler(self, mo):
        attr = mo.group(1)

        if attr in self.values:
            return "#define " + attr + " " + self.values[attr] + "\n"
        else:
            # The two strips are correct, one to get rid of excess
            # spaces, the other for the newline at the end
            return "/* " + mo.group(0).strip().strip("\n") + " */\n"

    def ApplyHeader(self, sourceName, destName):
        sourceFile = None
        destFile = None

        try:
            # Create the directory if needed
            tdir = os.path.dirname(destName)
            if os.path.isdir(tdir) == False:
                os.makedirs(tdir)

            sourceFile = open(sourceName, "rU")
            destFile = open(destName, "w")
            subre = re.compile("^#undef\\s+([a-zA-Z_][a-zA-Z0-9_]*)\\s*$")

            # Read a line from the input, and replace any items matching
            # '#undef name' and replace with '#define name value'
            for line in sourceFile:
                destFile.write(subre.sub(self.HeaderHandler, line))

        finally:
            if sourceFile is not None:
                sourceFile.close()
            if destFile is not None:
                destFile.close()

voiz said at 2008-02-23 00:54:25

I've done something similar, but with the following differences:

• It is implemented as a SCons builder;
• It manages dependencies (the target is only rebuilt if the input file or one of the substituted values change);
• It uses Python string substitutions instead of @-substitutions, so you would write "%(FOO)s" instead of @FOO@. This has two advantages:
  • It is faster: the substitution is implemented as a single call to the python % operator instead of relying on handler callbacks;
  • It allows replacing other things than strings (integers, floating-point numbers, ...).

Here's a very simple example usage: =================== BEGIN config.h.in ===================

/* Package name */
#define PACKAGE "%(PACKAGE)s"

/* Define if stdint.h is present. */
%(HAVE_STDINT_H)s

==================== END config.h.in ====================

================ BEGIN SConscript snippet ===============

import sconstools.subst
sconstools.subst.register (env)

env["PACKAGE"] = "foo"
if haveStdintH:
    env.Defined ("HAVE_STDINT_H")
else:
    env.Undefined ("HAVE_STDINT_H")

env.Subst ("config.h", "config.h.in")

================= END SConscript snippet ================

voiz said at 2008-02-23 00:58:59

Created an attachment (id=312) Implementation of the Subst builder

simpleton said at 2008-02-24 00:05:39

I like that a lot better, thanks. Especially support for more than just simple strings.

gregnoel said at 2008-02-24 10:44:35

How does yours differ from https://scons.org/wiki/SubstInFileBuilder written by Gary Oberbrunner? They seem to accomplish about the same thing. Could they be combined?

voiz said at 2008-02-24 11:11:38

I wasn't aware of the existence of Gary's SubstInFileBuilder when I wrote mine, but here are the differences I see:

• Gary's is much more flexible since it does environment variable substitution and allows arbitrary regular expression substitution;
• I haven't run any benchmark, but mine should be much faster (flexibility has a cost).

IMO, for most use cases, Gary's is probably waay overkill (I've been using SCons>for several years now, and I have never felt the need for that level of flexibility).

So, could they be merged? Not really. Which one (if any) should be included in SCons? That depends: we should run some speed benchmarks and if the difference is indeed significant, I would argue that mine is the better choice (or else include both). OTOH if the speeds are close, then Gary's is probably better.

gregnoel said at 2008-02-24 14:45:48

Gary, you should be looking at this.

The SubstInFileBuilder page should be updated to have this routine, or at least refer to it.

garyo said at 2008-02-25 08:23:24

I think Maciej's GSoC work in 2007 includes a modified and much improved version of my SubstInFile. So when that gets merged in, it will be part of SCons. I think (?) there's even doc.

As for Jérôme's version, it seems OK but I think the required format (Python "%" operator) is a little limiting. People often have .in files in a fixed format that need substitution.

simpleton said at 2008-02-25 12:09:15

A minor problem using the 're.sub' approach I've found. If I have something like:

build_dir = "build"
top_build_dir = os.path.join(os.getcwd(), build_dir)

SUBST_DICT = { "%TOP_BUILD_DIR%": top_build_dir }

If using this with re.sub, a file containing %TOP_BUILD_DIR% (like a windows nsis script) will not propererly contain it, the '\\b' will be processed by re.sub into 0x08. It must first be double by something like: top_build_dir.replace("\\", "\\\\")

voiz said at 2008-03-09 04:45:12

Looking closer, I have found the following issues with Gary's SubstInFile:

• It is non-deterministic! Depending on exactly what you put in SUBST_DICT, in what order and what Python version you are using, you may get different results. This is because it doesn't ensure that the replacements do not overlap, and because it uses the dict.__items__ function which returns the dictionary contents in random order;
• The substitution depends on the complete dictionary. So if you change a single value which is used in a single file, all the files will be rebuilt.

To address Gary's concern that my Subst builder is too limiting, I've made a new version which adds the possibility to define the substitution format using a single regular expression. So now, you can do:

# With python % substitution:
env.Subst ("output", "input")

# With automake-like variables: "@NAME@"
env.Subst ("output", "input", SUBST_REGEXP=re.compile (r"@(\w+)@"))

voiz said at 2008-03-09 04:46:12

Created an attachment (id=324) Implementation of the Subst builder with configurable format

gregnoel said at 2008-04-14 18:58:31

Bug party triage: Give to Gary to determine a comprehensive solution.

simpleton said at 2008-06-13 12:03:31

This is an additional idea. The substitution tool could be designed in such a way to support universal substitutions for any format:

Subst(target, source, pattern="...", found="...", notfound="...", replace = None)

pattern is a pattern to match, with a field 'var' such as '@(?P<var>\\w+)@' found is a replacement to use, such as '$2' $1 = name, $2 = value notfound is a replacement to use if the var is not found in env[var], of if it is None then the var must be found or it is an error replace can be used to do a simple replacement on the value of env[var] before substitution, such as replace = ('\\', '\\\\') to escape backslashes on one file but not another.

Simple usage:

env.Subst(os.path.join(top_build_dir, 'src', 'config.h'),
os.path.join(top_src_dir, 'src', 'config.h.in'), pattern = '^\\s*#undef\\s+(?P<var>\\w+)\\s*$', found='#define $1 $2', notfound='#define $1 0', replace = ('\\', '\\\\'))

env.Subst(os.path.join(top_build_dir, 'setup', 'setup.nsi'),
os.path.join(top_src_dir, 'setup', 'setup.nsi.in')

etc

Here is an example:

import re

def subst(source, pattern = '@(?P<var>\\w+)@', found = '$2', notfound = None,
replace = None, data = { }):
    def sfunc(mo):
        # pattern should contain a group (?P<var>...)
        try:
            var = mo.string[mo.start('var'):mo.end('var')]
        except IndexError:
            return '' # TODO: should probably abort

        # Get the name of that var
        try:
            value = str(data[var])
            use = found
        except KeyError:
            # value was not found in the dictionary
            value = ''
            use = notfound
            if notfound is None:
                # setting notfound to None means if the value is not
                # found it is an error, setting it to something else
                # allows a replacement for a not found value
                raise "not found"

        # If something to replace then replace it before substitution
        if replace is not None:
            value = value.replace(replace[0], replace[1])

        # First replace $1 with the name since the name should not contain
        # anything but normal letters/numbers/underscore
        result = use.replace('$1', var)
        result = result.replace('$2', value)

        return result

    r = re.compile(pattern, re.M)

    return r.sub(sfunc, source)

def subst_configure(input, replace = None, data = { }):
    return subst(input, replace = replace, data = data)

def subst_configh(input, replace = None, data = { }):
    return subst(input, pattern = "^\\s*#undef\\s+(?P<var>\\w+)\\s*$",
        found = "#define $1 $2", notfound = "#define $1 0", replace = replace, data = data)

# A 'config.h' style substitution
data = { 'HAVE_MATH_H' : 1, 'HAVE_STRDUP' : 1 }
input = "#undef HAVE_MATH_H\n#undef HAVE_STRDUP\n#undef HAVE_OTHER"

print subst_configh(input, data = data)

# A configure style substitition (the default) replaces @NAME@
data = { 'PACKAGE_NAME' : 'test', 'PACKAGE_VERSION' : '0.0.6' }
input = "Package: @PACKAGE_NAME@\nVersion: @PACKAGE_VERSION@"
print subst_configure(input, data = data)

# A 'config.h' style substitution, demonstrating replacement
data = { 'HAVE_MATH_H' : 1, 'HAVE_STRDUP' : 1, 'TOP_SRC_DIR' : '..\\..\\'}
input = "#undef HAVE_MATH_H\n#undef HAVE_STRDUP\n#ifdef DEBUG\n#undef
TOP_SRC_DIR\n#endif"

print subst_configh(input, data = data, replace = ('\\', '\\\\'))
# notfound may also commonly be '/* #undef $1 */

This has the advantage that it can be used for about any type of substitution.

simpleton said at 2008-06-13 12:08:18

Also the above would work with custom user defined values, such as env['APP_VERSION'] = '0.1' .. etc, as well as any values defined into env from configure-like tests such as checking for headers, libraries, functions

garyo said at 2008-06-13 13:21:15

I bow to the total power & configurability of this. I'd never expose this very sharp knife directly to users, but as simpleton shows, we can build pretty much whatever type of file-substituter we could want from this.

simpleton said at 2008-09-07 19:11:37

An additional item that would be nice in the above example would be to use an external subst dictionary if desired. If it is just used as:

env.SubstConfigure('setup.nsi', 'setup.nsi.in')

it would use 'env' as the dictionary, but if used as:

env.SubstConfigure('setup.nsi', 'setup.nsi.in', SUBST_DICT=other)

then it would use 'other' as the dictionary instead.

Additionally, the dictionary only needs to contain the named values without any decorators as the decorators can be specified in the advanced for of Subst, using the default match patterns for SubstConfigure and SubstConfigH, etc. Basically, the dictionary would be:

{ 'PACKAGE_NAME': 'abc', 'PACKAGE_VERSION': '1.2.3' }

and not:

{ '@PACKAGE_NAME@': 'abc', '@PACKAGE_VERSION@': '1.2.3' }

gregnoel said at 2008-12-26 13:29:20

Adjust triage of issues.

simpleton said at 2009-12-12 20:11:53

I've thought of a better way to make a generic substitution mechanism. I've currently implemented it myself and am working around to make it better (at least for me).

A generic substitution builder exists called SubstGeneric. It takes the target and source. The environment is the source of data for replacement. In addition a SUBST_PATTERN and SUBST_REPLACE are passed to the builder. SUBST_PATTERN is a regular expression pattern. It must contain at least a named parameter called "key", for use in building the dependencies. SUBST_REPLACE is a function that takes the environment object and the match object.

In addition to SubstGeneric, some extra helpers are provided via AddMethod: SubstFile (normal config-like substitution) and SubstHeader (for config.h type substitution, but a little bit different than autotools). It is up to the replacement function to determine how to behave, if a certain parameter must exist or not. For SubstFile, it would be an error if the parameter did not exist. For SubstHeader, it would take correct action.

SubstFile does simple replacement. "@key@" will be replaced with the value of key, using env.subst to fully expand it. "@@" will be replaced simply with a single "@".

SubstHeader is a little more difficult. It supports the following replacements:

#define @key@
#define @key@ default
#undef @key@

For any of them, if the key is found in the environment, it will expand to #define key value. If it is not, the behavior is as follows:

"#define @key@" will be replaced with "/* #define key */"
"#define @key@ default" will be replaced with "#define key default"
"#undef @key@" will be replaced with "#undef key"

The reason for using "@" still is to control what is and is not replaced even in the header input:

#ifdef SOMETHING
    // No substitution will occur here even if SOMETHING_ELSE
    // exists in the environment object (this is not a default
    // value if it is not found)
    #define SOMETHING_ELSE SOME_VALUE 
#endif

// These will be replace if found, or simple use the defaults provided
#define @VERSION@       0.0
#define @HAVE_STRDUP@   0
#define @HAVE_STRCAT@   0
#define @HAVE_BLAH@     0

simpleton said at 2009-12-12 20:15:14

Created an attachment (id=661) SubstGeneric builder with support for SubstFile and SubstHeader

simpleton said at 2009-12-12 22:59:37

Created an attachment (id=662) Improvement to generic subst tool

simpleton said at 2009-12-12 23:05:43

The second attachment improves upon the generic subst tool.

It fixes a small bug when scanning the source file for which keys are used for dependency tracking. If a variant directory is used and duplication is enabled for it, the file will not be copied over by at time the emitter function is called, and also str(s) returns the base name ("config.h.in") instead of the full path/name ("build/config.h.in" for "src/config.h.in"), so it uses the srcnode() path to read the file.

In addition, a missing key during SubstFile will now fail instead of silently substituting with an empty string. A missing key in SubstHeader will be replaced correctly based on the type of define.

simpleton said at 2010-03-19 11:59:34

I've done a little more work on this tool. Raw substitution is used when calling env.subst, since otherwise tabs and multiple spaces will be collapsed to a single space and it may be desired to substitute in the output file the complete form of the value. Also, for SubstHeader, I've made some filters available: str and chr. The str filter will escape the value and quote it., the chr does the same but single quotes and only the first character.

For this environment:

    env['WEBSITE'] = r'http://www.project.com'
    env['ROOTPATH'] = r'C:\Program Files\Something Else'

This template:

    #undef @WEBSITE@
    #undef @WEBSITE:str@
    #undef @ROOTPATH@
    #undef @ROOTPATH:str@

Would produce this output:

    #define WEBSITE http://www.project.com
    #define WEBSITE "http://www.project.com"
    #define ROOTPATH C:\Program Files\Something Else
    #define ROOTPATH "C:\\Program Files\\Something Else"

It may also be desired to have something similar for the general file substitution SubstFile. Also, for SubstHeader, it may be desired to have a way for the macro name to be different from the environment variable name, perhaps something like:

    #undef @WEBSITE>APP_WEBSITE:str@

This would use the environment variable named WEBSITE, but call it APP_WEBSITE in the header.

    #define APP_WEBSITE "http://www.project.com"

This can be done directly with SubstInFile:

    #define APP_WEBSITE "@WEBSITE@"
    #define APP_DISPLAY_NAME "@PACKAGE@ @VERSION@"

But this does not have the support for str/chr filters to escape some strings, and as each variable must be defined so there wouldn't be any support for #undef to be translated to #define KEY VALUE if it exists and remain #undef if it does not exist, etc. But I think all of that can be overkill as one can also do this:

    #define @PACKAGE:str@ ""
    #define @VERSION:str@ ""
    #define DISPLAY_NAME PACKAGE VERSION

simpleton said at 2010-03-19 12:01:08

Created an attachment (id=707) subst builder and simple tests

voiz attached subst.py at 2008-02-23 00:58:59.

Implementation of the Subst builder

voiz attached subst.py at 2008-03-09 04:46:12.

Implementation of the Subst builder with configurable format

simpleton attached subst.py at 2009-12-12 20:15:13.

SubstGeneric builder with support for SubstFile and SubstHeader

simpleton attached subst2.py at 2009-12-12 22:59:37.

Improvement to generic subst tool

simpleton attached subst_and_test.zip at 2010-03-19 12:01:08.

subst builder and simple tests

[mwichmann]

Don't have any idea what to do with all this work other than say - maybe drop into contrib?