Thoughts about making the config file more user friendly.

[bokov]

Background: config file requires deployers of DataFinisher to write SQL fragments. Many of them can be replaced with references to re-usable boilerplate. This ticket, as other tickets outside our internal tracker at this time, is more for documentation purposes than for a specific task. It will be closed when this feature is implemented.

Pseudocode:

def riget(self,value,recur=3):
  """ self.shortcuts is a dictionary attribute that we will have 
       added to the ConfigParser class already
       self.refrxp is a compiled regular expression attribute that
       we will have added to the ConfigParser class already
  """
  if recur == 0: return(value)
  recur -= 1
  if value in self.shortcuts.keys():
    return self.riget(self.shortcuts[value],recur)
  hits = self.refrxp.search(value)
  if len(hits) == 0:
    return value
  replacements = [self.riget(self.get(ii[section],ii[option]),recur) for ii in hits]
  for ii in replacements: self.refrxp.replace(value,ii)
  return(value)

Sample config sections (assuming a self.shortcuts of {'AAA':123,'BBB':'foo:optC'}):

[foo]
# to prevent df.py from trying to interpret section as part of the rules
in_use = -1
optA = simple value
# substitute in value from bar section
optB = ${bar:optX}
# substitute in value from same section
optC = the zebra returns a ${foo:optA}
# valid but useless... will return a literal ${foo:optD}
optD = ${foo:optD}

[bar]
in_use = -1
optX = another simple value
# multiple substitution, some of it several levels deep (but less than 3)
optY = the giraffe also returns a ${foo:optA} and ${foo:optB} and then ${foo:optX}
# cannot combine shortcuts with other stuff, returns 123
optZ = AAA
# returns 'the zebra returns a simple value'
optZZ = BBB

...there, got it out of my head, hopefully I can get some peace now.

[bokov]

Bleah. Tried to write my first doctest, but can't get it to be recognized by doctest.testmod(). On the other hand, if I manually execute these commands, the results match the ones shown here and are what they should be.

        """
        This one iterates over all the options in the 'foo' section (except 'in_use')
        >>> [cnf.rxget(cnf.get('foo',ii)) for ii in cnf.options('foo') if ii != 'in_use']
        ['simple value', 'another simple value', 'the zebra returns a simple value', 'the water buffalo likes to cuddle with the gazelles']
        Ditto for 'foo_subfoo'. Note the circular reference for foo_subfoo:opte, on which rxget()
        eventually gives up and returns the raw value.
        >>> [cnf.rxget(cnf.get('foo_subfoo',ii)) for ii in cnf.options('foo_subfoo') if ii != 'in_use']
        ['${foo_subfoo:optE}', 'the water buffalo likes to cuddle']
        Ditto for 'bar'
        >>> [cnf.rxget(cnf.get('bar',ii)) for ii in cnf.options('bar') if ii != 'in_use']
        ['another simple value', 'the giraffe also returns a simple value and another simple value and then another simple value']
        Testing out shortcut replacement
        >>> [cnf.rxget(ii) for ii in cnf.shortcuts]
        ['left join (select pn,', 'left join (select distinct pn,sd,', 'left join (select pn,sd,']
        """

Long story short, you can now interpolate multiple places in the same string having the pattern ${foo:bar} where foo is any config section and bar is any option in that section. You can also interpoalate a few selected magic words, or shortcuts.