IrpTransmogrifier

Parser for IRP notation protocols, with rendering, code generation, recognition applications.

This is a new program, written from scratch, that is intended to replace IrpMaster, DecodeIR, and (most of) ExchangeIR, and much more like potentially replacing "all" hand written decoders/renderers. The project consists of an API library that is also callable from the command line as a command line program. A GUI is possible, as well as the integration in GUI programs, like IrScrutinizer.

Acknowledgement

I would like to acknowledge the influence of the JP1 forum, both the programs (in particular of course DecodeIR, and the discussions (in particular with Dave Reed ("3FG") and Graham Dixon ("mathdon")). This work surely would not exist without the JP1 forum.

Background

This program can be considered as a successor of IrpMaster. The Irp parser therein is based upon a ANTLR3 grammar. The "new version" ANTLR4 is really not a new version, but a completely different tool, fixing most of the quirks that were irritating in IrpMaster (like the "ugliness" of embedding actions within the grammar, no left recursion, and much more). Unfortunately, this means that using version 4 instead of version 3 is not like updating a compiler or such, but "necessitates" a complete rewrite of the actions.

But that is of course not all: Both DecodeIR by John Fine and the Analyzer by Graham Dixon have shown to be essentially impossible to maintain and extend. Although DecodeIR and IrpMaster (through the data base IrpProtocols.ini) agree on most (but not all) protocols, this consists of two different, dis-coupled sources of protocol information. Finally, to be able to generate code from the IRP form is a natural wish.

Use cases

The reason for this project is not (just) to migrate IrpMaster to ANTLR4 -- version 3 is still operational, and no more broken than it was at the start. There are a number of interesting use cases for a nice parser/tree traverser:

Rendering

This use case corresponds to IrpMaster. It is probably to be implemented as a traversing of the parse tree of an IRP protocol, with numerical values assigned to the parameters.

In the present implementation, only one infinite repeat is allowed. (A realistic protocol, or use case, requiring more than one infinite repeat is not known to me.) Also, individual bitfields are restricted to 63 bits of length or less. (This is inherited from the use of Java's long type. It may be removed in the future.) With the exception of these restriction, the implementations should be complete, down to specification holes. There are also a few extensions to the IRP notation as described in the official documentation.

In IrScrutinizer, the word "generate" is used instead of "render". These words can be considered as synonyms (here).

Recognition

This use case corresponds to a dynamic version of DecodeIR: given a numerical IR signal, find the parameter/protocol combination(s) that could have generated the given signal. This is implemented by trying to parse the given signal with respect to the candidate protocols. It is thus very systematic, but comparatively slow.

It is not claimed that all protocols in the protocol data base are recognizable. Non-recognizable protocols are to be marked by setting the decodable parameter to false. To be recognizable, the IRP protocol should preferably adhere to some additional rules:

• The "+" form of repetitions is discouraged in favor of the "*" form.
• The width and shift of a Bitfield must be constant.
• The decoder is capable of simple equation solving (e.g. Arctech), but not of complicated equation solving.

Presently all but two protocols (zenith, nec1-shirriff, (bitfield width as parameter), fujitsu_aircon (would require non-trivial equation solving)) are recognizable. It is not guaranteed that new protocols automatically will be recognizable.

Loose matches, Guessing

Many captured signals are not quite correct according to their protocol. However, the firmware in a receiving device is often "forgiving", and accepts slightly flawed signals. It is thus desirable for a program of this type to find a near match, "guess", when an real match fails. This is known as loose mode, opposite of "strict" mode. For practical reasons, the loose mode is the default in the command line usage. The strict mode is enabled using the decode option --strict.

Code generation for rendering and/or decoding

For a particular protocol, generate target code (C, C++, Java, Python,...) that can render or decode signals with the selected protocol. As opposed to the previous use cases, efficency (memory, execution time) (for the generated code) is potentially an issue. This should be able to generate protocol renders for e.g. the Arduino libraries IrRemote, IrLib, and AGirs. At least in the first version, not all protocols describable by IRPs need to be supported. Not implemented in the first phase: Protocols with hierarchical bitspecs (rc6*, replay, arctech, entone), protocols with bitspec lenght as parameter (zenith, nec1-shirrif). Also default are not implemented, e.g. NEC1 has to be called with 3 parameters.

Two mechanisms are available: XML and Stringtemplate. The program does not come with an XSLT engine, so this has to be invoked independently on the XML export. The program just invokes the template, without caring what it does; if it generates a renderer or decoder. The user is instead governs this by invoking the style sheets or templates (s)he want using the --target (-t) option to the code sub subcommand. (For this reason, there is no --renderer or --decoder option to the code sub command.) It is also possible to pass target-specific parameters to the code generators using the --parameter (-p) argument.

Targets:

• Lircd.conf generation from IrScrutinizer. This is based on an XSLT-transformation (lirc.xsd) and generates an XSLT (version 1) file that can work with IrScrutinizer. Handling of definitions as well as expressions as bitfields not implemented, as well as a few other things (search for "omitted" in the above file), otherwise works. "90% complete", see this issue. To create: see (or execute) the shell script tools/generate-lirc.sh. In short, this generates the xml export, and then invokes xslt transformations on that xml file.
• Java. Essentially for testing. This is essentially working both for rendering and decoding, including a generated test rig (see the test project). Targets: java-decoder java-decoder-test java-renderer java-renderer-test. Not quite finished, see this issue.

Some possible future targets:

• Infrared4Arduino.
• IRremote
• Linux kernel modules in linux/drivers/media/rc (decoding only).

General code analysis

Not really connected to parsing IRP, but fits in the general framework. This has been inspired by to the Analyzer and the RepeatFinder in Graham Dixon's ExchangeIR (Java translation).

Protocol Data Base

The "ini"-file IrpProtocols.ini of IrpMaster has been replaced by an XML file, per default called IrpProtocols.xml. The XML format is defined by the W3C schema irp-protocols, and has the name space http://www.harctoolbox.org/irp-protocols. This format has many advantages in comparison with the simpler previous format, for example, it can contain embedded XHTLM fragments. It also can contain different parameters that can be used by different programs, for example, tolerance parameters for decoding. Arbitrary string-valued parameters are permitted. It is up to an interpreting program to determine the semantic.

There is also an XSLT stylesheet, which technically translates the XML to HTML, allowing for a user friendly reading of IrpProtocols.xml in the browser.

API documentation

Up-to-date API documentation, generated by Javadoc, is found here.

Integration in Maven projects

This project can be integrated into other projects using Maven. For this, include the lines

        <dependency>
            <groupId>org.harctoolbox</groupId>
            <artifactId>IrpTransmogrifier</artifactId>
            <version>1.2.3</version>  <!-- or another supported version -->
        </dependency>

in the pom.xml of the importing project.

Installation

Unpack the binary distribution in a new, empty directory. Start the program by invoking the wrapper (irptransmogrifier.bat on Windows, irptransmogrifier.sh on Unix-like systems like Linux and MacOS.) from the command line. Modify and/or relocate the wrapper(s) if desired or necessary. Do not double click the wrappers, since this program runs only from the command line.

Also, do not use the wrapper irptransmogrifier in the top top directory of the source tree. This is intended for development only, not by users.

Building from sources

The project uses Maven as build system. Any modern IDE should be able to open/import and build it (as Maven project). Of course, Maven can also be run from the command line, like

mvn install

Dependencies

The program depends on ANTLR4 (license), Stringtemplate, (license), as well as the command line decoder JCommander. (licensed under the Apache 2 license). When using Maven for building, these are automatically downloaded and installed.

Usage

Using from the command line, this is a command with sub commands. Before the sub command, common options can be given. After the command, command-specific options can be specified. Commands and option names can be abbreviated, as long as the abbreviation is unique. They are matched case sensitively, and can be abbreviated as long as the abbreviation is unambiguous.

The command

irptransmogrifier help --short

lists the subcommands. A command like

irptransmogrifier analyze --help

gives the usage for the subcommand analyze, while a command like

irptransmogrifier analyze --help

gives a possibly somewhat longer description for the subcommand analyze.

The subcommands are briefly described next.

Subcommand analyze

The "analyze" command takes as input one or several sequences or signals, and computes an IRP form that corresponds to the given input (within the specified tolerances). The input can be given either as Pronto Hex or in raw form, optionally with signs (ignored). Several raw format input sequences can be given by enclosing the individual sequences in brackets ("[]"). However, if using the --intro-repeat-ending option, the sequences are instead interpreted as intro-, repeat-, and (optionally) ending sequences of an IR signal.

For raw sequences, an explicit modulation frequency can be given with the --frequency option. Otherwise the default frequency, 38000Hz, will be assumed. If this option is given together with a Pronto type signal (which contains a modulation frequency), it is ignored.

Using the option --input, instead the content of a file can be taken as input, containing sequences to be analyzed, one per line, blank lines ignored. Using the option --namedinput, the sequences may have names, immediately preceeding the sequence. In both cases, the data is taken as IrSequences. IrSignals, with intro-, repeat-, and ending, are coerced into IrSequences.

In the Harctoolbox world, IR sequences start with a flash (mark) and ends with a non-zero gap (space). In some other "worlds", the last gap is omitted. These signal are in general rejected. The option --trailinggap <duration> adds a dummy duration to the end of each IR sequence lacking a final gap.

Input sequences can be pre-processed using the options --chop, --clean, and --repeatfinder.

The cleaner works according to this idea: The collected durations found in the sequence(s) are bundled into "bins" (disjoint intervals), according to absolutetolerance and relativetolerance. Every duration belonging to a bin is "close" (determined by those parameters) to the bin middle. All the durations within the bin are then replaced by the average of its members. It is thus not guaranteed that the distance between a duration and its relacement will be consistent with absolutetolerance and relativetolerance.

The input sequence(s) are matched using different "decoders". Normally the "best" decoder match is output. With the --all option, all decoder matches are output.

The options --statistics and --dump-repeatfinder (the latter forces the repeatfinder to be on) can be used to print extra information. The common options --absolutetolerance, --relativetolerance, --minrepeatgap determine how the repeat finder breaks the input data. The options --extent, --invert, --lsb, --maxmicroseconds, --maxparameterwidth, --maxroundingerror, --maxunits, --parameterwidths, --radix, and --timebase determine how the computed IRP is displayed.

Subcommand bitfield

The "bitfield" command computes the value and the binary form corresponding to the bitfield given as input. Using the --nameengine argument, the bitfield can also refer to names.

As an alternatively, the "expression" command may be used.

Subcommand code

Used for generating code for different targets.

Subcommand decode

The "decode" command takes as input one or several sequences or signals, and output one or many protocol/parameter combinations that corresponds to the given input (within the specified tolerances). The input can be given either as Pronto Hex or in raw form, optionally with signs (ignored). Several raw format input sequences can be given by enclosing the individual sequences in brackets ("[]").

For raw sequences, an explicit modulation frequency can be given with the --frequency option. Otherwise the default frequency, 38000Hz, will be assumed.

Using the option --input, instead the content of a file can be taken as input, containing sequences to be analyzed, one per line, blank lines ignored. Using the option --namedinput, the sequences may have names, immediately preceeding the signal.

In the Harctoolbox world, IR sequences start with a flash (mark) and ends with a non-zero gap (space). In some other "worlds", the last gap is omitted. These signal are in general rejected. The option --trailinggap <duration> adds a dummy duration to the end of each IR sequence lacking a final gap.

Input sequences can be pre-processed using the options --clean, and --repeatfinder.

The common options --absolutetolerance, --relativetolerance, --minrepeatgap determine how the repeat finder breaks the input data.

Subcommand help

This command list the syntax for the command(s) given as argument, default all. Also see the option --describe.

Subcommand lirc

This command reads a Lirc configuration, from a file, directory, or an URL, and computes a correponding IRP form.

Subcommand list

This command list miscellaneous properties of the protocol(s) given as arguments.

Subcommand render

This command is used to compute an IR signal from one or more protocols ("render" it). The protocol can be given either by name(s) (or regular expression if using the --regexp option), or, using the --irp options, given explicitly as an IRP form. The parameters can be either given directly with the -n option,or the --random option can be used to generate random, but valid parameters. With the --count or --number-repeats option, instead an IR sequence is computed,containing the desired number of repeats.

Subcommand version

Reports version number and license.

Usage message

Usage: IrpTransmogrifier [options] [command] [command options]
  Options:
    -a, --absolutetolerance
      Absolute tolerance in microseconds, used when comparing durations.
      Default: 100.0.
    -b, --blacklist
      List of protocols to be removed from the data base
    -C, --commentStart
      Character(s) to be considered starting a line comment in input and
      namedInput files.
    -c, --configfiles
      Pathname(s) of IRP database file(s) in XML format. Default is the one in
      the jar file. Can be given several times.
    --describe
      Print a possibly longer documentation for the present command.
    -f, --frequencytolerance
      Frequency tolerance in Hz. Negative disables frequency check. Default:
      2000.0.
    -h, -?, --help
      Print help for this command.
    -e, --encoding, --iencoding
      Encoding used to read input.
      Default: UTF-8
    -i, --irp
      Explicit IRP string to use as protocol definition.
    --logclasses
      List of (fully qualified) classes and their log levels, in the form
      class1:level1|class2:level2|...
      Default: <empty string>
    -L, --logfile
      Log file. If empty, log to stderr.
    -F, --logformat
      Log format, as in class java.util.logging.SimpleFormatter.
      Default: [%2$s] %4$s: %5$s%n
    -l, --loglevel
      Log level { OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL
      }
      Default: WARNING
    --min-leadout
      Threshold for leadout when decoding. Default: 20000.0.
    -g, --minrepeatgap
      Minimum gap required to end a repetition.
      Default: 5000.0
    --oencoding
      Encoding used in generated output.
      Default: UTF-8
    -o, --output
      Name of output file. Default: stdout.
    -O, --override
      Let given command line parameters override the protocol parameters in
      IrpProtoocols.xml
      Default: false
    -q, --quiet
      Quitest possible operation, typically to be used from scripts.
      Default: false
    --regexp
      Interpret protocol/decoder argument as regular expressions.
      Default: false
    -r, --relativetolerance
      Relative tolerance as a number < 1. Default: 0.3.
    --seed
      Set seed for the pseudo random number generation. If not specified, will
      be random, different between program invocations.
    -s, --sort
      Sort the protocols alphabetically on output.
      Default: false
    -t, --tsv, --csv
      Use tabs in output to optimize for the import in spreadsheet programs as
      cvs.
      Default: false
    -u, --url-decode
      URL-decode protocol names, (understanding %20 for example).
      Default: false
    --validate
      Validate IRP database files against the schema, abort if not valid.
      Default: false
    --version
      Report version. Deprecated; use the command "version" instead.
      Default: false
    -x, --xmllog
      Write the log in XML format.
      Default: false
  Commands:
    help      Describe the syntax of program and commands.
      Usage: help [options] commands
        Options:
          --describe
            Print a possibly longer documentation for the present command.
          -h, -?, --help
            Print help for this command.
          -l, --logging
            Describe the logging related options only.
            Default: false
          -c, --common, --options
            Describe the common options only.
            Default: false
          -s, --short
            Produce a short usage message.
            Default: false

    version      Report version and license.
      Usage: version [options]
        Options:
          --describe
            Print a possibly longer documentation for the present command.
          -h, -?, --help
            Print help for this command.
          -s, --short
            Issue only the version number of the program proper.
            Default: false

    list      List protocols and their properties.
      Usage: list [options] List of protocols (default all)
        Options:
          -a, --all
            Implies (almost) all of the "list xxx"-options.
            Default: false
          --check-sorted
            Check if the protocol are alphabetically.
            Default: false
          -c, --classify
            Classify the protocol(s).
            Default: false
          --describe
            Print a possibly longer documentation for the present command.
          --gui, --display
            Display parse diagram.
            Default: false
          --documentation
            Print (possible longer) documentation, as a dumb rendering of the
            HTML documenation.
            Default: false
          -d, --dump
            Print the IRP data base as DOC tree stringified, including initial
            XML comments.
            Default: false
          -h, -?, --help
            Print help for this command.
          --html
            Print (possible longer) documentation as HTML.
            Default: false
          -i, --irp
            List IRP form, as given in the database (unparsed, i.e. preserving
            comments and whitespace, not taking --radix into account).
            Default: false
          -m, --mindiff
            Compute minimal difference between contained durations.
            Default: false
          --name
            List protocol name, also if --quiet is given.
            Default: false
          -n, --normalform
            List the normal form.
            Default: false
          --prefer-overs
            List all protocol's prefer-overs, recursively.
            Default: false
          -r, --radix
            Radix of parameter output.
            Default: 10
          --stringtree
            Produce stringtree.
            Default: false
          --warnings
            Issue warnings for some problematic IRP constructs.
            Default: false
          -w, --weight
            Compute weight of the protocols.
            Default: false
          -x, --xml
            Like dump, but without XML comments.
            Default: false

    render      Render signal from parameters
      Usage: render [options] protocol(s) or pattern (default all)
        Options:
          -#, --count
            Generate am IR sequence with count number of transmissions
          -d, --decode
            Send the rendered signal to the decoder (for
            debugging/development).
            Default: false
          --describe
            Print a possibly longer documentation for the present command.
          -h, -?, --help
            Print help for this command.
          -m, --modulate
            Generate modulated form (EXPERIMENTAL)
            Default: false
          -n, --nameengine, --parameters
            Name Engine to use
            Default: <empty string>
          --number-repeats
            Generate an IR sequence containing the given number of repeats
          -P, --printparameters
            Print actual parameters values, for example by --random
            Default: false
          -p, --pronto, --ccf, --hex
            Generate Pronto hex.
            Default: false
          --random
            Generate random, valid, parameters
            Default: false
          -R, --raw-without-signs
            Generate raw form without signs.
            Default: false
          -r, --signed-raw
            Generate raw form.
            Default: false

    decode      Decode IR signal given as argument
      Usage: decode [options] durations in micro seconds, alternatively pronto
            hex
        Options:
          -c, --clean
            Invoke cleaner on signal
            Default: false
          --describe
            Print a possibly longer documentation for the present command.
          -R, --dump-repeatfinder
            Print the result of the repeatfinder.
            Default: false
          -f, --frequency
            Set modulation frequency.
          -g, --girr
            Generate output in Girr format (only)
            Default: false
          -h, -?, --help
            Print help for this command.
          -l, --ignoreleadinggarbage
            Accept decodes starting with undecodable pairs.
            Default: false
          -i, --input
            File/URL from which to take inputs, one per line.
          -k, --keep-defaulted
            In output, do not remove parameters that are equal to their
            defaults.
            Default: false
          -n, --namedinput
            File/URL from which to take inputs, one line name, data one line.
          -a, --all, --no-prefer-over
            Output all decodes; ignore prefer-over.
            Default: false
          -p, --protocol
            Comma separated list of protocols to try match (default all).
          --radix
            Radix used for printing of output parameters.
            Default: 10
          --recursive
            Apply decoder recursively, (for long signals).
            Default: false
          -r, --repeatfinder
            Invoke repeat finder on input sequence
            Default: false
          -s, --strict
            Require intro- and repeat sequences to match exactly.
            Default: false
          -T, --trailinggap
            Trailing gap (in micro seconds) added to sequences of odd length.
          --xslt
            File/URL name of XSLT transformation that will be applied to
            --input or --namedinput argument

    demodulate      Demodulate IrSequence given as argument (EXPERIMENTAL).
      Usage: demodulate [options] durations in micro seconds, alternatively
            pronto hex
        Options:
          --describe
            Print a possibly longer documentation for the present command.
          -h, -?, --help
            Print help for this command.
          -t, --threshold
            Threshold used for demodulating, in micro seconds.
            Default: 35.0

    analyze      Analyze signal: tries to find an IRP form with parameters.
      Usage: analyze [options] durations in microseconds, or pronto hex.
        Options:
          -a, --all
            List all decoder outcomes, instead of only the one with lowest
            weight.
            Default: false
          -b, --bit-usage
            Create bit usage report. (Not with --all.)
            Default: false
          -c, --chop
            Chop input sequence into several using threshold (in milliseconds)
            given as argument.
          -C, --clean
            Output the cleaned sequence(s).
            Default: false
          -d, --decoder
            Use only the decoders matching argument (regular expression, or
            prefix). Use the argument "list" to list the available decoders.
          --describe
            Print a possibly longer documentation for the present command.
          -R, --dump-repeatfinder
            Print the result of the repeatfinder.
            Default: false
          --eliminate-vars
            Eliminate variables in output form.
            Default: false
          -e, --extent
            Output the last gap as an extent.
            Default: false
          --fatgirr
            Generate Girr file in fat format. Inhibits all other output.
            Default: false
          -f, --frequency
            Modulation frequency of raw signal.
          -g, --girr
            Generate Girr file. Inhibits all other output.
            Default: false
          -h, -?, --help
            Print help for this command.
          -i, --input
            File/URL from which to take inputs, one sequence per line.
          --ire, --intro-repeat-ending
            Consider the argument as begin, repeat, and ending sequence.
            Default: false
          -I, --invert
            Invert the order in bitspec.
            Default: false
          -l, --lsb
            Force lsb-first bitorder for the parameters.
            Default: false
          -u, --maxmicroseconds
            Maximal duration to be expressed as micro seconds.
            Default: 10000.0
          -M, --maxparameterwidth
            Maximal parameter width.
            Default: 63
          --maxroundingerror
            Maximal rounding errors for expressing as multiple of time unit.
            Default: 0.3
          -m, --maxunits
            Maximal multiplier of time unit in durations.
            Default: 30.0
          -n, --namedinput
            File/URL from which to take inputs, one line name, data one line.
          -P, --parameterspecs
            Compute a dumb parameter specs in the IRP.
            Default: false
          -p, --parametertable
            Create parameter table.
            Default: false
          -w, --parameterwidths
            Comma separated list of either parameter widths or name:width
            pairs.
            Default: []
          --radix
            Radix used for printing of output parameters.
            Default: 16
          -r, --repeatfinder
            Invoke the repeatfinder.
            Default: false
          -s, --statistics
            Print some statistics.
            Default: false
          -t, --timebase
            Force time unit , in microseconds (no suffix), or in periods (with
            suffix "p").
          --timings
            Print the total timings of the compute IRP form.
            Default: false
          -T, --trailinggap
            Dummy trailing gap (in micro seconds) added to sequences of odd
            length.
          --validate
            Validate that the resulted protocol can be used for rendering and
            produces the same signal.
            Default: false
          --xslt
            File/URL name of XSLT transformation that will be applied to
            --input or --namedinput argument

    code      Generate code for the given target(s)
      Usage: code [options] protocols
        Options:
          --describe
            Print a possibly longer documentation for the present command.
          -d, --directory
            Directory in whicht the generate output files will be written, if
            not using the --output option.
          -h, -?, --help
            Print help for this command.
          --inspect
            Fire up stringtemplate inspector on generated code (if sensible)
            Default: false
          -p, --parameter
            Specify target dependent parameters to the code generators.
            Default: []
          -s, --stdirectory
            Directory containing st (string template) files for code
            generation.
            Default: st
        * -t, --target
            Target(s) for code generation. Use ? for a list.
            Default: []

    bitfield      Evaluate bitfield given as argument.
      Usage: bitfield [options] bitfield
        Options:
          --describe
            Print a possibly longer documentation for the present command.
          -h, -?, --help
            Print help for this command.
          -l, --lsb
            Output bitstream with least significant bit first.
            Default: false
          -n, --nameengine, --parameters
            Define a name engine for resolving the bitfield.
            Default: <empty string>
          --xml
            Generate XML and write to file given as argument.

    expression      Evaluate expression given as argument.
      Usage: expression [options] expression
        Options:
          --describe
            Print a possibly longer documentation for the present command.
          --gui, --display
            Display parse diagram.
            Default: false
          -h, -?, --help
            Print help for this command.
          -n, --nameengine, --parameters
            Define a name engine to use for evaluating.
            Default: <empty string>
          -r, --radix
            Radix for outputting result.
            Default: 10
          --stringtree
            Output stringtree.
            Default: false
          --xml
            Generate XML and write to file argument.

    lirc      Convert Lirc configuration files to IRP form.
      Usage: lirc [options] Lirc config files/directories/URLs; empty for
            <stdin>.
        Options:
          -c, --commands
            Also list the commands if the remotes.
            Default: false
          --describe
            Print a possibly longer documentation for the present command.
          -h, -?, --help
            Print help for this command.

For documentation, see http://www.harctoolbox.org/IrpTransmogrifier.html