XCDL template proposal

[ilg-ul]

Hi Kim (@kblomqvist),

I took a look at Jinja, I guess it is ok, to me the syntax looks similar to the Liquid metadata used by Jekyll. And although the documentation claims to be a generic template engine, to me it still looks more appropriate to html/xml pages.

Since my main interest is C/C++, I would like my template files to be as close as possible to C/C++ source files. Thus my current current point of view that for C/C++ template, the best idea is to embed metadata as C/C++ comments.

More details are documented at: http://xcdl.github.io/work/preprocessor/

Any comments on this?

[kblomqvist]

Hi

I haven't feel Jinja to be more appropriate to html/xml. Additionally I haven't thought that the template should be closely related to the target language. For example, this tamplate generates this from this. Here the template is not close at all with the output. Also the use case is completely different from html/xml, and Jinja was able to fulfill the requirements I think. Github user farceller used the template inheritance and macros even more heavily for the same purpose (to generate zinc.rs pioreg file from CMSIS-SVD).

I have also had plans to try Jinja (with Yasha) to generate Cortex-M processor-based "MCU header files" from CMSIS-SVD to C++ classes as discussed by Dan Saks in embedded.com. I don't know if it's a good idea to define memory-mapped devices as C++ classes but it would be fun to try.

To summarize. I'm not sure whether I have enough experience to say what's the best way. You have gone through quite many options and the preferences may get highly opinionated. So far I think that it's fine that the template looks completely different from the target language, and I have found Jinja to be highly extensible. For example, you can even fine tune the template language to fit better for your needs, as it was done for Latex.

[ilg-ul]

So far I think that it's fine that the template looks completely different from the target language

yes, for the examples you mentioned, the template is not really a template, it is more like a generator.

in my view, a template is a file containing mostly lines in the target language, and only a few lines which are generated.

this is my conclusion after spending hundreds of hours implementing and maintaining the GNU ARM Eclipse project templates, that started as simple files and ended absolutely hideous.

apart from personal preferences, the main advantage of using the C/C++ syntax for the C/C++ templates is that automatic source text formatters can be used to format the files, which will ensure the output files result formatted. I heavily use the Eclipse formatters, and I think they are a great feature.

for an example of such a template, here is a file that will hopefully be used in the next version of the Eclipse project templates:

https://github.com/gnuarmeclipse/plug-ins/blob/develop/ilg.gnuarmeclipse.templates.core/templates/common/src/main/main.xcdl-js.cpp

but generally you are right, the templates may look completely different from the target language.

I'll keep the Jinja syntax as a second choice, if I encounter cases when my JavaScript syntax will be less appropriate.

[kblomqvist]

The generator-like is a good description for my example templates.

If I understood correctly you would like to use the source text formatter to get correct indentation etc. The example you linked contains lot of comments, which don't get highlighted and makes the reading harder anyway. I have had a different approach in my mind as I would like to have an output previewer for the written template—similarly as there are Markdown previewers for Atom.

[ilg-ul]

I ran an imaginary test, and converted my template file to something that approximates the jinja/liquid syntax.

//
// This file is part of the GNU ARM Eclipse distribution.
// Copyright (c) 2014 Liviu Ionescu.
//
// ----------------------------------------------------------------------------
#include <stdio.h>
#include "diag/Trace.h"
{% if content == "blinky" %}

    {% if fileExtension == "c" %}
#include "Timer.h"
#include "BlinkLed.h"
    {% elif fileExtension == "cpp" %}
#include "Timer.h"
#include "BlinkLed.h"
    {% endif %} {# fileExtension #}
{% endif %} {# content #}

// ----------------------------------------------------------------------------
//
{% if content == "blinky" %}
    {% if syscalls == "none" %}
// Standalone {{ shortChipFamily }} led blink sample (trace via {{ trace }}).
// In debug configurations, demonstrate how to print a greeting message
// on the trace device. In release configurations the message is
// simply discarded.
//
// Then demonstrates how to blink a led with 1 Hz, using a
// continuous loop and SysTick delays.
    {% elif syscalls == "retarget" %}
// {{ shortChipFamily }} led blink sample (trace via {{ trace }}).
// In debug configurations, demonstrate how to print a greeting message
// on the trace device. In release configurations the message is
// simply discarded.
//
// To demonstrate POSIX retargetting, reroute the STDOUT and STDERR to the
// trace device and display messages on both of them.
//
// Then demonstrates how to blink a led with 1 Hz, using a
// continuous loop and SysTick delays.
//
// On DEBUG, the uptime in seconds is also displayed on the trace device.
    {% elif syscalls == "semihosting" %}
// Semihosting {{ shortChipFamily }} led blink sample (trace via {{ trace }}).
// In debug configurations, demonstrate how to print a greeting message
// on the trace device. In release configurations the message is
// simply discarded.
//
// To demonstrate semihosting, display a message on the standard output
// and another message on the standard error.
//
// Then demonstrates how to blink a led with 1 Hz, using a
// continuous loop and SysTick delays.
//
// On DEBUG, the uptime in seconds is also displayed on the trace device.
    {% endif %} {# syscalls #}
{% elif content == "empty" %}
    {% if syscalls == "none" %}
// Standalone {{ shortChipFamily }} empty sample (trace via {{ trace }}).
    {% elif syscalls == "retarget" %}
// {{ shortChipFamily }} empty sample (trace via {{ trace }}).
    {% elif syscalls == "semihosting" %}
// Semihosting {{ shortChipFamily }} empty sample (trace via {{ trace }}).
    {% endif %} {# syscalls #}
{% endif %} {# content #}
//
// Trace support is enabled by adding the TRACE macro definition.
// By default the trace messages are forwarded to the {{ trace }} output,
// but can be rerouted to any device or completely suppressed, by
// changing the definitions required in system/src/diag/trace_impl.c
// (currently OS_USE_TRACE_ITM, OS_USE_TRACE_SEMIHOSTING_DEBUG/_STDOUT).
//
{% if content == "blinky" %}
// The external clock frequency is specified as a preprocessor definition
// passed to the compiler via a command line option (see the 'C/C++ General' ->
// 'Paths and Symbols' -> the 'Symbols' tab, if you want to change it).
// The value selected during project creation was HSE_VALUE={{ hseValue }}.
//
// Note: The default clock settings take the user defined HSE_VALUE and try
// to reach the maximum possible system clock. For the default 8 MHz input
// the result is guaranteed, but for other values it might not be possible,
// so please adjust the PLL settings in system/src/cmsis/system_{{ CMSIS_name }}.c
//
    {% if fileExtension == "c" %}
// ----- Timing definitions ---------------------------------------------------
// Keep the LED on for 2/3 of a second.
#define BLINK_1S_TICKS  ({{ xxx }})
#define BLINK_1S_TICKS  (TIMER_FREQUENCY_HZ)
#define BLINK_ON_TICKS  (TIMER_FREQUENCY_HZ * 2 / 3)
#define BLINK_OFF_TICKS (TIMER_FREQUENCY_HZ - BLINK_ON_TICKS)
    {% elif fileExtension == "cpp" %} */
// Definitions visible only within this translation unit.
namespace
  {
    // ----- Timing definitions -----------------------------------------------

    // Keep the LED on for 2/3 of a second.
    constexpr Timer::ticks_t BLINK_1S_TICKS = $(xx);
    constexpr Timer::ticks_t BLINK_1S_TICKS = Timer::FREQUENCY_HZ;
    constexpr Timer::ticks_t BLINK_ON_TICKS = Timer::FREQUENCY_HZ * 2 / 3;
    constexpr Timer::ticks_t BLINK_OFF_TICKS = Timer::FREQUENCY_HZ - BLINK_ON_TICKS;
  }
    {% endif %} {# fileExtension #}
{% endif %} {# content #}

// ----- main() ---------------------------------------------------------------
// Sample pragmas to cope with warnings. Please note the related line at
// the end of this function, used to pop the compiler diagnostics status.
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunused-parameter"
#pragma GCC diagnostic ignored "-Wmissing-declarations"
#pragma GCC diagnostic ignored "-Wreturn-type"

int
main (int argc, char* argv[])
{
  trace_printf ("$" "(aaa)");
  trace_printf ("{{ aaa }}");
  trace_printf ({{ aaa }}, 0);
{% if content == "blinky" %}
    {% if syscalls == "retarget" %}
  // By customising __initialize_args() it is possible to pass arguments,
  // for example when running tests with semihosting you can pass various
  // options to the test.
  // trace_dump_args(argc, argv);
    {% elif syscalls == "semihosting" %}
  // Show the program parameters (passed via semihosting).
  // Output is via the semihosting output channel.
  trace_dump_args (argc, argv);
    {% endif %} {# syscalls #}

  // Send a greeting to the trace device (skipped on Release).
  trace_puts ("Hello ARM World!");

    {% if syscalls == "retarget" %}
  // The standard output and the standard error should be forwarded to
  // the trace device. For this to work, a redirection in _write.c is
  // required.
  // Send a message to the standard output.
  puts ("Standard output message.");

  // Send a message to the standard error.
  fprintf (stderr, "Standard error message.\n");
    {% elif syscalls == "semihosting" %}
  // Send a message to the standard output.
  puts ("Standard output message.");

  // Send a message to the standard error.
  fprintf (stderr, "Standard error message.\n");
    {% endif %} {# syscalls #}

  // At this stage the system clock should have already been configured
  // at high speed.
  trace_printf ("System clock: %u Hz\n", SystemCoreClock);

    {% if fileExtension == "c" %}
  timer_start ();

  blink_led_init ();

  uint32_t seconds = 0;

  // Infinite loop
  for (int i = 0;; i++)
    {
      blink_led_on ();
      timer_sleep (i == 0 ? BLINK_1S_TICKS : BLINK_ON_TICKS);

      blink_led_off ();
      timer_sleep (BLINK_OFF_TICKS);

      ++seconds;
      // Count seconds on the trace device.
      trace_printf ("Second %u\n", seconds);
    }
    {% elif fileExtension == "cpp" %}
  Timer timer;
  timer.start ();

  BlinkLed blinkLed;

  // Perform all necessary initialisations for the LED.
  blinkLed.powerUp ();

  uint32_t seconds = 0;

  // Infinite loop
  for (int i = 0;; i++)
    {
      blinkLed.turnOn ();
      timer_sleep (i == 0 ? BLINK_1S_TICKS : BLINK_ON_TICKS);

      blinkLed.turnOff ();
      timer.sleep (BLINK_OFF_TICKS);

      ++seconds;
      // Count seconds on the trace device.
      trace_printf ("Second %u\n", seconds);
    }
    {% endif %} {# fileExtension #}
  // Infinite loop, never return.
{% elif content == "empty" %}
  // At this stage the system clock should have already been configured
  // at high speed.
  // Infinite loop
  while (1)
    {
      // Add your code here.
    }
{% endif %} {# content #}
}

#pragma GCC diagnostic pop

// ----------------------------------------------------------------------------

I'm not sure the readability was improved, probably just slightly.

but the complexity of the code to process this template is now significantly higher.

in the JavaScript template case, all that is needed is to remove the //@XCDL prefixes, and to brace all other lines as out("... line ... "), and here we have a JavaScript that can be executed as is.

the above code would look like:

out('//')
out('// This file is part of the GNU ARM Eclipse distribution.')
out('// Copyright (c) 2014 Liviu Ionescu.')
out('//')
out('// ----------------------------------------------------------------------------')
out('#include <stdio.h>')
out('#include "diag/Trace.h"')
    if (content==="blinky") {
out('')
        if (fileExtension==="c") {
out('#include "Timer.h"')
out('#include "BlinkLed.h"')
        } else if (fileExtension==="cpp") {
out('#include "Timer.h"')
out('#include "BlinkLed.h"')
        } // fileExtension
    } // content

for jinja, a true parser must be implemented, with lots of commands.

for my templates, the commands are actually JavaScript statements, and any expression valid in JavaScript can be used in the template conditional code.

do you think the eventual increase in readability is worth the effort to parse the jinja syntax?

[kblomqvist]

do you think the eventual increase in readability is worth the effort to parse the jinja syntax?

I was more comparing generated-like vs. template-like. Though in Jinja you would use {# comment #} instead of // comment, which can be highlighted differently. So readability gets a bit better because there wouldn't be //@XCDL like template commands embedded with rest of the comments.

{% if content == "blinky" %}
    {% if syscalls == "none" %}
    {# Standalone $(shortChipFamily) led blink sample (trace via $(trace)).
        In debug configurations, demonstrate how to print a greeting message
        on the trace device. In release configurations the message is
        simply discarded.

        Then demonstrates how to blink a led with 1 Hz, using a
        continuous loop and SysTick delays. #}
    {% elif syscalls == "retarget" %}

    ...

//@XCDL     if (content==="blinky") {
//@XCDL         if (syscalls==="none") {
// Standalone $(shortChipFamily) led blink sample (trace via $(trace)).
// In debug configurations, demonstrate how to print a greeting message
// on the trace device. In release configurations the message is
// simply discarded.
//
// Then demonstrates how to blink a led with 1 Hz, using a
// continuous loop and SysTick delays.
//@XCDL         } else if (syscalls==="retarget") {

    ...

Or if the C-comments should end into output:

{% if content == "blinky" %}
    {% if syscalls == "none" %}
// Standalone {{ shortChipFamily }} led blink sample (trace via {{ trace }}).
// In debug configurations, demonstrate how to print a greeting message
// on the trace device. In release configurations the message is
// simply discarded.
//   
// Then demonstrates how to blink a led with 1 Hz, using a
// continuous loop and SysTick delays.
    {% elif syscalls == "retarget" %}

    ...

[ilg-ul]

yes, those C++ comments should end into the output, and should be indented to the left.

I edited the long post to use the correct jinja comments and variable substitutions.

I agree that it is more readable, but only slightly; instead we lose the possibility to use the Eclipse formatter and processing this syntax is significantly more complicated. is it worth?

generated-like vs. template-like

for generated-like, it was planned to reverse things, and use plain JavaScript files, without any comments, and pass the few C/C++ fragments with out('...').

I doubt jinja can express more than javascript can.

[ilg-ul]

just checked, and apparently there are several Django-Jinja / Ruby-Liquid template engines in node.js, so processing this syntax in the new xcdl tool wouldn't be an obstacle.

the drawback is that initially I will have to manually format/indent the code.

however, as good as the Eclipse formatter may be, the true solution would be to integrate a formatter into xcdl and run the template output though it. but this can be added to a future version.

[kblomqvist]

the drawback is that initially I will have to manually format/indent the code.

To be honest manual format/indentation is very tedious job. I used countless of hours to just get indentation working for this example. During the process I thought that there were some deviation/bugs in Jinja's white space control. To get good looking output I also had to define trim-blocks=True and lstrip-blocks=True. If I recall correctly these were False by default.

[ilg-ul]

white space control

yes, I also encountered this while using Jekyll, white space control in this template syntax requires special attention.

To be honest manual format/indentation is very tedious job.

fully agree, that's why I insisted on encoding the template metadata in C++ comments, to make use of the Eclipse formatter.

but, to be honest too, this is more a workaround than a solution. the true solution is a separate formatter, to be used after the template engine.

given the countless node.js packages, it might be one that formats C/C++; if not right now, it'll certainly be one in the near future.

so, based on your suggestions, I'll probably run some tests with the existing liquid template engines (for example shopify-liquid), and decide on one; later, if I'll find a source formatter, I'll add it too.

[kblomqvist]

but, to be honest too, this is more a workaround than a solution. the true solution is a separate formatter, to be used after the template engine.

Agreed.

if I'll find a source formatter, I'll add it too.

Are you aware of Artistic Style? It can even be called from Python but for Node I didn't find a wrapper.

In Eclipse do you use built-in formatters or some others? I do not have much experience with formatters. All I have done was to set up astyle with git pre-commit hook, because it was insisted in a certain project I was working at a company.

[ilg-ul]

Are you aware of Artistic Style?

yes, I used it many years ago, but after started using Eclipse intensely, I no longer needed it.

calling it from node via a wrapper is not exactly convenient, because this requires an extra binary to be installed by the users, and the binary must be made available for all platforms; a native JavaScript one would be preferred.

In Eclipse do you use built-in formatters

the built-in formatter. the CDT one has some oddities/bugs, but, as usual, there are workarounds. from Neon.2 probably a new version of the indexer will be added, with a new formatter, hopefully with less oddities.

[ilg-ul]

@kblomqvist, I have the first version using shopify-liquid templates fully functional, and I'm quite happy with the result.

the template engine implementation is quite young, but the author seems interested in maintaining and improving it. the code currently suffers from the classical white space control problem, which hopefully will be added soon.

the two templates I use are at:

https://github.com/xcdl/xcdl-js/tree/master/assets/templates

as for the code to use them, it is still work in progress, as I'm slowly learning JavaScript; the classical node.js code style is quite simplistic, and all asynchronous functions use callbacks, which is not very nice. the good part is that ES6 is a completely new toy, with classes, promises, generators, etc and I'll restructure my code around these features.

so... thank you for the hint!