Use of double quotes (") instead of grave accents (`)

[Baha]

Some days ago, I started developing a parser with Prolog (using DCGs) and I decided to use your tokenizer as the lexer. Then, I wanted to use the traditional mode (swipl --traditional) to test it, but a pair of errors (from the tokenizer) appeared when compiling my parser.

These errors were due to grave accents (`), which are not compatible with previous systems of Prolog. I replaced them with double quotes ("), that I think are equivalent in this case. This makes the tokenizer compatible with previous versions of SWI-Prolog.

Anyways, I think the use of grave accents is non-standard when working with Prolog.

[ghost]

I apologize for the delay in my reply. I'm very glad to hear that you were able to put this little package to use, @Baha! I am also very grateful for your contribution towards improving it!

I am all for making this package more backwards-compatible (and for improving it in whatever other ways might prove useful). However, I don't want to sacrifice compatibility with SWI-Prolog 7+.

In §5.2 of the SWI-Prolog reference manual, the new use of double quotation marks is explained:

As of SWI-Prolog version 7, text enclosed in double quotes (e.g., "Hello world") is read as objects of the type string. A string is a compact representation of a character sequence that lives on the global (term) stack.

The use of back quoted text is also explained in that section:

Back quoted text (as in text) is mapped to a list of character codes in version 7.

Of course, for the purposes of DCG parsing in this package, we need these terms to be lists of character codes, not strings. If we simply switch the back quotes to double quotation marks, this package will be compatible with earlier versions of SWI-Prolog, but broken for SWI-Prolog 7+.

I propose, instead, that we opt for one of these two approaches. Either of them should allow us to achieve compatibility with both earlier and later versions of SWI-Prolog.

1. Get rid of the quotation marks all together by using code lists.

   In the package's current state, double quotation marks are only used in 2 places. We could simply replace both of these with the literal character code list, i.e., [32]. This would obviate the problem of quoting, at the expense of readability. The big advantage of this approach is that it would also move towards compatibility with ISO standards. The big drawback is that committing to this choice now might make it significantly more cumbersome to extend the package moving forward, as it is very unpleasant to work with lists of character codes.

   However, I don't think the slight gain in ISO compatibility is really worth considering for this package at this stage. Moreover, the package already depends upon the SWI-Prolog 7 string library and data structure for one of it's de-tokenization formats.

2. Use the back_quotes environment control flag to specify that in this module back quotes should be interpreted as denoting lists of character codes.

   I believe this will still allow you to run SWI-Prolog in the traditional mode, while we use the modern SWI-Prolog 7 syntax in this module. I have not had a chance to test it, but I believe it should suffice to simply place the following declaration after the module imports:

   :- set_prolog_flag(back_quotes, codes).

   If this works while maintaining the new standard back quotes syntax, would this satisfy your needs? Would you think of this as an improvement?

My inclination is quite clear, but I am still open to discussion if you have other ideas or a different opinion on the matter. Otherwise, I would very happily accept a PR implementing the second solution.

[Baha]

Thanks for your reply, @aBathologist ! You don't have to apologize for anything, I assume you are doing this on your free time 😊

I must confess I started to use Prolog in my project so that I could improve my LP skills (I only knew about the basics until now), and I hoped that other people could take advantage of this package when I submitted my PR.

Initially, I thought that the change I suggested was an improvement "for free", since I couldn't find any documentation about the back quote, but now this is clearly not the case. The traditional mode is not required for my project, so I'm not using it.

In fact, I'm already using the string library you mention in my project. Thus, both the package and my project depend upon SWI-Prolog 7+ now.

In summary, I suggest you to reject my PR and perhaps add a comment on the README file or the Wiki about versions.

[ghost]

I will probably implement the change I described. Thanks very much for bringing to light this potential problem with the package! I'm closing now as you requested :) (Albeit with a long lag...)