Have a config file with 'personal' preferences.

[JdeH]

There are quite some command line switches now, and not everything is easily said in command line format.

So a config file will be added, most likely in the JSON format, since it's univeral, easy to parse and not as bloated and overly formalized as XML.

[pierrejean-coudert]

Regarding the config file syntax, I usually find the INI syntax easy tu use and readable. ( https://docs.python.org/3.5/library/configparser.html )

[JdeH]

Looks very straigthforward, but also somewhat limited. Yet another possibility is to make it plain Python, like I did for opy: see here.

In fact that is very powerful, since you can change settings according to an algorithm rather than one by one.

Don't know yet, but indeed a settings file of some sort would be handy.

[axgkl]

my 5 cents as well:

both are cool (yaml / json should be only output formatting options but not the source) and ini files are latest with systemd now also very well known and accepted in linux as declarative config format.

I like the python config also a lot since you can really help yourself adapt to environments and such via one and the same config in a cluster (which you have to invent in inis, like foo=$bar or even `minify="system.check_jdk_avail" and that is crap.

=> Maybe not think too long about one OR the other but make both accepted ?

Categories should be in, not just plain key values. (see ini) -> why not in python a format like:

[Unit]
Description=consul agent
Requires=network-online.target
After=network-online.target

[Service]
EnvironmentFile=-/etc/sysconfig/consul
Environment=GOMAXPROCS=2
(...)

as python, allowing imperative code as well:

cpus = os.popen.read('cat /proc/cpuinfo | grep processor  | wc -l')
class Unit:
    Description = 'consul agent'
    Requires    = 'network-online.target'
class Service:
    EnvironmentFile = '-/etc/sysconfig/consul'
    Environment='GOMAXPROCS=%s' % cpus

In the end the config parser gets something with categories and key values in for Jacques to process.

One indirection i.e. convention you often need in inis: When you want config values as non simple types, like list of js libs to include or templates for functions formatting - then you need to defer to outside files (like in the systemd example with EnvironmentFile, which it sources by convention).

• When you stick to declarations only both are a bijection.
• But should you need imperative stuff you use python and have the interpreter at hand to generate the final Category -> key -> value format as well.
• It is trivially mappable to environment exports, and easy to long cli options, see below
• Outside transcrypt parsers of the config run transcrypt --show-config=[json|yaml|ini|shell]

Regarding shell: We should not forget use cases of meta processes who run transcrypt, like build pipelines, on the same system but differently parametrized => they want CLI switches or exports, since should not have to generate the config file.

The order usually imho should be, expressed in pseudo code:

no_exists = 'cfg_n.a.'
def get_config_val(category, key, default):
      for cfg_in in cfg_cli, cfg_env, cfg_py, cfg_ini:
               val = cfg_in(category, key, no_exists)
               if val is not no_exists:
                   return val
     return default

with e.g. cfg_env sth like

def cfg_env(cat, key, default):
    return environ.get('TRANSCRYPT_%s_%s' % (cat.upper(), key.upper(), default)

In total we would have for e.g. category 'Minification' key 'jdk':

transcypt --minification-jdk=/usr/bin/jdk-1.7' ...

export TRANSCRYPT_MINIFICATION_JDK = "/usr/bin/jdk-1.7"

class Minification:
     jdk = "/usr/bin/jdk-1.7"

[Minification]
jdk = /usr/bin/jdk-1.7

[JdeH]

I have a strong preference for some Python format as standard. It isn't the simplest, but by far the most powerful, and we deal with programmers here. I would like to avoid doubling features. Keeping it lean and mean in every respect is the only way to keep it up to speed.

As for 'which format exactly', lets keep that open for a while. (Declarative components, imperative components, componentes involving dicts and resembling JSON like two drops of water, 'INI' -like components, components reading other formats...

For me this all needs some time to sink in. The above may look complex, but in fact we should keep it simple. Only have the freedom to implement clever solutions for complicated configuration problems that we don't yet anticipate.

Probably most parts of the config file will look much like an INI even if they subtly obey Python syntax.

[axgkl]

ups. did not see your post, sorry, then forget my comment except if you like it, then give me a go and I'll do it for you, think that stuff even I can do, i mean provide you in the transcrypt source with a get_config_val function. I know I will puke again about python 3's unicode crazyness (http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/) - but I'll do it.

or go the easy option and start with python only then we don't have all of this and can do what we want and your argument with the audience is very valid.

[JdeH]

You talk now about adding the INI-like approach with Python class based syntax?

[axgkl]

I think categories you should really really do, also in python not just plain lists of key values like in your example. Think about the tons of options when we do server sided components for transcrypt which @pierrejean-coudert also sees. And for categories, whats better readable than this class based format. IF we have categories than the ini would be trivial .

And further I think we need adhoc like you wrote for many reasons - so that long cli options and environment exports are also in the ballgame.

Edit: correction, not only long cli options, referring to the recursive x

[JdeH]

Using mere lists of values isn't what I want to advocate, it's indeed too limited. The main point is I think we should benefit from the power of Python syntax here. Class syntax makes it possible to combine INI semantics with Python syntax.

Following INI semantics for the straightforward parts (currently all there is, I guess) is a good idea, as @pierrejean-coudert points out it's well known and easy to comprehend.

About the class based syntax of @AXGKl: Am I right that this is processed just by CPython, no special purpose parsing or even special purpose execution involved.

So to interpret the config file you just run it through CPython and then find out what the class attribute's are by introspection of the class dict or something like that?

I like that very much. It is clean, simple and powerful.

@AXGKI: Could you give it a stab, making it a separate module. For now, the configfile goes in, and a set of configuration variables, isomorphic to the ones produced by commandArgs come out, encapsulated in e.g. and object of class Config.

So e.g. object config of class Config has its iconv attribute set to True, has its symbols attribute set to john$mary$deborah, has its esv attribute set to 6, etc.

Wouldn't want to claim any of your time for this, but if you would be willing to try: great!

If your module produces the config object, I'll integrate it with the command args processing. I think this is a very regular, sametime flexible solution. Of course in the long run, it may be possible to do more in the config file than from the command line. So while I propose to currently produce a config object having the same attribs as the commandArgs object, in the long run it may contain a superset.

[axgkl]

About the class based syntax of @AXGKl: Am I right that this is processed just by CPython, no special purpose parsing involved or even special purpose execution involved.

absolutely. Pure Python and the options of a category simply accessible like:

# cat cfg.py 
class Foo:
    bar = 'baz'
    i = 23
class Bar:
    j = [1,2,3]

# python -c 'm = {}; execfile("cfg.py", globals(), m); print [(cat, [key for key in dir(m[cat]) if not key.startswith("_")]) for cat in m]'

[('Foo', ['bar', 'i']), ('Bar', ['j'])]

[axgkl]

I'm happy and honoured to do it - but there is not such thing as a free lunch: You must allow me to accept also environement variables, if they are set (and I have no idea how that works in windows, there I would leave that open) ;-)

timewise: latest in a week or so, quite busy these days.

[axgkl]

but there is not such thing as a free lunch

hope that does not offend you was a terrible joke to say to someone who invested so much into a free gift just to make the world a better place. soooorry ;-)

[JdeH]

1. Don't hurry, I won't either. That's how I survive...
2. Also accepting environment variables sounds positive to me. As I understand you have a rare amount of experience in configuration matters, I'd like very much to see what this leads up to.
3. One thing I realize is that I'll have to keep concentrating upon the core to be able to keep up with both Python and JS developments. And I'll have to keep that core lean and as familiar as my backyard. So that's why I'd like things like configfile parsing (combined with evt. reading of envir vars) in a separate module (or set of modules). If you make it, I'd also like to get back to you if new options are added (or to others willing to adapt it). Can't do everything myself so I welcome the cooperation, given a clear separation of tasks.

As for the joke, hadn't taken it that way. And as for developing open source software: It is really rewarding to do things that matter. I've been paid for things that don't and it made me unhappy. So now I try to strike a balance. Programming isn't work for me, it is what swimming is for a fish. When I was 15 my father took me to the IBM 360/135 mainframe and taught me the basics. It is still my thing, one way or another.

BTW I've tried to assign this to you but somehow GitHub won't let me. Is your email address with GitHub still up to date?

[axgkl]

Programming isn't work for me

A painter has to paint. We are built to be creative and an open editor is like a canvas, just far less restricted regarding possibilities to express yourself.

Regarind you focussing on the core and familiar with it: Yeah, I'm already nervous for the metaclass PR...

Regarding the need to get back to me for new options:

Don't think so - did not look into your config machinery right now but it has to be in any case possible to add new understood keys and optional category in a python module and also a generation of a full python config with defaults and help strings within (actually just by omitting the underbar attrs)

Sth like module config_defs.py:

class Minification:
    '''Javascript Minification options''''
    _short, = 'xm'
    # minify: Run the minifier at each compile
    enable = True
    # jdk_location: Specify the location of your jdk executable or leave empty to use your system default.
    jdk = ""

and that contains all infos for CLI help and parsing of options in long and short format including defaults and also environment key and location of the key within the py config.

[axgkl]

oh and I forgot naturally also the output into javascript format so that the config is also in the browser (there is a nice tool around generating javascript from python btw.) but also output format into json, yaml, markdown, html and sphinx/ReST.

Lets see how complex the configuration options will get, maybe someone will ask for metaclass support sooner or later also for the part in the browser 👯

[JdeH]

The meta-info you're looking for is already there in ..\transcrypt\modules\org\transcrypt\utils.py, which you can import.

class CommandArgs:
    def parse (self):
        self.argParser = ArgumentParser ()

        self.argParser.add_argument ('source', nargs='?', help = ".py file containing source code of main module")
        self.argParser.add_argument ('-a', '--anno', help = "annotate target files that were compiled from Python with source file names and source line numbers", action = 'store_true')
        self.argParser.add_argument ('-b', '--build', help = "rebuild all target files from scratch", action = 'store_true')
        self.argParser.add_argument ('-c', '--complex', help = "enable complex number support, locally requires operator overloading", action = 'store_true')
        self.argParser.add_argument ('-dc', '--dcheck', help = "debug: perform lightweight consistency check", action = 'store_true')
        self.argParser.add_argument ('-da', '--dassert', help = "debug: activate assertions", action = 'store_true')
        self.argParser.add_argument ('-de', '--dextex', help = "debug: show extended exception reports", action = 'store_true')
        self.argParser.add_argument ('-dm', '--dmap', help = "debug: dump human readable source map", action = 'store_true')
        self.argParser.add_argument ('-dt', '--dtree', help = "debug: dump syntax tree", action = 'store_true')
        self.argParser.add_argument ('-ds', '--dstat', help = "debug: validate static typing using annotations", action = 'store_true')
        self.argParser.add_argument ('-e', '--esv', nargs='?', help = "ecma script version of generated code, default = 5. The symbol __esv<versionnr>__ is added to the global symbol list, e.g. __esv6__.")
        self.argParser.add_argument ('-f', '--fcall', help = "enable fastcall mechanism by default. You can also use __pragma__ ('fcal') and __pragma__ (\'nofcall\')", action = 'store_true')
        self.argParser.add_argument ('-g', '--gen', help = "enable generators and iterators. Disadvised, since it will result in a function call for each loop iteration. Preferably use __pragma__ ('gen') and __pragma__ ('nogen')", action = 'store_true')
        self.argParser.add_argument ('-i', '--iconv', help = "enable automatic conversion to iterable by default. Disadvised, since it will result in a type check for each for-loop. Preferably use __pragma__ ('iconv') and __pragma__ (\'noiconv\') to enable automatic conversion locally", action = 'store_true')
        self.argParser.add_argument ('-j', '--jskeys', help = "interpret {key: 'value'} as {'key': 'value'} and forbid {key (): 'value'}, as JavaScript does. Disadvised, since it's less flexible than the Python interpretation. Either follow Python semantics by using {'key': 'value'} explicitly if you want literal keys or use __pragma__ ('jskeys') and __pragma__ ('nojskeys') locally instead to make clear local deviation from Python semantics", action = 'store_true')
        self.argParser.add_argument ('-k', '--kwargs', help = "enable keyword arguments by default. In general this is disadvised, use __pragma__ ('kwargs') and __pragma__('nokwargs') locally instead to prevent bloated code", action = 'store_true')
        self.argParser.add_argument ('-l', '--license', help = "show license", action = 'store_true')
        self.argParser.add_argument ('-m', '--map', help = "generate source map", action = 'store_true')
        self.argParser.add_argument ('-n', '--nomin', help = "no minification", action = 'store_true')
        self.argParser.add_argument ('-o', '--opov', help = "enable operator overloading by default. In general this is disadvised, use __pragma__ ('opov') and __pragma__('noopov') locally instead to prevent slow code", action = 'store_true')
        self.argParser.add_argument ('-p', '--parent', nargs='?', help = "object that will hold module, default is window. Use -p .none to generate orphan module, e.g. for use in node.js. Use -p .user to generate module that has to be explicitly initialized by calling <modulename> (), e.g. after the full page has loaded")
        self.argParser.add_argument ('-r', '--run', help = "run source file rather than compiling it", action = 'store_true')
        self.argParser.add_argument ('-s', '--symbols', nargs='?', help = "names, joined by $, separately passed to main module in __symbols__ variable")
        self.argParser.add_argument ('-t', '--tconv', help = "enable automatic conversion to truth value by default. Disadvised, since it will result in a conversion for each boolean. Preferably use __pragma__ ('tconv') and __pragma__ (\'notconv\') to enable automatic conversion locally", action = 'store_true')
        self.argParser.add_argument ('-v', '--verbose', help = "show all messages", action = 'store_true')
        self.argParser.add_argument ('-*', '--star', help = "Like it? Grow it! Go to GitHub and then click [* Star]", action = 'store_true')

        self.__dict__.update (self.argParser.parse_args () .__dict__)

The last line above copies all relevant configuration attributes to the dict of this. In the same way you can get hold of them for the config parser, harvesting them from the commandArgs singleton that you import from utils.

Glad to hear you like your job as well! Must be, why else participate in an open source project.

[axgkl]

got it. we put all of the single letter ones into a category 'Main' I would say which the user can omit (in the long cli format) and the d ones into category 'Debug' then, with _short='d'. Happy to see that all shorts for the keys can be built by first letter of the long version, so we just need to specify long options within a category. If thats a restriction then you can force a different short within a category by specifying the key as a tuple of short and long.

[JdeH]

Indeed,

And then -l and -x to come. Let's forget about -xx for now (unless you have no coffee machine)

And... o, not all the short ones are the first letter of the long ones. -* isn't.

[axgkl]

BTW I've tried to assign this to you but somehow GitHub won't let me. Is your email address with GitHub still up to date?

email is up to date yes. Maybe you have to add me as a collaborateur? Maybe it would be good for Transcrypt marketing wise to have more people as collabs on github anyway, think you once said this as well.

[JdeH]

@AXGKl

What's the status of this? Should I add the label postponed? No intention to hasten you in any way, just want to manage expectations. Also of course feel free to abandon the effort (as I find myself doing regularly). Some things are surprisingly more difficult than could be anticipated.

Let me know how it stands and I'll label / close accordingly.

KR J

[axgkl]

Hi, thanks for patience. I wanted to make a general lib for it (class based config) but I have too many things in my head about what it should be able to support especially for a longer going project I also want to have sth like that but don't know really when I find time to continue with it.

Transcrypt's current way is anyway well structured regarding CLI, with the categories (like "d") and so I would say: Lets postpone the config file /environ var awareness for another, say, 3 months - and add new features in the existing way on the CLI.

Until then users can help themselves, e.g. by calling transcrypt with variables from the environment which they can source from config files, or directly realiasing transcrypt.

If I can't deliver sth until then we'll have to do sth lightweight, "only" for Transcrypt to be able to support user config files.

Sorry for postponing this .

[JdeH]

Don't be sorry! Most of us only got two hands and one head...

KR J