Feedback on some possible improvements (e.g. generate type def files from Typescript)

[PerryBhandal]

(Excuse the title change, hit the Comment button too quickly...)

Here are a few ideas for improvements. I'm not sure which of them I'll pursue, but feedback would be welcome.

• Provide a translator for typescript definition files so that Python equivalents can be generated for use with static type checking tools and autocomplete. I'd likely implement this as a separate project and then provide some tool (potentially a command line flag in Transcrypt? Maybe a separate command line app that can be included in Transcrypt's repo?) to make it easier to use with Transcrypt.

• Support for string formatting using %s, %d, etc. Was the exclusion of those (in favor of {} and the .format method) an intentional design decision and is there any reason the addition of % based string formatting could (should?) not be added? I can take care of implementation. Here is a gist outlining the issue:

https://gist.github.com/PerryBhandal/de0c3ae4eb17e8b80d57cdc68e80cebd

Ideally having both methods of formatting would be nice. Failing that, can a check be added for that at compile time? I've always used % based string formatting so I overlook it here and there, so having it caught at compile time would save me time and frustration.

[geatec]

Hi

On the first point: That would mean that using libraries written in TypeScript would be able to be used from Transcrypt in a typesafe manner, if I understand you well. I think that would be very useful, since the number of TypeScript libs will probably grow strongly in the near future. There's even word of TypeScript typing becoming part of JavaScript, although I don't know if these rumors have substance.

Anyhow I consider it useful, and if there would be a separate tool that could be bundled with Transcrypt or pip installed to be used much like is the case with mypy now, I would welcome that. In that case indeed it would be good to include a command line switch for it, making it easy to use in an integrated way. Would be nice if the tool besides being stand alone, could also be imported and used as a module, so that we don't have to go through the OS and start another Pyhon instance.

Should you embark on such an undertaking, probably some examples should be provided. I can be of little help here, since I don't know much about TypeScript.

On the second point: C-style string formatting (with the % etc.) has been left out in favor of .format () to keep Transcrypt lean. Since that's almost the same thing, TS now also supports fstrings. However the mini string-formatting language was not implemented since it would bloat the runtime to much. Maybe some of it can be added later as an optional module, governed by a command line switch. Generating a warning for %s, %d etc. seems like a good idea to me. I'll add it as a separate improvement issue.

KR Jacques

[JdeH]

Sorry, I answered from my company account. I've you'd like to work on the Typescript def file translator, let me know, then I'll assign it to you.

[PerryBhandal]

Hi Jacques, Appreciate the feedback.

Your interpretation of my first point is correct. I agree it could be quite useful, I'll begin looking into it and post a final outline once I have a clear picture of how it will be implemented.

[PerryBhandal]

A few other pain points:

1) Adding support for the @property decorator. Am I correct in my understanding that is not currently supported? From what I can see you tend to use explicitly defined setter/getter methods and then reference them in a call to property(). I generally prefer to use the decorator approach.

2) In general, having an easy way to feed code into Transcrypt that lets me see where my code does not conform to Transcrypt's specs would be quite helpful. An example is the string formatting case from my original post, the @property decorator in the above point is another. The use case I see for this is that I want to be able to quickly validate whether some existing code I have works with Transcrypt. So I can take some old library I have written and feed it in and it can point out "you're using printf formatting here, you're using a @property decorator here" etc and I can just quickly make changes for those cases. The right approach here seems to involve handling it directly when parsing the AST, but with most of the immediate cases I can think of a regex may be able to handle it. I wonder if you have any thoughts on how feasible and valuable this would be, and how best it could be implemented.

In many ways the compiler could be sufficient to deal with this case, but there are a few reasons why I think a separate tool/command line option is warranted. First, given that many of these errors can be detected without requiring that a compile succeeds, it will make it easier to quickly do a once over on a project. If we rely on the compiler to detect those errors then it's a bit more involved/tedious as I can't verify that until my project is in a compiling state (even if a successful compile is not necessary to detect said errors). Presumably a compile also terminates at the first error it sees, so we would not be able to see all incompatibilities at once, we would have to address them one by one (if the incompatibilities result in compile errors/terminations, instead of a simple warning).

3) Add a watch option to the command line executable. The way I have my workflow currently set up is I have a script called compile.py which just uses pyinotify to watch for file changes and re-run the compile. Seems like it could be easy to implement this directly in Transcrypt's command line app.

4) Earlier when I attempted to use a relative path for my input file I got an error. I ended up changing my script so I simply CD'd into the appropriate dir and compiled there. That's not a big issue (and it's possible that it was an issue on my end unrelated to relative paths). I'll take a look tomorrow and post more details on the specific exception.

[JdeH]

OK for now I'll assign it to you, but of course feel free to abandon. Just to prevent duplicate effort.

[JdeH]

1. Support for @property. Indeed that wasn't added yet, mainly because I never use it myself. But that shouldn't be the sole criterion. So I'll add an issue for it as a "nice to have".

2. Better reports on things that aren't supported. This would require successful parsing, not necessarily successful code generation. Unfortunately not only the compiler, but in fact the CPython parser terminates of the first error it sees. Writing a special parser for this seems overkill to me. That it stops at the first error is not without reason. E.g. a C++ parser goes on and on, generating screens full of errors when leaving out a semicolon, causing misinterpretation of the correct code that follows. Maybe indeed a clever regex search could find some additional things, or at least hint at them. Will be hard to do properly, since it isn't always trivial to find out if something is inside a string or comment or not.

The low hanging fruit is gradually catching more things compile time.

3. The best place to do that is indeed in the command line app. At first compilation Transcrypt recursively draws in all imported modules, so after that it knows exactly which files to watch.

4. This has been brought up before and really should be added indeed.

[PerryBhandal]

1) I'd be happy to give this a shot. It may be some time before I can make this modification as my sense is that the changes required, while likely small, will require a reasonably detailed understanding of Transcrypt's code.

2) Valuable feedback. Thanks. For now I've written a regex to handle my \@property and string formatting issues. If in time I find that a regex approach is accurate enough to be useful I'll bring up the issue so it can be revisited/considered for addition.

3) To be sure, I'm assuming this would handle the case where we add new files, correct? My thinking is that you would add a new file, that on its own wouldn't trigger a recompile, but as soon as you import the file from one of the existing components of the app it would trigger a recompile and that would cause the new file to be watched for subsequent recompiles. Assuming that is correct I can likely handle this change. I do wonder whether that is overkill though. Is there any compelling reason for not simply watching all py files in the current dir and subdirectories and doing a recompile when any py file is changed (or any py file is added)?

[JdeH]

1. I'll assign that issue to you for now, again to prevent duplication of effort.
2. If the regex approach turns out to work rather good, I propose to add it as a compile time option, just like lightweight consistency checks and static type checks.
3. Actually it would watch only IMPORTED files, since it would register whatever gets imported in a watch list. The directory approach, on the other hand, is nice and simple. But it wouldn't work if a project is spread over multiple separate directory hierarchies, like e.g. when it uses other projects.

[JdeH]

Correction, GitHub doesn't allow me to assign the @property issue to you, since I created the issue and you're not part of the dev team yet. Could you yourself create a separate issue for @property, then I can assign you to it.

[PerryBhandal]

Property issue

Done. It's issue #294

1 and 2

Sounds good on both accounts.

3

Oh, good point. For now I won't address that as I already have a solution for directory watching so it doesn't warrant the effort. If I find I have time down the road (or if as I get to know the codebase better it ends up being an easy change) I'll bump this issue.

[JdeH]

I can imagine that the directory watch is good enough in most practical cases.

[PerryBhandal]

Hi Jacques, Some further details on TS definitions after some research.

1) An accurate, reliable implementation of this will require parsing TypeScript's AST. An additional benefit of relying on an AST is that we could easily add support for auto completion (without type checking) for JS libraries that do not have an associated TS type file.

2) For the best user experience we'll need to rely on importing JS/TS libraries as if they are Python libraries, as opposed to the current approach of retrieving them from the window object. That said, any autocomplete which supports PEP484 would allow you to get autocompletion for TS libraries as long as you explicitly type the object reference you retrieve from the window object.

3) My hope is that my implementation will enable both, as I far prefer the prior. Manually maintaining those type hints for every library I import diminishes its usefulness.

4) My usage of Transcrypt involves both typescript and Python code. There are areas (e.g. developing UI in React) where I find Typescript to be superior, and areas (e.g. using model objects from my Python back end, doing sorting, iterating over text) where I prefer Python. My hope is that we can also generate TypeScript typing files from the original Python source code so effective autocompletion and compile time static type checking can be done in both directions.

5) Given Transcrypt's focus on using JS libraries instead of transpiling Python libraries (e.g. as is done in PyJS and Brython) it would be beneficial to offer a deeper integration with NPM. I think this would be true in any case, but it becomes particularly beneficial once the above changes are implemented. The usage described below would provide an excellent workflow:

• Type transcrypt install_npm react
• This uses NPM to install the library and its associated type files (if they exist)
• Convert them/generate the Python bindings and make them immediately available within your Python search path.
• In your editor type from ts.react import React

Not only do you allow yourself to easily manage scripts/generation of typing files, you also avoid having to rely on retrieving objects from the window, managing script imports inside your HTML (as opposed to a single import of a bundle.js file containing all libraries) and, if desired, you could even easily bundle Typescript, Javascript and Transcrypt code into a single minified JS file with a single command.

--

I believe points 1 to 3 are pretty solid at this point, so I'm confident those will be implemented as I have outlined. Points 4 and 5 would definitely be nice, but I would likely not include them in a first implementation, and may abandon them entirely.

In any case, I hope to have some free time in the next few weeks to get that first version implemented. If at any point I decide to abandon it I'll let you know (as opposed to silently disappearing). To review, here are the features I hope to implement in addition to what I've described above:

• Adding support for the @property decorator

• Adding support for watching directories using pyinotify (which will accept multiple directories). This is the basic implementation mentioned in your last post. This is a tiny change, so I should have some time to get acquainted with Transcrypt's source and implement this before the week is out.

[JdeH]

Hi Perry,

I've annotatated your thoughts with some questions that came up in my mind:

An accurate, reliable implementation of this will require parsing TypeScript's AST. An additional benefit of relying on an AST is that we could easily add support for auto completion (without type checking) for JS libraries that do not have an associated TS type file.

What do you mean by autocompletion in this context? What can be left out and is automatically filled in. Not the typechecks you say? I know autocompletion only as an editor trick to save typing, but it seems you mean something different.

For the best user experience we'll need to rely on importing JS/TS libraries as if they are Python libraries, as opposed to the current approach of retrieving them from the window object. That said, any autocomplete which supports PEP484 would allow you to get autocompletion for TS libraries as long as you explicitly type the object reference you retrieve from the window object.

Currently, as an example, the fabric.js library is imported as a "true module" (with its own namespace), rather than attaching everything from it to "window".

http://sterlicht.alwaysdata.net/transcrypt.org/docs/html/integration_javascript.html#three-ways-of-integration-with-javascript-libraries

However to properly encapsulate a JS lib as a Python lib, also Pythoninc constructors would have to be provided, avoiding the need for __new__, as is outlined here:

http://sterlicht.alwaysdata.net/transcrypt.org/docs/html/special_facilities.html#creating-javascript-objects-with-new-constructor-call

I don't immediately see how this could be automated, so for a JS lib with quite a lot of classes, quite a lot of facade constructors would have to be written. Still, maybe the type info can help to automate this, but it doesn't seem trivial to me.

My hope is that my implementation will enable both, as I far prefer the prior. Manually maintaining those type hints for every library I import diminishes its usefulness.

My usage of Transcrypt involves both typescript and Python code. There are areas (e.g. developing UI in React) where I find Typescript to be superior, and areas (e.g. using model objects from my Python back end, doing sorting, iterating over text) where I prefer Python. My hope is that we can also generate TypeScript typing files from the original Python source code so effective autocompletion and compile time static type checking can be done in both directions.

Unfortunately I don't know a lot about Typescript, but I guess that would be possible, and certainly useful, since it would allow to write typed libraries in Transcrypt, to be used from TypeScript. Again you mention 'autocompletion', but now 'in both directions': same question as above.

All in all I like the idea of being compatible with Typescript, as Typescript is really becoming mainstream.

Given Transcrypt's focus on using JS libraries instead of transpiling Python libraries (e.g. as is done in PyJS and Brython) it would be beneficial to offer a deeper integration with NPM. I think this would be true in any case, but it becomes particularly beneficial once the above changes are implemented. The usage described below would provide an excellent workflow:

Type transcrypt install_npm react This uses NPM to install the library and its associated type files (if they exist) Convert them/generate the Python bindings and make them immediately available within your Python search path. In your editor type from ts.react import React Not only do you allow yourself to easily manage scripts/generation of typing files, you also avoid having to rely on retrieving objects from the window, managing script imports inside your HTML (as opposed to a single import of a bundle.js file containing all libraries) and, if desired, you could even easily bundle Typescript, Javascript and Transcrypt code into a single minified JS file with a single command.

Bundling everything in one minified JS file can be done rightnow, as you can see from the pong/fabric.js example referred to earlier. Automated generation of the encapsulation code would considerably streamline the process.

For fabric.js this encapsulation code is in its simplest form:

__pragma__ ('noanno')

fabric = __pragma__ ('js',
    '''
(function () {{
    var exports = {{}};
    {}  // Puts fabric in exports and in global window
    delete window.fabric;
    return exports;
}}) () .fabric;
    ''',
    __include__ ('com/fabricjs/__javascript__/fabric.js')
)

Note no encapsulation of __new__ calls in a set of Python facade constructors has been provided here. Such a facade could also include type info, and would ideally be generated automatically.

About monolitic minified JS files: I like them, since they can be minified very effectively if Google Closure gets a bit more clever (right now at any but the lowest "compression" levels it makes semantical errors. On the other hand many people make mixed pages, using an imported JS lib both in Transcrypt code and in JS code on the same page. In that case integrating the lib in the minified file would cause double inclusion, so larger downloads. So it should be an option, not mandatory.

--

I believe points 1 to 3 are pretty solid at this point, so I'm confident those will be implemented as I have outlined. Points 4 and 5 would definitely be nice, but I would likely not include them in a first implementation, and may abandon them entirely.

In any case, I hope to have some free time in the next few weeks to get that first version implemented. If at any point I decide to abandon it I'll let you know (as opposed to silently disappearing). To review, here are the features I hope to implement in addition to what I've described above:

Adding support for the @property decorator

Adding support for watching directories using pyinotify (which will accept multiple directories). This is the basic implementation mentioned in your last post. This is a tiny change, so I should have some time to get acquainted with Transcrypt's source and implement this before the week is out.

Take your time! I am a bit thinly spread these weeks, so maybe my feedback will be with some delay.

[PerryBhandal]

What do you mean by autocompletion in this context? What can be left out and is automatically filled in. Not the typechecks you say? I know autocompletion only as an editor trick to save typing, but it seems you mean something different.

I am referring to autocompletion "as an editor trick." Typescript has .d.ts files which just contain type information (e.g. https://raw.githubusercontent.com/DefinitelyTyped/DefinitelyTyped/master/jquery/index.d.ts). By relying on Typescript's AST library to convert that into equivalent Python definitions (as opposed to trying to do this by writing a basic parser that doesn't use Typescript's AST) we also get the ability to generate Python bindings for Javascript code (i.e. not simply a typing file in Typescript, an actual Javascript file with function definitions, imports, code etc) as Typescript is a superset of Javascript.

So, concisely, with Typescript libraries we'll get the same developer experience (as far as compile time type checking and autocompletion) you would get for Python code that has typed parameters and return values, while Javascript libraries (or any Typescript code that omits optional type hints) will give us the same experience we get in Python with code that does not use type hints.

new

Let me get back to you on this. My hope was that we would be able to generate Python constructors, but I should be able to answer your queries more effectively once I've investigated further.

fabric.js encapsulation

I hadn't seen that. Thanks for pointing it out. Admittedly even if I had I generally would not use that approach myself. It's pretty critical for me to have code that I can test using Python unit testing tools. I've run into some issues with that so far, and to an extent that may end up being impractical given Transcrypt's reliance on JS libraries. I view that as a 'cost.' I view that as a disadvantage relative to Brython and PyJS. Now, that said, Brython and PyJS are not really options for me given the performance cost of using transpiled Python libraries instead of Transcrypt's focus on JS libs. Additionally, the size of the resulting code from Brython and PyJS is far too large for my use case.

I suppose that, broadly, articulates my objectives: What modifications can be made to Transcrypt to provide a more Pythonic development experience (both in terms of how code looks, and its ability to be used with existing Python tools like unit testing, auto completion, type hinting etc) while not giving up Transcrypt's speed/file size advantages over PyJS and Brython?

[JdeH]

I am a "bare bones" developer:

• Never using a debugger
• Typing everything at speech speed with ten fingers, even if my editor has autocomplete it only annoys me and I switch it off.
• Roll my own unit test tools

I am aware that deviates quite a lot from the way many people develop. To become a success it is very important to connect to the mainstream way of editing, developing, testing etc. So it would a great improvement if you could contribute to Transcrypt becoming more suitable that regular kind of use.

And I guess things like you propose do not have to have a large impact on size or speed of the generated code, if at all. So your contribution in this area is very welcome.

[JdeH]

This seems to have come to a pause or halt. Will classify it as postponed and close for now. Feel free to reopen.

Kind regards Jacques