External transpilers for additional languages

[jayvdb]

Would you be interested in adding support for more languages by utilising existing py2x transpilers? I'm especially interested in JS atm.

e.g. for JavaScript https://github.com/atsepkov/RapydScript does a good job of converting Python to JavaScript with a similar result as py2many provides. (unlike http://www.transcrypt.org/ which generates a bastard of the two languages) I havent looked at whether https://github.com/kovidgoyal/rapydscript-ng is fundamentally the same still, but it appears to be.

https://github.com/pseudo-lang/pseudo-python also might be useful, even though it is a subset of Python, which excludes globals it seems (ugh; and class level attributes are disallowed!), but it very likely has more of python supported than py2many in its current form. (dev has also stalled, it seems)

https://github.com/onelang/OneLang and https://github.com/jarble/transpiler/ look very interesting.

It would only involve an enhancement to cli.py , and a test module (which I think I would disable by default), which processed the test cases here against the external transpilers.

We could then build our own implementations side-by-side and swap over when the local impl is close to reaching parity with the external tool.

[adsharma]

dart2js is another way to get there.

[jayvdb]

Right. The question is whether this project will use other transpilers if they have been pre-installed. I am willing to do it, as it has slight benefits for me, but only if the PR is about how we do it, and not if it is within scope of this project.

[adsharma]

I don't mind having one "external" transpiler, which takes a static configuration option (Essentially the command to run) that calls into another transpiler.

py2many.py --external=1 test/case/coverage.py

Any additional complexity worries me from a maintenance perspective. I'd like to keep the test matrix and test run times roughly the same as what we have now.

[jayvdb]

I don't mind having one "external" transpiler, which takes a static configuration option ..

This isnt enough to justify working on this enhancement. If I wanted that, I would just invoke the transpiler directly.

To be useful, the extra languages needs to be in the cli.py settings, it knows how to invoke it, just like "native" languages. That brings the "cheap win" of getting extra languages quickly, making it easier to for people to adopt py2many as opposed to using other tools. The more languages the tool supports, the more likely a manager will approve a developer using py2many or allocating time to helping py2many.

I'd like to keep the test matrix and test run times roughly the same as what we have now.

I agree, esp wrt to CI. There is no benefit testing the external transpilers in each CI run. For CI, it only makes sense to have unit tests to verify the logic in cli which invokes external transpilers is working - a mock would suffice.

Above I mentioned that I would disable any tests for external transpilers. i.e. even if you had the binaries, the test would not run unless explicitly enabled, e.g. with a envvar TEST_EXTERNAL=1.

[jayvdb]

I tested dart2js, and it works on the two global test cases, so that would be a good choice for me. Its output is horrid, but I dont care about that much.

I also like that approach because then the externals can be only a post-process phase on top of one of our native transpilers, so they are still limited by bugs in our transpilers, so in order to get features working in an external transpiler, someone needs to fix the bugs in one of the native transpilers.

[adsharma]

Summarizing my thoughts on adding js as an external transpiler:

[+] one less thing for users to worry about and use py2many as a universal transpiler front end [+] js is popular, so having human readable code would be nice [+] python -> js is more performant vs having dart as an intermediate step

[-] js is a language with rough edges. dart does away with many of those. [-] python and js being dynamic languages, people may have an expectation that they support all dynamic features that map. The main idea of py2many is to support only a tiny subset of python features that map to statically typed languages. This expectation mismatch could add some friction.

Overall: I'm not opposed to having another language backend that calls into an external transpiler, as long as it doesn't add much code into base and doesn't regress CI.

[jayvdb]

https://github.com/dmitrii-eremin/python-lua/issues/11 looks good for Lua.

[adsharma]

dmitrii-eremin/python-lua#11 looks good for Lua.

Yes - this code looks similar to what we have in py2many:

https://github.com/dmitrii-eremin/python-lua/blob/master/pythonlua/nodevisitor.py

Perhaps it can address a larger set of use cases by making use of the generic type inference code we have in py2many.

[jayvdb]

Javascript:

1. RapydScript - doesnt implement assert. Within the unittest module, they implement it using a simple helper. https://github.com/atsepkov/RapydScript/blob/6e18686a693b2a03e3146a968d97b6307285c85c/src/lib/unittest.pyj#L17 . We could create a rewriter for that.
2. RapydScript-ng - implements assert, but not typing. https://github.com/kovidgoyal/rapydscript-ng/issues/126
3. dart2js - requires main, i.e. no libraries, and it eliminates any code which is unreachable from the main, making its output very different from typical py2many output
4. dartdevc - works with libraries, doesnt remove unused code, and can be run with node using hidden arg --modules=node, but it depends on dart_sdk.js which is of course huge! (5M un-minified)

[jayvdb]

gopherjs creates a 1.3M .js file for print.py

Transcrypt creates runtime of 43K + 1.5JK for print.js . Our wrapper could merge them into a single file.

So far I dont see a clear winner, so it is looking like RapydScript needs to be explored more by creating an assert rewriter.

If that doesnt work, perhaps we need a Haxe transpiler, which might be a better route to get JS.

[jayvdb]

RapydScript has its own typing system using "Number" and "String" and probably others. Ugh.

[jayvdb]

RapydScript 0.5.62 is the latest release on npm, but https://github.com/atsepkov/RapydScript/releases has 0.6.00 :/

After using https://github.com/abarker/strip-hints and autoflake to remove the hints, 0.5.62 and 0.6.00 both fail on coverage.py due to not implementing any of:

• x if y else z ternary operator, which would require rewriting to injection functions for each of these
• sets of form {1, 2}, easy enough to rewrite as set() which I assume it accepts
• list.append, could be rewritten using +=, if it accepts that, or similar

enum and lambda are broken.

Implicit dit.keys() works, however dict.py is not working as .keys() and .values() are not implemented.

I think with a list.append rewriter (and my other rewriters), there is enough working tests cases that RapydScript provides a good-enough implementation to be useful, and we'd then try to get our own JavaScript implementation to provide similar results. But RapydScript is definately a dead-end, and it seems like a Haxe transpiler is going to be needed sooner or later.

[adsharma]

Haxe is an interesting target. It's been around for a while and has grown organically.

[jayvdb]

There is a good list of Python to JavaScript transpilers at https://github.com/jashkenas/coffeescript/wiki/List-of-languages-that-compile-to-JS#python

Many are defunct, but https://pypi.org/project/javascripthon/ and https://pypi.org/project/pscript/ look ok.

[jayvdb]

Both javascripthon and pscript lack set support, but pscript did a better job with our test suite, c.f. https://github.com/flexxui/pscript/issues/59 , and has reasonable generated code, and a fairly extensive test suite.

https://github.com/flexxui/pscript/pull/60 is an attempt to add set support. If that can be supported, adding pscript looks like a good solution. It doesnt do typing, but it also doesnt fail when typing is used due to unreleased https://github.com/flexxui/pscript/pull/56 . This isnt so critical as JavaScript isnt typed anyway, but it does mean the generated output uses JS functions which handle dynamically handle more types than the input script needs.