python3 type hints

[brollb]

I have been looking for a good in-browser python parser for embedding in a project of mine (so I can do some code analysis on the user's code). Skulpt looks like a really good fit - especially with the source mapping functionality.

However, it looks like it doesn't support python3 type hinting (https://docs.python.org/3/library/typing.html). Is this something that might be coming in the future? Would this be something you would be open to supporting?

[eah13]

Hi and welcome!

I think a PR for this would definitely be accepted, but since this project's educationally focused I'm not sure which current project members would have the same needs. Type hinting doesn't make it into browser-based curricula very often in my experience.

In the meantime, an embeddable server-side Python 3 interpreter might fit your needs more closely. You could host your own, or repl.it, Trinket (which I'm a cofounder of), and maybe PythonAnywhere all provide this as a service, and there are free options for each. Trinket and I think repl.it both have a share button to easily get embed code with your edits.

If you use IPython/Jupyter notebooks, there's a self-hosting path there or check out Anaconda Cloud.

This issue can serve as a place for others to +1 or claim this scope, but I wouldn't expect movement on it soon.

Hope that helps!

[brollb]

Cool, thanks for the options!

I think I will look more into adding type hint support here myself. If you have any pointers about adding this support it would be appreciated (otherwise, no worries :) ).

Also, is there any documentation about using the source mapping?

[eah13]

Sweet! This will probably be a pretty intense piece of work, but totally doable. What follows are some quick notes that I hope are helpful (and correct - anyone please feel free to point out errors).

First of all, HACKING.md should have lots more details for you, though the source code (unfortunately) remains the best place to learn how Skulpt works, by example. Recently merged PRs from core contributors are also great resources since they tend to be more 'idiomatic' than code that hasn't been touched in a while.

When developing a builtin module, a big initial question is how much of it will be python and how much will be js. The cpython typing module has lots of imports Skulpt won't support, but Skulpt's coverage of builtins is pretty dern good these days (including as of a few weeks ago super()!). You can either try to use this Python code and replace those calls with inline python (e.g. defining any missing classes) or calls to a js file (maybe _typing.js) you write, or try the whole thing in JS.

type is a js file in Skulpt, while types is written in Python (I think it's verbtim cpython code). You'll want to become intimately familiar with both, especially type.js.

If you go to reuse CPython code, a surprising number of its unsupported imports can be factored out or worked around without reimplementing them in my experience. They're often for edgecasey features that can get cut from scope anyways.

Like most skulpt module implementations, you'll probably want to restrict the scope of what you implement to the Venn diagram of what's most useful to you and what's non-horrible to get working. Select that scope, extract the corresponding CPython code bit by bit.

You'll need to add support for the def...(agrs: [annotation]) -> [annotation]: syntax, storing them in __annotations__ but only in Py 3 mode (see below). That's above my pay grade (I have so far only hacked modules & builtins, not the compiler) but someone else may be able to help you there. This should probably be its own PR.

You'd need to do the compiler changes first since the typing code relies on them. Then I'd say try to rehabilitate the CPython code.

An additional complexity is that typing.py is only in Python 3 (3.5+), so you'll need to gate your code with a Sk.python3 conditional (like #703) and add Py3 unit tests for both annotations and typing (see #688). You'll probably want to raise an ImportError in typing if Sk.python3 is false since it shouldn't exist in Python 2. Hopefully there aren't too many 2 to 3 changes in types and our implementation is solid enough to use for this (I haven't looked).

RE source mapping: if you mean seeing the compiled js from a python file, you can ./skulpt.py run myfile.py, which will prepend the js to the file's Skulpt output. For large or complex python files, it's probably best to redirect or pipe so you can less through the output since it'll probably exceed your scrollback. If you're talking about source mapping the minified skulpt.min.js file through to the uncompressed version in a browser, I'd recommend developing on the uncompressed version so you don't need to sourcemap. You can still put the sourcemap in your production environment if you want but I use ./skulpt.py dist -ud to build an uncompressed version and skip the docs to test a Skulpt dev build in our app.

huh. My quick note somehow got really long. One thing tends to lead to five others in Skulpt.

Does that help? If you're able to make progress but get stuck, a work in progress PR would help others help you on the diff.

[j-f1]

I’ve taken a look at the dependency tree of typing.py:

It’s pretty long; click here to see it

- [ ] [`abc`](https://github.com/python/cpython/blob/3.6/Lib/abc.py)

Members:

- [ ] `abstractmethod` - [ ] `abstractproperty` - [ ] `Counter` - [ ] `ABCMeta`

- [ ] [`_weakrefset`](https://github.com/python/cpython/blob/3.6/Lib/_weakrefset.py) - [ ] [`_weakref`](https://github.com/python/cpython/blob/3.6/Modules/_weakref.c) [native] - [x] [`collections`](https://github.com/python/cpython/blob/3.6/Lib/collections/__init__.py) [⚠️](https://github.com/skulpt/skulpt/blob/master/src/builtin/collections.js)

Members:

- [x] `deque` - [x] `defaultdict` - [x] `Counter` - [ ] `ChainMap` [optional] - [x] `namedtuple` - [x] `OrderedDict`

- [ ] [`contextlib`](https://github.com/python/cpython/blob/3.6/Lib/contextlib.py) - [ ] `abc` - [x] `sys` [⚠️](https://github.com/skulpt/skulpt/blob/master/src/builtin/sys.js)

Members:

- [ ] `exc_info` - [x] `stdout` - [x] `stderr`

- [ ] [`_collections_abc`](https://github.com/python/cpython/blob/3.6/Lib/_collections_abc.py) - [ ] `abc` - [x] `sys` [✅](https://github.com/skulpt/skulpt/blob/master/src/builtin/sys.js)

Members:

- [x] `maxsize`

- [x] `collections` [✅](https://github.com/skulpt/skulpt/blob/master/src/builtin/collections.js) - [ ] [`functools`](https://github.com/python/cpython/blob/3.6/Lib/functools.py) - [ ] `abc` - [x] `collections` [✅](https://github.com/skulpt/skulpt/blob/master/src/builtin/collections.js) - [x] `types` [✅](https://github.com/skulpt/skulpt/blob/master/src/lib/types.py) - [ ] [`weakref`](https://github.com/python/cpython/blob/3.6/Lib/weakref.py) - [ ] `_weakref` [native] - [ ] `_weakrefset` - [x] `collections` [⚠️](https://github.com/skulpt/skulpt/blob/master/src/builtin/collections.js)

Members:

- [ ] `MutableMapping`

- [x] `sys` [❌](https://github.com/skulpt/skulpt/blob/master/src/builtin/sys.js)

Members:

- [ ] `exc_info` - [ ] `excepthook`

- [ ] `itertools` [native] - [ ] [`reprlib`](https://github.com/python/cpython/blob/3.6/Lib/reprlib.py) - [ ] `_thread` [native, optional] - [ ] `functools` - [x] `re` [✅](https://github.com/skulpt/skulpt/blob/master/src/lib/re.js) - [x] `sys` [⚠️](https://github.com/skulpt/skulpt/blob/master/src/builtin/sys.js)

Members:

- [ ] `version_info` - [ ] `_getframe` - [x] `modules`

- [x] `types` [✅](https://github.com/skulpt/skulpt/blob/master/src/lib/types.py)

[albertjan]

Since our compiler is based on python2 it won't understand the : type notation needed for typing. So you'll have to opt for the mypy notation in comments: https://mypy.readthedocs.io/en/latest/python2.html.

[brollb]

Cool. Thank you to everyone for the pointers!

It might take me a little bit to get started on this (my schedule is pretty backed up atm) but I will definitely start digging into it once I get a couple free cycles!

[brollb]

@albertjan what if the python3 option is set? Could the compiler be updated to support this notation in this case?

[rixner]

@brollb, it's potentially possible, but it's going to be very tough (I think) to make the compiler simultaneously support both Python 2 and 3 syntax in the long run. The infrastructure wasn't really designed for something like that. There is some discussion about what we might do to support both 2 and 3 compilers in #688, but no agreed upon resolution...