Bundling with Parcel/npm

[doconix]

I just committed a branch that makes transcrypt work with Parcel Bundler. Parcel is a zero-config bundler that is gaining traction. The branch is at:

https://github.com/QQuick/Transcrypt/tree/parcel-bundler

The example run is at transcrypt/parcel-plugin-python/. See the README.md file for installation instructions.

When the test server is run, the parcel bundler starts with index.html and index.js, which contains:

import {main} from "./some.py";
main();

The .py extension tells parcel to run the file through transcrypt.

I've said this before, but IMO transcrypt must work seamlessly with bundlers for it to see wide use. Modern JS projects include many coding languages (JS, CoffeeScript, TypeScript, Rust). The role of the bundler is to transpile them all to regular JS and then bundle them together. Transcrypt is fundamentally different from bundler-based transpilers because it acts both as a transpiler AND a bundler. It doesn't just transpile single files, but it also includes everything those file import. For example, if `some.py` contains "import mylib", transcrypt compiles BOTH `some.py` and `mylib.py`. A single run of transcrypt usually results in a number of .js files, which are placed together in `__target__`. Regular transpilers generally stick to single files. When asked to compile `some.py`, they compile ONLY that file and don't follow any import statements. The import statements continue to keep the extensions of the files they import (transcrypt switches everything to .js). It's the job of the *bundler* to follow the imports and call the right transpiler. So while this branch makes transcrypt work with Parcel, it is a bit of hack because transcrypt isn't meant to work this way. It also breaks the normal transcrypt process because it keeps import file extensions (in revisit* methods). See the diff in `compile.py` for where this happens. The recent change to ES6 and units is a HUGE step toward doing things the bundler way, but a few things remain: 1. Transcrypt needs to keep import extensions (e.g. not change them). Like this: Python Source: ``` from calculator import addit ``` JS Output: ``` import {addit} from './calculator.py'; ``` This way the bundler knows to run transcrypt again for `calculator.py`. 2. Since transcrypt will only transpile a single .js file (the __main__ one), there's no need to copy other files to the `__target__` directory. This means the `__runtime__`, imported local files, and built-in modules (`re`, `time`, etc.) DON'T need to exist as siblings. Instead, these should be put into the npm repository just like any other imports. In fact, since only a single file export is needed, we can do away with `__target__` entirely. An example import from the built-ins: Source: ``` import re ``` JS Output: ``` import * as __module_re__ from 'transcrypt/re/__init__.py'; ``` Note that the JS import is not relative (./), so the bundler will look in `node_modules/transcrypt/re/__init__.py` rather than in the local directory. It will expect the `re` module to be in the npm repository. Importing the `__runtime__` should be the same way. I know JdeH wants to keep the current transpiler+bundler abilities of transcrypt, and I think that's fine. But I'd really like to see transcrypt become a first-order transpiler that works with bundlers out of the box. Right now it's not quite set up right to be one, but I don't think it needs a lot of changes to get there.

[artyprog]

Great 🙂

[JdeH]

@doconix There isn't a doubt in my mind that this should be in somehow, and on a short term if possible!

Some questions that come up:

1. How to proceed with sourcemaps and the location to store them? Simplicity should also be retained for people who come from a pure Python world...

2. I get your point about modules having to reside in the npm repository, but what exactly does that mean for e.g. the re module? It won't be named \_\_init\_\_.py on npm, would it, since that name would occur many times. What, in your view, would the e.g re module look like on npm?

3. Having modules like re, math and iterator on npm is bound to lead to name conflicts. But I'd like to keep the original Python module names and import syntax. However, a Python statement like import math simply doesn't contain enough info to prevent a name conflict if all such modules are stored on npm in "worldwide namespace". Transcrypt could add a prefix, but this cannot be expected from Javacript modules importing packages written in Transcrypt.

In general, how can we cross this bridge to the npm/bundler world, while keeping full connection to the Python way of doing things.

Don't be misled by my seemingly critical questions: I am enthusastic about your idea of matching the bundler / multilanguage world. But these issues cannot be ignored. So, while I am thinking about all this myself, any good ideas welcome!

One thing that's getting more and more important is retaining backward compatibility. I've noticed that Transcrypt increasingly finds use in large companies for critical applications. With that, protecting investments in existing user code has become an prime issue. The recent step from monolith to modules was unavoidable in my view, but from now on I'd like to avoid breaking existing user code, user scripts, user habits as much as possible, at the same time adding support for best practices like common use of npm and bundlers. These two are contradictory, but both require attention.

So: We should introduce npm and bundler compatibility, but, if possible, first time right, and somehow backward compatible. It has become quite hard to oversee the many ways in which TS is used. So we should move very carefully. Having said that, enable use of Transcrypt in the mix of languages handled by bundlers is absolutely crucial.

[doconix]

Awesome! I am encouraged by your response and questions. I also can appreciate the need for backwards compatibility.

There are straightforward answers to your questions, but I can't write the response right now. I'll answer more in depth tomorrow. Things like the sourcemaps are done by the other languages (usually streamed, I think). The namespacing has a good solution too. Look for my response tomorrow.

[doconix]

FYI, here's the current npm stats on the bundlers. webpack is still the goliath and is going strong. Their last release supposedly did a lot to alleviate the "config hell" it required. My impression is the crazy configs of webpack put off a lot of python devs -- the entire thing was pretty orthogonal to python culture (e.g. one way to do things, focus on simplicity and clarity, etc.). Webpack requires a lot of wrangling in my experience. I guess the most recent release fixes much of that, but my personal work with it showed it was still a pain (although please factor in my bias against the wild west of JS culture, despite me living in the actual wild west :).

Parcel was released in Dec, 2017, so it's about a year old. It's an up-and-coming bundler, but I doubt it could ever overtake webpack (which has significant momentum right now). It advertises zero config and is a reaction against the webpack mess. My experiences with it over the last couple weeks is it is MUCH simpler than webpack, but it's still not the "clear and simple" that python devs would want. My opinion (which could probably easily be swayed) is that Parcel is currently the bundler that python devs would be most comfortable with.

But hopefully if we can get a bundler like Parcel working seamlessly, it will prepare transcrypt to be easily integrated into the other bundlers as well. Parcel plugins are pretty easy to code, although I've had to get into the actual parcel source code A LOT. Like most things JS, the documentation doesn't go very deep (even if their docs are pretty).

So again, here's the stats on the bundlers. You can see in the picture that Parcel doesn't really register. However, it's large number of stars is encouraging.

[doconix]

How to proceed with sourcemaps and the location to store them?

Let me try to summarize how the TypeScript parcel plugin works. It's a good example of how all the transpiler plugins work.

1. Parcel sees an import statement in a JS file: import * as bar from './foo.ts'. This triggers the plugin to go to work.
2. The TS plugin loads the source from foo.ts into a variable and does the transpile.
3. The plugin needs to return a JS object containing the following to Parcel:
   1. transpiled.outputText - A string containing the output JS
   2. transpiled.sourceMapText - A string containing the source map in JSON format.
   3. In some plugins, the result might also include the actual AST tree using Babel AST objects (a widely-used library). If this isn't there, Parcel creates it through the babel library.

Note that the TS plugin doesn't have to use the filesystem at all and doesn't have to stream anything between processes because it's written in JS. It runs right in the Parcel process and returns the objects.

Transcrypt can't do this because it needs to shell a Python process out of the plugin. That's not hard from JS/Node, but we need to figure out a good way to transfer back 1) the transpiled JS and 2) the sourcemap JSON.

Remember that transcrypt is transpiling the imported file ONLY, not other .py files that the file might import. So we really only have TWO things to send back. Any other (imported) .py files in the project will be run in their own transcrypt processes.

It seems to me that streaming the result back to the plugin is the right way to do it so we don't clash with files on the filesystem. The stream could be JSON that contains the source + sourcemap. Alternatively, we could use two files placed in the __target__ directory like transcrypt currently does. The plugin is doing it this way right now -- it deletes the __target__ directory after reading the two files.

There are only two issues I saw with transpiling a single file rather than the entire an entire import tree. I mentioned them in my original post but I'll say them again because they apply to this point:

1. [easy fix] The import statements in the output JS source need to keep the file extensions of imported files.
2. [more difficult fix] The python statement from bar import * requires information from bar.py because the JS doesn't have this concept. The JS concept of import * as {name} from 'foo.js' is instead related to python's import foo. That means that the line from bar import * has to be transpiled as import { e1, e2, e3, ... } from './foo.js. We have to name each of the exported items in the imported file one by one, and that means we have to parse the imported file. This is doable when the imported file is Python, but it's a problem if the imported file is typescript or coffeescript or javascript.

One easy solution is to simply disallow the use of from bar import *. It's generally considered to be bad python anyway. This probably isn't the right thing to do long term, but you could punt on this one for the short term. Feel free to flame me on this one because I feel dirty even suggesting it. :)

Another solution is to cache the top-level symbols that exist in imported files somewhere on the filesystem. Parcel uses a stack for the files it is bundling, so imported files run before the files they are imported into. I think we can guarantee this. So if re imports re.translate, we can be guaranteed that re.translate has already been transpiled by the time re finishes. (edit: we'd need to read the parcel page on circular imports and ensure all is ok)

Edit 1: The second solution is wrong. We aren't guaranteed imported files run before where they are imported. :(

Edit 2: I just posted a new commit to the branch that splits Module into a superclass called ImportedModule. Since we only build one file at a time, the revisitImport methods now create only ImportedModules and don't do the full compile. Again, just a proof of concept, so the commented out code would need to be put into "if" statements to keep the old way working as well.

[doconix]

I get your point about modules having to reside in the npm repository, but what exactly does that mean for e.g. the re module? It won't be named __init__.py on npm, would it, since that name would occur many times. What, in your view, would the e.g re module look like on npm?

Hopefully the following gets the idea across -- it will take experimenting with examples to get the "final" layout, so this isn't an actual proposal. I'll do the re module since others are just siblings. I'm also thinking clean slate without regard to backwards compatibility at this point (so it will need adapting before going forward):

node_modules/
    transcrypt/
        src/
            modules/
                re/
                    __init__.py :: `import translate`
                    translate.py
        lib/
            modules/
                re.js :: "import {translate} from './re/translate.py';"
                re/
                    translate.js

My reading of npm best practices says both the original language (for reference) and the transpiled JS (for actual use) get published to npm. The JS directory of node_modules/transcrypt/lib/modules always gets used during bundling. Other languages like TypeScript can import the modules without having to install Python. I realize there's no real use case for importing transcrypt modules into TypeScript, but doing it this way keeps us in line with typical npm expectations.

Edit 1: Perhaps the src/modules and lib/modules directories should be collapsed to modules-src and modules-lib? Maybe I'm too deep with the proposed structure.

Edit 2: There is the idea of scoped packages that we could use instead of subdirectories for the standard libraries. I don't really like this but want to bring it up. My argument against it is I think we get a repository for each module. But probably something to look at. See scoped packages.

Edit 3: Pay special attention to the way __init__.py gets pushed up to the parent directory as re.js. Then we keep the re directory for regular filenames in that directory. I think this is the JS equivalent to what Python does and makes the imports work. See the next post on resolution order.

Edit 4: I see that the structure above is different than my original post on this thread. I think the structure here is better. But perhaps could even be improved. The idea here is what is important.

[doconix]

Having modules like re, math and iterator on npm is bound to lead to name conflicts. But I'd like to keep the original Python module names and import syntax. However, a Python statement like import math simply doesn't contain enough info to prevent a name conflict if all such modules are stored on npm in "worldwide namespace". Transcrypt could add a prefix, but this cannot be expected from Javacript modules importing packages written in Transcrypt.

Since transcrypt does the resolution of imports in .py files, we have full control over this. The generated import statement tells the bundler exactly where to look for the import: the source directory, the transcrypt standard library directory, or the top-level npm modules directory. The only issue I see is when python standard modules shadow over npm modules. I'll discuss that at the end.

Here's an example of various cases:

Python source: (pother is a local sibling python file, jother is a local sibling js file, re is a Python/transcript standard module, moment is a js library in npm repo):

import pother
import jother
import re
import moment

As transcrypt parses through the file, it decides how to resolve each one. The resulting JS import line determines where the bundler finds each file.

1. import pother: transcrypt finds pother.py in the source directory. It generates import * as pother from './pother.py'. (Parcel will call transcrypt separately to transpile pother.py)
2. import jother: transcrypt doesn't find jother.py but finds jother.js in the source directory. It generates import * as jother from './jother.js'.
3. import re: transcrypt doesn't find re.py or re.js in the source directory. So it goes to the npm repo, first looking in transcrypt's location: node_modules/transcrypt/lib/modules/. It finds node_modules/transcrypt/lib/modules/re.js. It generates import * as re from 'transcrypt/lib/modules/re.js'.
4. import moment: transcrypt doesn't find moment.py or moment.js in the source directory. It doesn't find moment.js in the transcrypt npm dir. It finds node_modules/moment/, a normal npm module. It generates import * as moment from 'moment'.

JS output:

import * as pother from './pother.py';
import * as jother from './jother.js';
import * as re from 'transcrypt/lib/modules/re.js';
import * as moment from 'moment';

So looking at the above, I think everything works beautifully with the one possible exception of a npm module being shadowed. Looking on npm, there is a module named re. That package can never be imported from python source code because transcrypt will always resolve to the standard python re.js (as above) and never to node_modules/re/. I would argue that this is an edge case, so perhaps the solution is as simple as __pragma__ ("js", "import * as re from 're'") to gain access to the shadowed npm repo?

(edited to fix some copy/paste mistakes)

[doconix]

Feel free to PM me if you want to set up a F2F call to discuss any of this. Also be sure to check the diff on the actual transcrypt source code (master branch vs. parcel-bundler branch). The changes are pretty minimal but were just a proof of concept rather than finished code. I think that diff shows how close transcrypt is to making this happen, but I'm leaving the decisions to you on how/what/where to modify the transcrypt code if we go forward with this kind of thing.

[doconix]

A thought on the backwards compatability... I agree you can't leave people hanging in the dust. Especially after the change to ES6 that just happened. I'm not sure what the cleanest way to go forward is.

My thought is I'd guess that 90% of the users that are NOT using a bundler only because it's a pain and because they lack the know-how. JS bundling can be difficult for python devs to understand because JS project structure is not what the python community is used to. Python compiles .py files to .pyc, but otherwise keeps the source structure at runtime.

Bundling requires setting up npm, node_modules, and multiple config files. The import landscape is a battleground of similar-looking but quite different standards and libraries: <script src=>, jQuery plugins, commonjs, amd, requirejs, npm, ES6 imports, etc. For Python devs who are used to a benevolent dictator solving divisive issues (like Guido did with the m if x else n issue), the chaotic and evolving JS ecosystem can be overwhelming.

In its current form, transcrypt provides a one-stop shop for including Python in web projects. So we use it that way. But as we get more experienced with JS and start to decipher how the competing standards finally worked out, we convert our projects over to npm-based, bundled structures. But then we find that transcrypt doesn't quite fit in our new structure.

Concluding my soapbox rant here, transcrypt is strategically placed to guide python devs over to the JS bundling world. If we can get it working seamlessly with Parcel, we can create a few model projects that show how it is done. I'd bet 90% of transcrypt users who aren't currently bundling would jump on it if they had boilerplate code and related walkthrough.

Bottom line, while you obviously can't hang existing users out to dry, I bet the backwards compatibility issue is more a function of education than actual need.

[ disclaimer: All above statistics pulled out of the air. YMMV! ]

[doconix]

FYI I'm implementing some of these things today (parcel-bundler branch, not in master).

[doconix]

Status report: The changes to transcrypt to allow direct plugin use are almost done. I'll probably finish it tomorrow.

[PythonLinks]

@doconix

Your initial post on the parcel bundler was quite brilliant. Really easy to understand. It looks like you had been thinking about this stuff for a long time. I am just starting to learn about Transcrypt, but if the rest of the documentation is written this well. Awesome.

As I read further on this issue, my eyes began to glaze over. I am sure that it is true that for larger Javascript projects (That is where the money is) , Transcrypt has to work with bundlers. But the complexity just gets overwhelming.

The problem is that the mental load on the user is too great. I have to understand the whole Python world. I have to understand the whole Javascript world. And I have to understand the mapping between the two of them.

I am thinking of doing a talk/video on Transcrypt. I think that for presenting it to the users a very simple conceptual model is required. “Here we have Python on both client and server. Transcrypt takes a Python-Centric view on the world. You do not need to worry (too much) about the Javascript details. But they are there if you need them. "

As soon as you try to adapt to all of the strangeness in the (broken) Javascript world, you corrupt your clean message. You add huge conceptual burden to your users.

So much better to say that Transcrypt includes a bundler. The existing bundler’s do not have the correct mental model.

Then really focus on the smaller market of projects run by people with a Python-centric world view. Rather than the larger market of people with a Javascript-centric view of the world. In the Python community, your probability of adoption is way way higher. Even this smaller market is large enough for you. Don’t be greedy. Focus on a clear and simple message to a well defined target market.

The software world really needs simplification, rather than additional complexity.

From: https://www.simple-is-better.org/ It seems that perfection is achieved not when there is nothing more to add, but when there is nothing more to take away. — Antoine de Saint-Exupéry: Terre des hommes

[JdeH]

@doconix

I'd like to evaluate the changes you made, but have trouble getting an overview since lots of whitespace changes are also marked as differences. I've tried the ?w=0 option in the diff view but it doesn't solve the problem. Would it be possible for you to have only relevant (non-whitespace) etc. changes on GitHub, enabling comparison to the master branch. I've tried an external file compare tool but if failed on the job, ignoring relevant differences. Sorry I have to ask this, but since the work on Transcrypt really takes up lots of my time now I feel forced to streamline. Know that I greatly appreciate all the effort you put in!

@PythonLinks @doconix

It's agreed that Transcrypt has to stay as simple as possible. I must say I find it impossibly hard to get any kind of grip upon the explosion of half-compatible, half-contemplated approaches in the JavaScript world. The prime reason Python is becoming so popular is its simplicity, although part of that recently is getting lost in overstressing facilities like iterators, decorators, metaclasses, static typing etc.. I hope Python is spared from getting as complex as e.g. C++, although I really like that language and have used it most of my working life. Especially with Guido no longer at the wheel I am worried that the "the more the merrier" approach takes over. To be honest it isn't the low threshold common sense scripting language anymore that I once bumped into.

From the docs of Transcrypt:

http://www.transcrypt.org/docs/html/what_why.html#what-is-transcrypt

One of the main goals of Transcrypt is to keep things simple, preventing it from disolution into an ocean of feature creep and conflicting demands. At its core it will stay lean and mean.

Wanting, or rather needing, this simplicity for me was the reason to abandon JS development of browser apps and start the Transcrypt project.

Transcrypt is primarily meant for Pythonista's who want to make browser applications in Python while using JS libs. Still the possibility of tight integration with the JS world is valuable. In this JS world an ever changing diversity of bundlers seems to be a fact of life. I don't expect these to converge as many JS developers don't seem to be particularly interested in that kind of thing. While popularity may shift, I don't expect it to ever really stabilize unless the ECMA committee proclaims a standard.

To support use of Transcrypt in combination with those, I propose the following criteria:

• The current simple way of working, including use of the flat __target__ directory holding everything including sourcemaps, should be preserved as default.
• Changes aimed at making Transcrypt fit in the bundler world should not complicate TS's sourcecode too much, as it will all have to be maintained.
• Changes in Transcrypt itself should not be tied to one particular bundler. They should comply with emerging ECMA standards (still very virtual in this area), as does the module mechanism.

At the same time it would be very nice if Pythonista's could write e.g. datascience/AI related stuff in Python that would be available as JS compatible libraries that can be drawn into JavaScript apps the way JS programmers are used to, so using popular bundlers. This would enable Pythonista's to enter the enormous JS and Web Client world and make Python a true lingua franca. That's why I am very enthusiastic about what @doconix is trying to do. But it isn't exactly simple to do this and meanwhile indeed keep it simple...

I want to be very clear: my heart is with Python, not with JavaScript. But since it's a massive stream, Pythonista's should be able to tap into it, even if I hope their main focus will be making algorithmically interesting server applications, rather than fancy front-ends whose main purpose is to look good. But not being able at all to make a browser front end is something not many developers can afford. And JavaScript is just, well...

I find it very hard to determine the right course here. So all I can do now is following @doconix' experiments, absorb opinions and insights like those of @PythonLinks, try to determine how it all fits in , develop some connecting ideas and wait, wait, wait (despite the hurry that I perceive as real) until it all feels solid and future-safe. And with future I don't mean 2 years (guess I am a bit too old to consider that "long term"). Wouldn't want to fall in the same cracks that litter the JavaScript landscape due to it's never ending earthquake...

[doconix]

@JdeH Thanks for the feedback on this. I've been at this the last couple days, and I'm pretty sure it will be done by today sometime. I'm almost there, but as you know, bugs can be a long tail.

The changes to transcrypt are actually minimal, and I've been able to keep full backwards compatability. The bundle way of doing things is just an option.

Some of the ideas I proposed didn't work as expected, so don't waste too much time trying to go through it. I think the finished code will be clear.

The implementation is very much in line with your 3 criteria. Well, except one. :) I was able to implement a deeper target structure rather than flat. I think it's more clean this way because the target directory matches the source directory. BUT, you da boss, and I can step back to flat if you want. But perhaps you can wait until you see the running code to make a final decision...

I'll post again when done.

[doconix]

@JdeH Status update: It's working as of about an hour ago but I still need to do tests and ensure things are working. It will be another day or two. I have to step away from it for a bit.

[doconix]

@JdeH I am now working in the parcel-bundler2 branch. I restarted from scratch (and copied relevant code from the other branch) so I could be back to a pristine copy of transcrypt. I'm still putting the finishing touches on it, but I want to put out a few questions to you:

1. Right now I have the code in the transcrypt/parcel/ folder. Do you want this code in the transcrypt codebase, or do you want me to put it on a separate project? The plugin gets posted to npm, so it really doesn't matter where the source lives. It is really just a matter of preference on whether you want to keep things together or split them up.

2. I'd like to write a page in docs/ on using the parcel bundler, but again I want to ask if that is ok. Separate place or integrated with transcrypt docs?

3. Despite me making lots of changes to transcrypt's code in various tries, the end result is almost no changes! The one change is in the constructor of Program.__init__:

The plugin needs to know what .py files transcrypt found and what .js files transcrypt created. I'm putting these in the .options file you create in Program.__init__. The need for this is so the plugin can autoreload during development (recreate bundle anytime a .py file is changed). This is the ONLY thing that I had to change in the codebase, and all of it is in compiler.py.

The file now uses json format (instead of pickle) so my JS plugin can read it, and the name is changed: <main>.options -> <main>.transcrypt.json. There's no real need for that name change; I changed it simply to be closer to its new purpose. My reading of the code says that all the pickleableOptions are ok types for json.

Let me know if it's OK to expand the options file like this.

4. No changes to command line arguments.

5. The plugin is using the flat structure. No need for deep structure. I did have that code working if you ever want to explore it, but flat works fine for the plugin.

6. I tested the built-in modules (__runtime__, re, etc.) on npm, but it actually worked better to keep them within the __target__ directory. So no links to npm repo in the plugin.

7. The plugin is not tied directly into transcrypt, so other plugins and other bundlers hopefully can use (improve) the strategy I used for webpack, etc. No favoritism to this plugin. After a break, I can probably adapt it for webpack pretty easily. The current webpack plugin does work, but it's not seamless (imo).

Let me know about this code living inside transcrypt's github vs. separate project. And let me know about the docs. Regardless of your decisions on those two, I hope we can reference the docs where people can find them. I want to make it as easy as possible for people to use transcrypt in bigger projects.

[doconix]

Boom baby! The first version of the parcel plugin is posted to npm:

https://www.npmjs.com/package/parcel-plugin-python

It also includes a step-by-step tutorial. Hopefully this helps others figure out how to get bundling working.

@JdeH Check out my questions in my last post when you get free. Right now the plugin lives in the parcel-bundler2 branch, but let's get it into either master or into a separate github repo. Also the question about the docs. Thanks.

[doconix]

@JdeH Just a ping on this one.

[JdeH]

@doconix

Today I've started evaluating your solution very carefully. I will take the necessary time for that, as I consider this a crucial piece of the Transcrypt puzzle!

[EDIT]

I have to look through all the details before coming to a final conclusion, but still I'd like to add my first impression: THIS LOOKS ABSOLUTELY FABULOUS!!

Will get back to you A.S.A.P.

Jacques

[JdeH]

@doconix

I am attempting to run the examples, following the directions at

https://www.npmjs.com/package/parcel-plugin-python

but at the first example I get:

Cannot find module 'parcel-bundler/src/Asset'

and

Cannot find module 'parcel-bundler/src/Logger'

I am using npm 6.5.0, parcel-bundler 1.11.0, parcel-plugin-python 1.0.18, transcrypt 3.7.12

Any idea how to proceed? I'd love to find out how it all fits together...

[EDIT]

I've made some progress, certain things were missing from the parcel-bundler\src dir:

Logger.js emoji.js prettyError.js

I looked them up on github and added them. Rightnow it complains about not being able to resolve:

D:\activ_tosh\geatec\transcrypt\bundling\test\main.py:1:14: Cannot resolve dependency './targetmain.js' at 'D:\activ_tosh\geatec\transcrypt\bundling\test__target__main.js'

It looks like Windows \ rather than Linux / got it confused. Probably you tested on Linux? Guess I'll just persevere a bit... Will keep you informed, insights / directions welcome.

[EDIT 2]

The first demo now works 😄 . I had to change line 46 and 47 of asset.js to:

        this.importPath = './' + '__target__' + '/' + this.pyModule + '.js';
        this.runInfoPath = this.options.rootDir + '/' + '__target__' + '/' + this.relativeDir + '/' + this.fileinfo.name + '.transcrypt.json';

to make it work on Windows. Somehow path.join didn't work properly on Windows. I will now try out the other examples.

[EDIT 3]

Tests in Transcrypt/parcel now also work, with same modifications.

[EDIT 4]

Transcrypt 3.7.13 contains the necessary changes to enable dynamic reloading by Parcel. 🏳️‍🌈🏳️‍🌈🏳️‍🌈

The only change with respect to your code is that the file containing the files to be watched is called .project rather than .runInfo.json.

Reason for adapting this:

• Naming it .runInfo could lead to devs thinking that it plays a role at runtime (of the application rather than of Transcrypt).
• I skipped the .json extension since only a dev working on either TS or something compatible with it, like parcel-plugin-python would have to know that it's actually JSON. So I preferred a brief extension here, reminding the 'user' of such files in some IDE's.

Furthermore there are still the three missing files in bundler itself, as mentioned above. And the construction of runInfoPath and importPath, as also mentioned, no idea why this gave problems on Windows, looked quite alright to me... I adapted the latter in asset.js, in such a way that it now (also?) works on Windows. Unfortunately my virtualbox Linux install currently has the flu, so I can't test in Linux.

I also had to skip setting an environment variable to 'test' in one of the json files, since this understandably didn't work on my Windows box.

There may be some loose ends and possible improvements, but all in all I think this is a GREAT improvement, and I really want to thank you for this contribution.

A few questions remain open, most notable whether or not this can best be distributed as part of Transcrypt or separately, same question for the docs.

Currently the sources are part of the Trancrypt git project, but I cannot update NPM, which isn't the handiest situation, since I'd like to do that automatically. I am not sure what's best here. The technical problem seems to be solved (by you) which is by far the most important thing.

Lets have some further one to one contact on how to distribute this. Also in the long run it may be good to include your tests in the autotest, even if parcel-plugin-python would be a separate project. In that case the situation is comparable to the node-parts of the autotests.

I propose to have some sleep over this, and then solve it next year (which is fairly close).

[doconix]

@JdeH Sorry I've been out of touch on this. Thanks for spending time on it. I don't know why those files would be missing in the parcel-bundler package....

Anyway, agreed that we should discuss offline. Perhaps a screen share would be useful so I could show you what it's doing, and we can walk through each part.

I'd also like to mirror the ideas directly into webpack so transcrypt can work with both parcel and webpack (and the two plugins are similar to develop/maintain).

[doconix]

@JdeH I just posted a new update to the parcel-bundler2 branch. This does the following:

1. Moved the testing files for Parcel into the manual_tests directory. Made adjustments to code and docs as needed to match.
2. Created a Webpack loader that mirrors the way the Parcel plugin works.

One cool thing about the webpack loader is it is programmed in Transcrypt rather than JS. I tried switching the Parcel plugin to Transcrypt, but that has to extend the Parcel Asset JS class, so I reverted back to JS for it.

I still need to do the testing files for the webpack loader.

[pauleveritt]

Just curious, is it safe to write a tutorial showing this Parcel-based approach?

[pauleveritt]

As followup, here's a tweet from @jviide (with a tweet before it) showing VDOM and rendering in Python, as well as browser via Transcrypt. For folks interested in Python and SSR, it's an important step.

[JennaSys]

Logger.js emoji.js prettyError.js

Is there any plan to fix the missing file issue? Other than that, this Parcel plugin has been working great.

[JennaSys]

Following up, there was a change in the Parcel library where the logging module was moved. It can be fixed by changing the reference to the logging module import in asset.js as I mentioned here.

[JennaSys]

Now that Parcel v2 is out and the current Parcel plug-in for Transcrypt is incompatible with the new version, it might be worth revisiting this to create a parcel-plugin-transcrypt-v2 that deprecates the old one that is broken anyway.

[JennaSys]

I just published a Python transformer for Parcel V2 that uses Transcrypt. I'm still working on the docs and it needs way more testing in different environments than I've done so far, but I figured I'd put it out there early and get some feedback.

Feel free to open up a GitHub issue if you run into any problems with it.

Edit: I just posted a write-up on using Parcel v2 with Transcrypt and the new plugin.