Export Bundled PyScript Files

[JeffersGlass]

As in this discussion, having the pre-bundled @pysccript/core/index.js or core @pyscript/core/dist/core.js would be helpful in situations someone wants to use pyscript in a situation where they might want to re-bundle PyScript into their own package. It would be useful to have them included in the exports key.

From what I understand, there are good reasons why re-bundling PyScript is not advisable, since it could lead to situations where, say, Polyscript is bootstrapped twice in one page. But having the possibility would be useful for some workflows. (Tagging @e-nikolov)

[WebReflection]

the module itself does the bootstrap (nothing new, "classic" did the same) and from the module, if X 3rd party modules bundle it, they'll share nothing, as every of them will re-bootstrap the MutationObserver or the interpreter itself (X different pyodide that all works out of the same py-config, as example, and yet still share nothing across them).

We can work on fixing exports but my "evil thoughts" around this issue is that the exports field should point to a file that throws an error out of the box where the message explain why that's a bad idea ... ugly? sketchy? disturbing? Yes, but it would also be for the best usage of our offer.

[JeffersGlass]

So I think I'm missing something here. Let's say I'm creating another project which incorporates PyScript - in which I'd like to do import { PyWorker} from "PyScript". Big quotes around what "PyScript" is there, the import is the important part.

If I could bundle PyScript in, then users can simply link in my project: <script type="module" src="https://example.com/Python4.js"></script>

If I can't bundle PyScript in... then everyone who wants to use my project has to use an importmap as well?

<script type="importmap">
 {
    "imports": {
        "@pyscript/core": "https://pyscript.net/releases/2056.11.1/core.js"
    }
}
</script>
<script type="module" src="https://example.com/Python4.js"></script>

Have I got this right?

[JeffersGlass]

It seems like we're trying to prevent multiple external modules from bootstrapping PyScript (i.e. Mutation Observer and Pyodide) but in a way which makes it uglier to build things on top of PyScript, which I know is a stated goal. ("Let many flowers bloom", as @ntoll is fond of saying).

[WebReflection]

the issue is how ECMAScript standard modules work ... if you re-bundle PyScript twice nothing is shared across their own bootstrap or exports (hooks). If we even decide to leak a global entry to ensure there's only one bootstrap, nothing in terms of hooks or anything else, literally, will work, because further modules won't have a say to the only, first-come-first-server, import of PyScript that is out there.

This is nothing new for the project, it's been always meant as script before, so we didn't have this "issue", but making it an ECMAScript module doesn't mean everyone should embed it, same way nobody should embed polyfills on the Web, and that's a well known story, as these will repeat with no guarantees, the same polyfill all over.

In that regard, PyScript is a polyfill for the Web that patches scripts and custom tags around, and including the same polyfill twice, leads to shenanigans, as nothing will work for the second module that bundled the same poly ...

Will it work if your own thing is the only one on the page? Yes

Will it work for every other re-bundled same module out there? No, simply because that's how Web standards or JS standard modules work.

[WebReflection]

If this is a must have, we need to break everything again:

• we publish PyScript not as module, we leak hooks or any of its exports on the global as we used to do, problem solved, as nobody can import anything from it, we live in the 90s' where ECMAScript modules never existed (maybe for a good cause)
• we really stress our module should not be bundled for basic to most convoluted use cases beacuse that is the way to go around our module ...
• we follow previous point and we leak something on the global context (window) to be sure if that ever happens, we still refer to the same initial module ... a slippery slope to me, but maybe a good way to avoid bootstrap conflicts out there

[JeffersGlass]

Certainly not my decision alone if we decide we want to break the world again! But thank you as always for taking the time to explain the situation (both what one should do now, and the consequences/requirements should we decide to change course).

Let's get the team involved and assess?

To be clear, this seems like a sufficiently big decision that we shouldn't try to change anything before the upcoming release. Just my opinion. Hopefully it stirs up even more interest and gets people trying to use PyScript and integrate it into their own projects... and we can know a little better what way to head. Seem reasonable?

[WebReflection]

So far, at least polyscript, is a successful story as a module and users already built a lot around it ... this is not to state the module approach is ace, just to state modules are well known and well documented out there, but maybe there are no use cases yet to consider about multiple pre-bundled polyscript.

The main blocker to me is that a module cannot be both:

• exports must be explicit at the top level, so no if (!globalThis.PyScript) export thing; is allowed, but maybe const thing = globalThis.stuff ?? stuff; export {thing}; is (still ugly, but...) maybe possible
• if we ship a global leak in 2023 there will be questions from the Web community and I can sympathize with all of them already

Like you said, not my call neither, but technical constraints must be understood.

[WebReflection]

P.S. the global leak approach will inevitably allow anyone to embed PyScript too, without any guarantee their version matched, once live, the current one served from the first embedder .... this is although true anyway we go so maybe just an extra detail to add.

[JeffersGlass]

Just so we have a jumping off point to bring others in - Is it fair to say that the code sample I posted above is the way for other project developers incorporate PyScript at this moment in time, assuming we didn't make any changes? Just want to be sure I've got the right tactic for the momnt.

I'm interested to get into this, since I think it's essentially for the plugins story as well. (Since at the end of the day, "a plugin" is just a module that imports Hooks from PyScript and does things with them, yes?)

[WebReflection]

P.S.2 we can make it somehow happens, and it will be ugly and hacky, but I think it might work too ... still tons of changes to land if we want to allow Pscript as embedded module that works both as stand-alone or as shared module.

[WebReflection]

I'm interested to get into this, since I think it's essentially for the plugins story as well.

me and @ntoll demoed and I've stressed already that plugins should not re-embed the whole PyScript core in them ... reasons are: performance, no conflicts, expectations!

that wasn't by accident ... so to me is a matter of documentation, because I don't want to have all non-core plugins re-embedding PyScript for bandwidth and performance sake, but at the same time, I perfectly understand that's how everyone careless or not advanced enough would create plugins ... so I am game for the global leak, but that adds also complexity for the worker story and I haven't yet realized what could possibly go wrong there.

[e-nikolov]

In my TypeScript project I was initially using polyscript as a dependency via yarn and bundling with vite (which internally uses rollupjs for prod and esbuild for dev). I recently switched to @pyscript.core but had errors due to @pyscript.core having relative imports to the internals of polyscript. I worked around it by using yarn patch to export the polyscript internals and remove the relative imports.

I am relatively new to the JavaScript and TypeScript ecosystems, and I didn't find another way to import @pyscript.core that makes both vite and vscode's intellisense happy.

My project is a library that has both a TypeScript and a PyScript component that interact with each other. I still have not figured out a good way to make my library usable by other projects.

[bugzpodder]

@e-nikolov I've been doing something similar like what you are trying to do. https://bugzpodder.github.io/pysandbox/

it basically is a typescript library that wraps polyscript. I've also mentioned some of the things I am looking for: https://github.com/pyscript/pyscript/issues/1763#issuecomment-1749680771

My question to you and @JeffersGlass is what are you looking for in @pyscript.core outside of polyscript? Is it the stdlib/pydom? the plugins? It seems PyWorker is XWorker + stdlib? I like some of the additions like pydom but I also don't want to deal with the customElements things in my app since they are unnecessary for anyone who is already using a js framework. In my mind pyscript.core can be a framework that pyscript is today (building apps in html), while polyscript is the library everyone can incorporate. and the plugins ideally can be their own npm packages like @pyscript/plugin-pydom

[WebReflection]

FYI even polyscript if re-embedded twice will have the same issue about dual bootstrap ... I will work on this to avoid ever doing that in both projects.

[e-nikolov]

My question to you and @JeffersGlass is what are you looking for in @pyscript.core outside of polyscript? Is it the stdlib/pydom? the plugins?

I tried moving to @pyscript.core because it exports the Hooks and has them attached to the XWorker. Also the exported typings are more accurate in @pyscript.core.

[WebReflection]

I am tackling all the issues discussed in here in this MR https://github.com/pyscript/pyscript/pull/1800

[ntoll]

"Let many flowers bloom", as @ntoll is fond of saying

I feel seen @JeffersGlass :tulip: :grin: But I'd add that sometimes we need to cultivate the flower beds to make sure the right flowers are blooming in the right place. :wink:

For the sake of my understanding...

• Folks want to bundle PyScript into their third party JS/TS projects. This is "a good thing" :tm:. :+1:
• If several such projects are used on the same page, unexpected or broken things may happen. :boom:
• Right now, we don't export the JS files needed for bundling PyScript to avoid such potential breakage. :no_entry_sign:
• We could do something about this, but it would be complicated, require a global leak and other "shenanigans" that the wider web community wouldn't appreciate. :grimacing:
• Perhaps folks only need PolyScript (although folks appear to prefer the API exposed by PyScript). :question:
• We, as a project, should agree a story about this context and document it. :memo:
• This is similar (but not quite the same as) plugins - for which we're also trying to get our story straight. :electric_plug:

Assuming I've summarised this more-or-less correctly... for the sake of a straw man solution (please feel free to take this apart so we can explore this specific problem space - I'm not precious, my aim is to get the best solution given our current understanding or capabilities), here are my naive questions / preferences:

• What is the current idiomatic solution in the web world for such a situation? Can we just do that?
• I believe very clear documentation and guidance is key, but for that to happen we need to have a clearly defined "story" to tell. Let's figure this out together before we actually commit to anything (but I like how we're experimenting over in #1800 - this will help clarify the conversation and may result in an acceptable solution... bit WiP :construction:).
• I like the idea of folks bundling PolyScript (in the same way PyScript bundles it, it feels to me that is the right thing folks ought to bundle) - but then we need to figure out what folks need from PolyScript that they're missing so prefer to try to bundle PyScript.
• Assuming the "just bundle a more refined PolyScript" is the way forward, what changes do we need to make to avoid problematic behaviour? As I see it we have two choices:
  1. Warn and fail when two projects bundling PolyScript appear on the same page. Is this idiomatic? I don't know. Let's discuss.
  2. Modify PolyScript so we mitigate the problems we've enumerated. But this ought to be done in an idiomatic way so the web world get to say, "looks legit" about our approach. I.e. we shouldn't bodge or hack together a half arsed solution that kinda works but smells like an old pair of socks. :socks: :nose:

Thoughts, explorations, answers and exceptional technical discussion most welcome. We've got this! But we need to explore a little first. :rocket:

[e-nikolov]

Folks want to bundle PyScript into their third party JS/TS projects.

To be fair, in my case I just assumed that I should bundle PyScript like all other dependencies. If there is a better way, I don't mind switching to it as long as it integrates well with the rest of the toolchain (IDEs like VSCode/TypeScript/yarn/vite, etc..).

although folks appear to prefer the API exposed by PyScript

I could not figure out from the documentation of polyscript how to start a Pyodide worker from JS with Hooks enabled. I think the part I was missing was that I need to use XWorker.call(new Hook(...), ...)

[ntoll]

@e-nikolov this is great feedback, thank you for letting us know (and your patience). Like I said, we need to get our story straight so it's clear from the docs how you do all the things you need to do. :+1:

[e-nikolov]

I may be mistaken, but I also get the impression that PyScript's main focus is to provide an easy way of creating UIs and interacting with the DOM from Python. It appears to be expected to start from a Python script tag like:

<script type="pyodide">
    import sys
    print(sys.version)
</script>

and then possibly start Python workers from the Python script. I imagine this is why bootstrapping is a concern when bundling PyScript.

In my case, however, I wanted to do all UI things in JavaScript, start some Python workloads in a worker and be able to send Python code from the JS main-thread to be executed by the worker. This way I can also avoid UI issues while Pyodide is loading. I also use WebRTC for peer-to-peer communications, which I think cannot be done with Python alone.

I could probably use Pyodide directly to start workers, but Polyscript's XWorkers have some nice features:

• support for multiple interpreters, e.g. pyodide, micropython, etc.
• configs for:
  • fetching files
  • preloading dependencies (Note: I think Pyodide recently added support for this too)
• the worker.sync proxy that simplifies the communications with the main thread.

Questions:

• Am I correct in assuming that using Polyscript only for the XWorkers should not be causing any bootstrapping issues when bundling it?
• Is supporting XWorker-first use-cases a design goal for Polyscript?
• Would it make sense to have a "@pyscript/xworker" package that is safe to bundle?

[WebReflection]

I.e. we shouldn't bodge or hack together a half arsed solution that kinda works but smells like an old pair of socks. 🧦 👃

Jut to clarify, I tend to never do that and I actually usually oppose to also do that in general ... now the rest ...

If several such projects are used on the same page, unexpected or broken things may happen

not anymore after this morning, at least not in Polyscript, and once #1800 lands not in PyScript neither.

That MR mitigates the problem but I cannot possibly solve it without leaking something on the global main thread ... so let me describe what is the issue and how does that solve it ...

The Issue

Because the standard way to write JS these days, and the suggested way too from all standards, is by using modules, we now have a situation where every project using modules (99% of modern JS projects) include a dependency and call it a day.

This means that the amount of duplicated code on the Web became exponential since nobody needs to add blocking scripts on top of the page and lurk for some leaked global on the window, but on the other hand if 2 projects include the exact same polyfill or the exact same library, this will be automatically duplicated and it will share nothing within the two version, even if these versions are the exact same code, as the bundling procedure is to guarantee your final product will work with or without that dependency around.

This implies that every polyfill MUST be defensive and very careful in not fixing the env multiple times, which means more code, and it doesn't fix the underlying issue.

Then how to fix the underlying issue?

By using importmap standard, which was born exactly to avoid duplicating same dependencies all over the place, and converge all imports to the same modules out of CDNs or simply by loading every single time the same module. In this scenario, the module never bootstraps or gets executed more than once, but its exports will be always and repeatedly available pointing at the exact same module every single time.

The elephant in the room is that bundlers don't care and so don't developers, except when libraries are massive, but even THREE.js or jQuery gets bundled infinite amount of times since everyone just switched to ECMAScript Modules in the last years, and this is a pity.

The Solution

Both polyscript and PyScript (after #1800 gets merged) now do something both wasteful yet clever:

• if the module is bundled multiple times in different projects, the first one will register a non configurable, non writable, and not enumerable Symbol("library-name") to the global, parking there its exports, effectively bypassing and replacing the whole point of importmap except it's done at the user-land level and not at the module registry.
• anyone bundling that module exports now will get either the only module on the page, or the first same module that already landed on such page. As the code speaks for itself, bundlers can't circumvent this approach without breaking everything so ... it's not the cleanest approach but, since there is no clean approach, it's good enough to me.
• both MutationObserver and customElement.define('py' / 'mpy') now never happens more than once, avoiding also awkward errors on the globally shared registry, as that's how Custom Elements work, so the current MR fixes that too.
• the only redundant code that runs is the one that brings in the whole module in the first place, as is dead code out of the box if the library was already on the page, or in the PyScript case the config resolution dance, but that's also not a real issue because the config doesn't do anything until it's passed down the active module, so it's very little, if not irrelevant, extra CPU/RAM that does little work to then result into being Garbage Collected ASAP if the module was already on the page.
• the solution and the importmap approach are not mutually exclusive, they both provide the exact same guard and work seamlessly together

Technical Details

The tiny library used to ensure the module exports are the same is called sticky-module and it uses the passed name to register itself as Symbol in the least leaking/greedy/clashing way, so that Symbol.for("polyscript") and Symbol.for("@pyscript/core") (note the convention around the unique module name in the npm registry) are just parked to inform the next bundled library (polyscript or PyScript) that exports are already defined so that there's nothing else to do.

This means that if a page has dozen polyscript form dozen different URLs, only the first one that landed on that page will bootstrap the MutationObserver and orchestrate everything else, including PyScript bundling polyscript in itself, as we kinda created the issue in the first place but that's also why I used relative exports, I didn't want others to have an easy life re-embedding PyScript in their projects.

But the community gotta do community things so I rather move forward with the defensive, yet working, approach, still suggesting to never bundle PyScript or Polyscript directly but we can't prevent early users to make mistakes, we can although help them not failing their expectations around our modules.

Last, but not least, the polyscript module now has an explicit exports entry that points to a file we can use to import everything we need, avoiding common bundlers to bail-out when PyScript is, for a reason or another, embedded in their code too.

[WebReflection]

P.S. hooks exported by PyScript are also shared like the PyWorker or any other future export we want, meaning that if a plugin wants to embed PyScript this will work regardless even in pages where PyScript is already there because the code basically does exactly what the importmap standard would do: ensure exports from a module are always the same once the module already landed.

[WebReflection]

@e-nikolov

I could not figure out from the documentation of polyscript how to start a Pyodide worker from JS with Hooks enabled

Hooks are first class citizen in PyScript which is why I suggest to use PyScript to handle these properly. In polyscript there are low-level primitives to make the dance work but it's complicated and admittedly not well documented because the effort to make hooks "easy-peasy lemon-squeezy" is in PyScript. polyscript really is an extremely low-level API/utility library but as you can see we are trying to offer a much better UX via PyScript around hooks, config, plugins, and so on.

That being said, you surely can import Hook from polyscript and pass an object with your own hooks but I am afraid I won't be too reactive if issues will come up, as it's responsibility of PyScript to make that story work as meant/needed, and it's via PyScript that we also change and adjust anything not working super well in polyscript ... so that using polyscript will always depend on the development of PyScript too, I hope you can understand where I come from here.

Questions

Am I correct in assuming that using Polyscript only for the XWorkers should not be causing any bootstrapping issues when bundling it?

you don't have this issue anymore with polyscript latest but you were not correct neither: in order to understand all various interpreters fueling possible scirpts on the page, polyscript needs to bootstrap a MutationObserver that initializes and takes over all scirpts found in the page or at runtime ... with previous versions that would've lead to MO race conditions and infinite amount of shenanigans, duplicated interpreters and so on ... but then again, latest is fine so you should not worry about it.

Is supporting XWorker-first use-cases a design goal for Polyscript?

just put a worker attribute in a script and you have first-class XWorker support ... the config story, hence hooks and other things, is handled properly by custom types as custom types are the next building block enabled by polyscript.

Would it make sense to have a "@pyscript/xworker" package that is safe to bundle?

it will always have the same racing condition issue ... nodes on the page need to be observed and not replaced, unless you are suggesting you want just that but then again you have it if you import only that in your project and latest doesn't suffer multiple bootstrap issues so it's already safe today to do that.

[e-nikolov]

just put a worker attribute in a script and you have first-class XWorker support ... the config story, hence hooks and other things, is handled properly by custom types as custom types are the next building block enabled by polyscript.

nodes on the page need to be observed and not replaced,

I am only using the XWorker() functionality of Polyscript, I don't use any script tags like <script type="pyodide"> and ideally there would be a way for me to disable the part of Polyscript that observes and replaces nodes on the page.

Or are you saying that even if I don't have any custom Githubissues.

Githubissues is a development platform for aggregating issues.