update to pyodide 0.24

[baclemen]

pyodide 0.24 has a range of new features. it would be cool if they would also be accessible if used in pyodide worker runner

[alexmojaki]

What happens if you just npm/yarn install pyodide 0.24 and pyodide-worker-runner together? It'd be good to update the default and test it, but ideally (hopefully already) pyodide-worker-runner shouldn't need an update every time pyodide updates.

[baclemen]

when i install it first, it just gets warnings. but when i use npm i the next time, this happens.

i think the reason why this happens is because it sees is as a "major" version update.

[alexmojaki]

Maybe the peer dependency version should just be a minimum (probably older than 0.23). What happens if you use --force or --legacy-peer-deps as suggested?

[baclemen]

--force turns the errors into warnings and makes it possible to install. With --legacy-peer-deps it just works, but i have to do it anytime i use npm install and this could also lead to dependency issues with other libraries.

[alexmojaki]

Right, I'm just asking you to confirm that pyodide-worker-runner is actually working in your app with the new pyodide. Then all we have to do is update package.json.

[baclemen]

yes it seems to work fine

[alexmojaki]

now you have to try 0.25 lol

[alexmojaki]

With both 0.24 and 0.25, I'm unable to build webpack inside the test folder. First I get:

Module not found: Error: Can't resolve 'node-fetch' in '/home/alex/work/pyodide-worker-runner/node_modules/pyodide'

If I install node-fetch, I get:

WARNING in ../node_modules/node-domexception/index.js 5:31-56
Module not found: Error: Can't resolve 'worker_threads' in '/home/alex/work/pyodide-worker-runner/node_modules/node-domexception'
 @ ../node_modules/fetch-blob/from.js 3:0-44 86:16-28
 @ ../node_modules/node-fetch/src/index.js 27:0-34:28 37:0-68 37:0-68 37:0-68 37:0-68 37:0-68 37:0-68
 @ ../node_modules/pyodide/pyodide.mjs
 @ ../lib/index.ts 13:0-61 23:47-57 26:29-40
 @ ./index.ts 11:0-39 34:27-40

WARNING in ../node_modules/fetch-blob/streams.cjs 29:19-36
Module not found: Error: Can't resolve 'buffer' in '/home/alex/work/pyodide-worker-runner/node_modules/fetch-blob'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "buffer": require.resolve("buffer/") }'
        - install 'buffer'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "buffer": false }
 @ ../node_modules/fetch-blob/index.js 6:0-22
 @ ../node_modules/node-fetch/src/body.js 12:0-30 137:13-17
 @ ../node_modules/node-fetch/src/index.js 17:0-47 193:13-18 370:2-15
 @ ../node_modules/pyodide/pyodide.mjs
 @ ../lib/index.ts 13:0-61 23:47-57 26:29-40
 @ ./index.ts 11:0-39 34:27-40

and

ERROR in node:buffer
Module build failed: UnhandledSchemeError: Reading from "node:buffer" is not handled by plugins (Unhandled scheme).
Webpack supports "data:" and "file:" URIs by default.
You may need an additional plugin to handle "node:" URIs.
    at /home/alex/work/pyodide-worker-runner/node_modules/webpack/lib/NormalModule.js:832:25
    at Hook.eval [as callAsync] (eval at create (/home/alex/work/pyodide-worker-runner/node_modules/tapable/lib/HookCodeFactory.js:33:10), <anonymous>:6:1)
    at Object.processResource (/home/alex/work/pyodide-worker-runner/node_modules/webpack/lib/NormalModule.js:829:8)
    at processResource (/home/alex/work/pyodide-worker-runner/node_modules/loader-runner/lib/LoaderRunner.js:220:11)
    at iteratePitchingLoaders (/home/alex/work/pyodide-worker-runner/node_modules/loader-runner/lib/LoaderRunner.js:171:10)
    at runLoaders (/home/alex/work/pyodide-worker-runner/node_modules/loader-runner/lib/LoaderRunner.js:398:2)
    at NormalModule._doBuild (/home/alex/work/pyodide-worker-runner/node_modules/webpack/lib/NormalModule.js:819:3)
    at NormalModule.build (/home/alex/work/pyodide-worker-runner/node_modules/webpack/lib/NormalModule.js:963:15)
    at /home/alex/work/pyodide-worker-runner/node_modules/webpack/lib/Compilation.js:1371:12
    at NormalModule.needBuild (/home/alex/work/pyodide-worker-runner/node_modules/webpack/lib/NormalModule.js:1253:32)
 @ ../node_modules/node-fetch/src/index.js 13:0-35 375:20-31 396:29-43 401:5-19 402:5-19
 @ ../node_modules/pyodide/pyodide.mjs
 @ ../lib/index.ts 13:0-61 23:47-57 26:29-40
 @ ./index.ts 11:0-39 34:27-40

[baclemen]

i just tested it with pyodide 0.25 and it worked fine on my application

[Hephaistos7]

I do think it would be wise to update the pyodide dependencies such that they include all versions after 0.23.2 instead of exactly that one. It is bad form to --forcedependencies, especially in bigger projects.

Your code does work for different versions of pyodide, so ideally the dependencies reflect this too.

Would you be willing to look into it?

[alexmojaki]

I tried in https://github.com/alexmojaki/pyodide-worker-runner/pull/35 and ran into some weird JS build problems. I can't remember what exactly and the logs have expired.