Closed simonw closed 2 years ago
Here's hello world in ASGI:
async def hello_world(scope, receive, send):
await send(
{
"type": "http.response.start",
"status": 200,
"headers": [
[b"content-type", b"text/plain"],
],
}
)
await send(
{
"type": "http.response.body",
"body": b"Hello, world!",
}
)
I'm going to try for a hello world service worker just written in JavaScript first.
From examining this demo: https://mdn.github.io/sw-test/ - service worker code here: https://mdn.github.io/sw-test/sw.js
It looks like the key to this will be the event.respondWith()
mechanism for the fetch
event, see this fragment:
self.addEventListener('fetch', (event) => {
event.respondWith(
cacheFirst({
request: event.request,
preloadResponsePromise: event.preloadResponse,
fallbackUrl: '/sw-test/gallery/myLittleVader.jpg',
})
);
});
https://developer.mozilla.org/en-US/docs/Web/API/FetchEvent/respondWith says:
The
respondWith()
method ofFetchEvent
prevents the browser's default fetch handling, and allows you to provide a promise for aResponse
yourself.
You can read event.request
to get at the request: https://developer.mozilla.org/en-US/docs/Web/API/Request
https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers is a great starting tutorial.
Service workers are restricted to running across HTTPS for security reasons. GitHub is therefore a good place to host experiments, as it supports HTTPS. In order to facilitate local development,
localhost
is considered a secure origin by browsers as well.
So I can test with a localhost
web server.
I got that working - first TIL is: https://til.simonwillison.net/service-workers/intercept-fetch
Starting to mess around with Pyodide now: https://pyodide.org/en/stable/usage/quickstart.html says:
Python code is run using the
pyodide.runPython
function. It takes as input a string of Python code. If the code ends in an expression, it returns the result of the expression, translated to JavaScript objects (see Type translations).
That sounds like an easy place to bridge between JavaScript and an ASGI app.
There's documentation on using it with web workers here: https://pyodide.org/en/stable/usage/webworker.html
I imagine that will work fine with service workers too.
I'm seeing if I can install Datasette using micropip
. Trying that here:
https://jupyterlite.readthedocs.io/en/latest/_static/lab/index.html
I ran:
import micropip
await micropip.install("datasette", keep_going=True)
And got this:
ValueError: Couldn't find a pure Python 3 wheel for: 'python-baseconv==1.2.2', 'click-default-group~=1.2.2'
I got close. I made my own wheels for python-baseconv==1.2.2
and click-default-group~=1.2.2
by cloning those repos and running python3 -m build
in them:
git clone https://github.com/semente/python-baseconv
cd python-baseconv
python3 -m build
Then I uploaded those wheels to my own S3 bucket with CORS enabled, see:
They are at:
It turned out I still needed to remove them from the setup.py
list in Datasette - I applied this diff:
diff --git a/setup.py b/setup.py
index 7f0562f..46bb6ad 100644
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,7 @@ setup(
install_requires=[
"asgiref>=3.2.10,<3.6.0",
"click>=7.1.1,<8.2.0",
- "click-default-group~=1.2.2",
+ # "click-default-group~=1.2.2",
"Jinja2>=2.10.3,<3.1.0",
"hupper~=1.9",
"httpx>=0.20",
@@ -57,7 +57,7 @@ setup(
"PyYAML>=5.3,<7.0",
"mergedeep>=1.1.1,<1.4.0",
"itsdangerous>=1.1,<3.0",
- "python-baseconv==1.2.2",
+ # "python-baseconv==1.2.2",
],
entry_points="""
[console_scripts]
Then I ran python3 -m build
and uploaded that wheel file to my bucket too:
Then in https://pyodide.org/en/stable/console.html (to ensure the latest Pyodide) I ran this:
Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import micropip
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
>>> await micropip.install(
"[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl)
[l](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl)"
)
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
>>> micropip.list()
Name | Version | Source
------------------- | --------- | ---------------------------------------------------------------------------------------------------------
------------------------------
python_baseconv | 1.2.2 | https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl
click | 8.0.4 | [https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-)
[d/click-8.0.4-py3-](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-)
<long output truncated>
sette | 0.61.1 | https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl
pyyaml | 6.0 | pyodide
packaging | 21.3 | pyodide
pluggy | 1.0.0 | pyodide
typing-extensions | 4.1.1 | pyodide
six | 1.16.0 | pyodide
markupsafe | 2.1.1 | pyodide
distutils | 1.0 | pyodide
micropip | 0.1 | pyodide
pyparsing | 3.0.7 | pyodide
>>> from datasette.app import Datasette
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/python3.10/site-packages/datasette/app.py", line 9, in <module>
import httpx
File "/lib/python3.10/site-packages/httpx/__init__.py", line 2, in <module>
from ._api import delete, get, head, options, patch, post, put, request, stream
File "/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module>
from ._client import Client
File "/lib/python3.10/site-packages/httpx/_client.py", line 9, in <module>
from ._auth import Auth, BasicAuth, FunctionAuth
File "/lib/python3.10/site-packages/httpx/_auth.py", line 10, in <module>
from ._models import Request, Response
File "/lib/python3.10/site-packages/httpx/_models.py", line 16, in <module>
from ._content import ByteStream, UnattachedStream, encode_request, encode_response
File "/lib/python3.10/site-packages/httpx/_content.py", line 17, in <module>
from ._multipart import MultipartStream
File "/lib/python3.10/site-packages/httpx/_multipart.py", line 7, in <module>
from ._types import (
File "/lib/python3.10/site-packages/httpx/_types.py", line 5, in <module>
import ssl
File "/lib/python3.10/ssl.py", line 98, in <module>
import _ssl # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
>>> import pyodide_js
>>> pyodide_js.version
'0.20.0'
>>> import ssl
>>> import _ssl
>>> from datasette.app import Datasette
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/python3.10/site-packages/datasette/app.py", line 14, in <module>
import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
>>> import pkg_resources
>>> from datasette.app import Datasette
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/python3.10/site-packages/datasette/app.py", line 29, in <module>
import uvicorn
File "/lib/python3.10/site-packages/uvicorn/__init__.py", line 2, in <module>
from uvicorn.main import Server, main, run
File "/lib/python3.10/site-packages/uvicorn/main.py", line 24, in <module>
from uvicorn.supervisors import ChangeReload, Multiprocess
File "/lib/python3.10/site-packages/uvicorn/supervisors/__init__.py", line 3, in <module>
from uvicorn.supervisors.basereload import BaseReload
File "/lib/python3.10/site-packages/uvicorn/supervisors/basereload.py", line 12, in <module>
from uvicorn.subprocess import get_subprocess
File "/lib/python3.10/site-packages/uvicorn/subprocess.py", line 14, in <module>
multiprocessing.allow_connection_pickling()
File "/lib/python3.10/multiprocessing/context.py", line 170, in allow_connection_pickling
from . import connection
File "/lib/python3.10/multiprocessing/connection.py", line 21, in <module>
import _multiprocessing
ModuleNotFoundError: No module named '_multiprocessing'
>>> import _multiprocessing
Traceback (most recent call last):
File "<console>", line 1, in <module>
ModuleNotFoundError: No module named '_multiprocessing'
>>>
So in that case it broke because Uvicorn is a dependency.
Interesting how running import ssl
before a line causes the error from a missing _ssl
to not show up any more - same for pkg_resources
.
I tried the same sequence in Safari and got a different error:
Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyodide_js
>>> pyodide_js.version
'0.20.0'
>>> import micropip
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
>>> await micropip.install(
"[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
[4-py3-none-any.wh](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
l"
)
File "<console>", line 2
"[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
[4-py3-none-any.wh](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
^
SyntaxError: unterminated string literal (detected at line 2)
File "<console>", line 1
l"
^
SyntaxError: unterminated string literal (detected at line 1)
File "<console>", line 1
)
^
SyntaxError: unmatched ')'
>>> await micropip.install(
"[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl)
[4-py3-none-any.whl](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl)")
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
ConsoleFuture exception was never retrieved
future: <ConsoleFuture finished exception=SyntaxError('unterminated string literal (detected at line 2)', ('<console>', 2,
5, ' "[https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)
[-8.0.4-py3-none-any.wh](https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.wh)', 2, 5))>
ConsoleFuture exception was never retrieved
future: <ConsoleFuture finished exception=SyntaxError('unterminated string literal (detected at line 1)', ('<console>', 1,
2, 'l"', 1, 2))>
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
yield self # This tells Task to wait for completion.
File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
future.result()
File "/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception
File "/lib/python3.10/asyncio/tasks.py", line 234, in __step
result = coro.throw(exc)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install
transaction = await self.gather_requirements(requirements, ctx, keep_going)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements
await gather(*requirement_promises)
File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
yield self # This tells Task to wait for completion.
File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
future.result()
File "/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception
File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
result = coro.send(None)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 245, in add_requirement
await self.add_wheel(name, wheel, version, (), ctx, transaction)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement
await self.add_wheel(
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 291, in add_requirement
await self.add_wheel(
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 316, in add_wheel
await self.add_requirement(recurs_req, ctx, transaction)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 276, in add_requirement
raise ValueError(
ValueError: Requested 'h11<0.13,>=0.11', but h11==0.13.0 is already installed
>>>
My hunch here is that I can probably do a custom build of datasette
(or datasette-core
) that excludes uvicorn
and get it working - if I build some extra wheels as seen above.
Idea: I could set up CI on a branch in the simonw/datasette
repo that builds a wheel and uploads it to an S3 bucket with CORS enabled created using s3-credentials
.
I still haven't managed to get Pyodide to load in a Service Worker.
This may have a clue:
I managed to get a debug shell up and running against the service worker via about:serviceworkers
in Firefox and then searching for localhost
- I'm seeing this error when I inspect it:
Maybe this example (found via GitHub codesearch) could help: https://github.com/kahowell/nutrition/blob/2b74d5a6fc79d8ba526017a0ee93bb0b22903c99/pyodideapp/sw.js#L4
No, that's caching Pyodide but not actually executing it.
My latest failing attempt looks like this:
importScripts("https://cdn.jsdelivr.net/pyodide/v0.20.0/full/pyodide.js");
let PYODIDE = null;
async function load() {
if (!PYODIDE) {
PYODIDE = await loadPyodide();
console.log(await pyodide.runPythonAsync(`
import sys
sys.version
`));
}
return PYODIDE;
}
self.addEventListener('fetch', async (event) => {
const request = event.request;
const url = (new URL(request.url));
if (url.pathname == "/" || /\.js$/.exec(url.pathname) || /\.tar$/.exec(url.pathname) || /packages\.json$/.exec(url.pathname) || /pyodide.*$/.exec(url.pathname)) {
// Don't intercept hits to the homepage or .js files
return;
}
let pyodide = await load();
// Pyodide is now ready to use...
let pythonVersion = await pyodide.runPythonAsync(`
import sys
sys.version
`);
const params = new URLSearchParams(url.search);
const info = {
url: request.url,
method: request.method,
path: url.pathname,
params: Array.from(params.entries()),
pythonVersion: pythonVersion,
a: 2,
l: 'thang: ' + loadPyodide
};
event.respondWith(new Response(
`<!DOCTYPE html><p>Hello world! Request was: <pre>${JSON.stringify(info, null, 4)}</p>`, {
headers: { 'Content-Type': 'text/html' }
}));
});
I'm going to try the code from this example: https://github.com/pyodide/pyodide/blob/6900afdc5bc6bcb791484a69ba0c10240d2730ed/docs/usage/webworker.md#web-worker
Managed to spot this error in Chrome DevTools:
Could it be that XMLHttpRequest
is not available in Service Workers but IS available in Web Workers?
https://stackoverflow.com/a/37114241/6083 confirms that XMLHttpRequest
is available in service workers but not in web workers.
https://web.dev/workers-overview/ explains the difference between service workers and web workers a bit, but doesn't mention XMLHttpRequest.
Filed an issue here:
Maybe I can do this all in a Web Worker instead? I could intercept browser history events and run code when they happen instead.
I built a Datasette wheel with the import uvicorn
bit removed from app.py
as well and it ALMOST worked...
Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import micropip
>>> micropip.list()
Name | Version | Source
--------- | ------- | -------
distutils | 1.0 | pyodide
micropip | 0.1 | pyodide
packaging | 21.3 | pyodide
pyparsing | 3.0.7 | pyodide
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl?",
keep_going=True
)
Traceback (most recent call last):
File "/lib/python3.10/site-packages/packaging/requirements.py", line 102, in __init__
req = REQUIREMENT.parseString(requirement_string)
File "/lib/python3.10/site-packages/pyparsing/core.py", line 1134, in parse_string
raise exc.with_traceback(None)
pyparsing.exceptions.ParseException: Expected string_end, found ':' (at char 5), (line:1, col:6)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
yield self # This tells Task to wait for completion.
File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
future.result()
File "/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception
File "/lib/python3.10/asyncio/tasks.py", line 234, in __step
result = coro.throw(exc)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 183, in install
transaction = await self.gather_requirements(requirements, ctx, keep_going)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 173, in gather_requirements
await gather(*requirement_promises)
File "/lib/python3.10/asyncio/futures.py", line 284, in __await__
yield self # This tells Task to wait for completion.
File "/lib/python3.10/asyncio/tasks.py", line 304, in __wakeup
future.result()
File "/lib/python3.10/asyncio/futures.py", line 201, in result
raise self._exception
File "/lib/python3.10/asyncio/tasks.py", line 232, in __step
result = coro.send(None)
File "/lib/python3.10/site-packages/micropip/_micropip.py", line 248, in add_requirement
req = Requirement(requirement)
File "/lib/python3.10/site-packages/packaging/requirements.py", line 104, in __init__
raise InvalidRequirement(
packaging.requirements.InvalidRequirement: Parse error at "'://s3.am'": Expected string_end
>>> await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
>>> micropip.list()
Name | Version | Source
------------------- | --------- | -----------------------------------------------------------------------------------------
------------
python_baseconv | 1.2.2 | [https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none](https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl)
[-any.whl](https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl)
click | 8.1.3 | pypi
click_default_group | 1.2.2 | [https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-](https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl)
[none-any.whl](https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl)
jinja2
<long output truncated>
sette | 0.61.1 | https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl
markupsafe | 2.1.1 | pyodide
typing-extensions | 4.1.1 | pyodide
packaging | 21.3 | pyodide
pluggy | 1.0.0 | pyodide
pyyaml | 6.0 | pyodide
six | 1.16.0 | pyodide
distutils | 1.0 | pyodide
micropip | 0.1 | pyodide
pyparsing | 3.0.7 | pyodide
>>> from datasette.app import Datasette
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/python3.10/site-packages/datasette/app.py", line 9, in <module>
import httpx
File "/lib/python3.10/site-packages/httpx/__init__.py", line 2, in <module>
from ._api import delete, get, head, options, patch, post, put, request, stream
File "/lib/python3.10/site-packages/httpx/_api.py", line 4, in <module>
from ._client import Client
File "/lib/python3.10/site-packages/httpx/_client.py", line 9, in <module>
from ._auth import Auth, BasicAuth, FunctionAuth
File "/lib/python3.10/site-packages/httpx/_auth.py", line 10, in <module>
from ._models import Request, Response
File "/lib/python3.10/site-packages/httpx/_models.py", line 16, in <module>
from ._content import ByteStream, UnattachedStream, encode_request, encode_response
File "/lib/python3.10/site-packages/httpx/_content.py", line 17, in <module>
from ._multipart import MultipartStream
File "/lib/python3.10/site-packages/httpx/_multipart.py", line 7, in <module>
from ._types import (
File "/lib/python3.10/site-packages/httpx/_types.py", line 5, in <module>
import ssl
File "/lib/python3.10/ssl.py", line 98, in <module>
import _ssl # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
>>>
>>> import ssl/
File "<console>", line 1
import ssl/
^
SyntaxError: invalid syntax
>>> import ssl
>>> import ssl/
File "<console>", line 1
import ssl/
^
SyntaxError: invalid syntax
>>> from datasette.app import Datasette
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "/lib/python3.10/site-packages/datasette/app.py", line 14, in <module>
import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
>>> import pkg_resources
>>> from datasette.app import Datasette
>>> ds = Datasette(memory=True)
>>> await ds.client.get("/")
Traceback (most recent call last):
File "/lib/python3.10/site-packages/datasette/app.py", line 1253, in route_path
response = await view(request, send)
File "/lib/python3.10/site-packages/datasette/views/base.py", line 134, in view
return await self.dispatch_request(request)
File "/lib/python3.10/site-packages/datasette/views/base.py", line 89, in dispatch_request
await self.ds.refresh_schemas()
File "/lib/python3.10/site-packages/datasette/app.py", line 350, in refresh_schemas
await self._refresh_schemas()
File "/lib/python3.10/site-packages/datasette/app.py", line 355, in _refresh_schemas
await init_internal_db(internal_db)
File "/lib/python3.10/site-packages/datasette/utils/internal_db.py", line 65, in init_internal_db
await db.execute_write_script(create_tables_sql)
File "/lib/python3.10/site-packages/datasette/database.py", line 113, in execute_write_script
results = await self.execute_write_fn(_inner, block=block)
File "/lib/python3.10/site-packages/datasette/database.py", line 144, in execute_write_fn
self._write_thread.start()
File "/lib/python3.10/threading.py", line 928, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
<Response [500 Internal Server Error]>
>>>
So the problem now is threads: Datasette tries to start them, and you can't do that in Pyodide.
The test routine is:
import micropip
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
import ssl
import pkg_resources
from datasette.app import Datasette
ds = Datasette(memory=True)
await ds.client.get("/")
OK well I got that to work!
Welcome to the Pyodide terminal emulator 🐍
Python 3.10.2 (main, Apr 9 2022 20:52:01) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import micropip
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
import ssl
import pkg_resources
from datasette.app import Datasette
ds = Datasette(memory=True)
await ds.client.get("/")
<Response [200 OK]>
>>> r = _
>>> r.text
'<!DOCTYPE html>\n<html>\n<head>\n <title>Datasette: _memory</title>\n <link rel="stylesheet" href="/-/static/app.css
?cead5a">\n <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">\n\n<link rel="alterna
te" type="application/json+datasette" href="http://localhost/.json"></head>\n<body class="index">\n<div class="not-footer">
\n<header><nav>\n \n \n</nav></header>\n\n\n\n \n\n\n\n<section class="content">\n\n<h1>Datasette</h1>\n\n\n\n\n\n
<h2
<long output truncated>
r detailsClickedWithin = null;\n while (target && target.tagName != \'DETAILS\') {\n target = target.parentNode;\
n }\n if (target && target.tagName == \'DETAILS\') {\n detailsClickedWithin = target;\n }\n Array.from(d
ocument.getElementsByTagName(\'details\')).filter(\n (details) => details.open && details != detailsClickedWithin\n
).forEach(details => details.open = false);\n});\n</script>\n\n\n\n<!-- Templates considered: *index.html -->\n</body>\n
</html>'
>>>
Changes are in this branch: https://github.com/simonw/datasette/commits/8af32bc5b03c30b1f7a4a8cc4bd80eb7e2ee7b81
But it's just this diff so far:
diff --git a/datasette/app.py b/datasette/app.py
index d269372..6c0c5fc 100644
--- a/datasette/app.py
+++ b/datasette/app.py
@@ -15,7 +15,6 @@ import pkg_resources
import re
import secrets
import sys
-import threading
import traceback
import urllib.parse
from concurrent import futures
@@ -26,7 +25,6 @@ from itsdangerous import URLSafeSerializer
from jinja2 import ChoiceLoader, Environment, FileSystemLoader, PrefixLoader
from jinja2.environment import Template
from jinja2.exceptions import TemplateNotFound
-import uvicorn
from .views.base import DatasetteError, ureg
from .views.database import DatabaseDownload, DatabaseView
@@ -813,7 +811,6 @@ class Datasette:
},
"datasette": datasette_version,
"asgi": "3.0",
- "uvicorn": uvicorn.__version__,
"sqlite": {
"version": sqlite_version,
"fts_versions": fts_versions,
@@ -854,23 +851,7 @@ class Datasette:
]
def _threads(self):
- threads = list(threading.enumerate())
- d = {
- "num_threads": len(threads),
- "threads": [
- {"name": t.name, "ident": t.ident, "daemon": t.daemon} for t in threads
- ],
- }
- # Only available in Python 3.7+
- if hasattr(asyncio, "all_tasks"):
- tasks = asyncio.all_tasks()
- d.update(
- {
- "num_tasks": len(tasks),
- "tasks": [_cleaner_task_str(t) for t in tasks],
- }
- )
- return d
+ return {"num_threads": 0, "threads": []}
def _actor(self, request):
return {"actor": request.actor}
diff --git a/datasette/database.py b/datasette/database.py
index ba594a8..b50142d 100644
--- a/datasette/database.py
+++ b/datasette/database.py
@@ -4,7 +4,6 @@ from pathlib import Path
import janus
import queue
import sys
-import threading
import uuid
from .tracer import trace
@@ -21,8 +20,6 @@ from .utils import (
)
from .inspect import inspect_hash
-connections = threading.local()
-
AttachedDatabase = namedtuple("AttachedDatabase", ("seq", "name", "file"))
@@ -43,12 +40,12 @@ class Database:
self.hash = None
self.cached_size = None
self._cached_table_counts = None
- self._write_thread = None
- self._write_queue = None
if not self.is_mutable and not self.is_memory:
p = Path(path)
self.hash = inspect_hash(p)
self.cached_size = p.stat().st_size
+ self._read_connection = None
+ self._write_connection = None
@property
def cached_table_counts(self):
@@ -134,60 +131,17 @@ class Database:
return results
async def execute_write_fn(self, fn, block=True):
- task_id = uuid.uuid5(uuid.NAMESPACE_DNS, "datasette.io")
- if self._write_queue is None:
- self._write_queue = queue.Queue()
- if self._write_thread is None:
- self._write_thread = threading.Thread(
- target=self._execute_writes, daemon=True
- )
- self._write_thread.start()
- reply_queue = janus.Queue()
- self._write_queue.put(WriteTask(fn, task_id, reply_queue))
- if block:
- result = await reply_queue.async_q.get()
- if isinstance(result, Exception):
- raise result
- else:
- return result
- else:
- return task_id
-
- def _execute_writes(self):
- # Infinite looping thread that protects the single write connection
- # to this database
- conn_exception = None
- conn = None
- try:
- conn = self.connect(write=True)
- self.ds._prepare_connection(conn, self.name)
- except Exception as e:
- conn_exception = e
- while True:
- task = self._write_queue.get()
- if conn_exception is not None:
- result = conn_exception
- else:
- try:
- result = task.fn(conn)
- except Exception as e:
- sys.stderr.write("{}\n".format(e))
- sys.stderr.flush()
- result = e
- task.reply_queue.sync_q.put(result)
+ # We always treat it as if block=True now
+ if self._write_connection is None:
+ self._write_connection = self.connect(write=True)
+ self.ds._prepare_connection(self._write_connection, self.name)
+ return fn(self._write_connection)
async def execute_fn(self, fn):
- def in_thread():
- conn = getattr(connections, self.name, None)
- if not conn:
- conn = self.connect()
- self.ds._prepare_connection(conn, self.name)
- setattr(connections, self.name, conn)
- return fn(conn)
-
- return await asyncio.get_event_loop().run_in_executor(
- self.ds.executor, in_thread
- )
+ if self._read_connection is None:
+ self._read_connection = self.connect()
+ self.ds._prepare_connection(self._read_connection, self.name)
+ return fn(self._read_connection)
async def execute(
self,
diff --git a/setup.py b/setup.py
index 7f0562f..c41669c 100644
--- a/setup.py
+++ b/setup.py
@@ -44,20 +44,20 @@ setup(
install_requires=[
"asgiref>=3.2.10,<3.6.0",
"click>=7.1.1,<8.2.0",
- "click-default-group~=1.2.2",
+ # "click-default-group~=1.2.2",
"Jinja2>=2.10.3,<3.1.0",
"hupper~=1.9",
"httpx>=0.20",
"pint~=0.9",
"pluggy>=1.0,<1.1",
- "uvicorn~=0.11",
+ # "uvicorn~=0.11",
"aiofiles>=0.4,<0.9",
"janus>=0.6.2,<1.1",
"asgi-csrf>=0.9",
"PyYAML>=5.3,<7.0",
"mergedeep>=1.1.1,<1.4.0",
"itsdangerous>=1.1,<3.0",
- "python-baseconv==1.2.2",
+ # "python-baseconv==1.2.2",
],
entry_points="""
[console_scripts]
Here's the proof of concept with a web worker: webworker.js
contains:
importScripts("https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js");
async function startDatasette() {
self.pyodide = await loadPyodide({indexURL : "https://cdn.jsdelivr.net/pyodide/dev/full/"});
await pyodide.loadPackage('micropip');
await pyodide.loadPackage('ssl');
await pyodide.loadPackage('setuptools'); // For pkg_resources
await self.pyodide.runPythonAsync(`
import micropip
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
from datasette.app import Datasette
ds = Datasette(memory=True)
`);
}
let readyPromise = startDatasette();
self.onmessage = async (event) => {
// make sure loading is done
await readyPromise;
// event.data has the incoming data - ignore for the moment
try {
let results = await self.pyodide.runPythonAsync(
`
import json
json.dumps((await ds.client.get("/-/versions.json")).json())
`
);
self.postMessage({ results });
} catch (error) {
self.postMessage({ error: error.message });
}
};
And index.html
contains:
<!DOCTYPE html>
<h1>Web worker demo</h1>
<script>
const datasetteWorker = new Worker("/webworker.js");
datasetteWorker.onmessage = (event) => {
console.log(event, event.data);
};
datasetteWorker.postMessage({"path": "/"});
</script>
Latest version:
<!DOCTYPE html>
<h1>Web worker demo</h1>
<script>
const datasetteWorker = new Worker("/webworker.js");
datasetteWorker.onmessage = (event) => {
console.log(event.data);
document.getElementById("output").innerHTML = event.data.text;
document.getElementById("status").innerHTML = event.data.status;
};
</script>
<form>
<p><input id="path" type="text" style="width: 80%" value="/.json"><input type="submit" value="Go"></p>
</form>
<p id="status"></p>
<div id="output"></div>
<script>
document.forms[0].onsubmit = function(ev) {
ev.preventDefault();
var path = document.getElementById("path").value;
datasetteWorker.postMessage({path});
}
</script>
And webworker.js
:
importScripts("https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js");
async function startDatasette() {
self.pyodide = await loadPyodide({indexURL : "https://cdn.jsdelivr.net/pyodide/dev/full/"});
await pyodide.loadPackage('micropip');
await pyodide.loadPackage('ssl');
await pyodide.loadPackage('setuptools'); // For pkg_resources
await self.pyodide.runPythonAsync(`
import micropip
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/python_baseconv-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/click_default_group-1.2.2-py3-none-any.whl",
keep_going=True
)
await micropip.install(
"https://s3.amazonaws.com/simonwillison-cors-allowed-public/datasette-0.61.1-py3-none-any.whl",
keep_going=True
)
from datasette.app import Datasette
ds = Datasette(memory=True)
`);
}
let readyPromise = startDatasette();
self.onmessage = async (event) => {
// make sure loading is done
await readyPromise;
console.log(event, event.data);
try {
let [status, text] = await self.pyodide.runPythonAsync(
`
import json
response = await ds.client.get(${JSON.stringify(event.data.path)})
[response.status_code, response.text]
`
);
self.postMessage({status, text});
} catch (error) {
self.postMessage({ error: error.message });
}
};
Result:
Need to solve CSS. I might just hard code that into index.html
!
<!DOCTYPE html>
<link rel="stylesheet" href="https://latest.datasette.io/-/static/app.css?cead5a">
<div style="padding: 1em">
<h1>Web worker demo</h1>
<script>
const datasetteWorker = new Worker("/webworker.js");
datasetteWorker.onmessage = (event) => {
console.log(event.data);
document.getElementById("output").innerHTML = event.data.text;
document.getElementById("status").innerHTML = event.data.status;
};
</script>
<form>
<p><input id="path" type="text" style="width: 80%" value="/.json"><input type="submit" value="Go"></p>
</form>
<p id="status"></p>
<hr></div>
<div id="output"></div>
<script>
document.forms[0].onsubmit = function(ev) {
ev.preventDefault();
var path = document.getElementById("path").value;
datasetteWorker.postMessage({path});
}
</script>
I need to hook up event handlers so clicks on links and forms within that area are turned into web worker postMessage
calls. Then I need to hook up the HTML5 history API - or probably switch to ugly #
URLs because I don't want to have people get a 404 when they first visit a /foo/bar
page.
This prototype has served its purpose - I know that this is possible now. Moving this to a new repository and continuing the work in additional issues.
Eventual goal is to run Datasette. For the moment I just want a "Hello world" served from a Python ASGI app that runs in a service worker - and ideally can respond "Hello NAME" if you hit it at
/name
.