[Documentation]: Create Pyodide example for running PyNWB in the browser

oruebel commented 1 year ago

What would you like changed or added to the documentation and why?

It would be nice if we could create a short tutorial to show how one could run PyNWB in the browser with Pyodide. The example below shows that we can at least install PyNWB with Pyodide.

Basic example for installing PyNWB with Pyodide

<html>
  <head>
    <meta charset="utf-8" />
  </head>
  <body>
    <script type="text/javascript" src="https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js"></script>
    <script type="text/javascript">
      async function main() {
        let pyodide = await loadPyodide();
        await pyodide.loadPackage("micropip");
        const micropip = pyodide.pyimport("micropip");
        await pyodide.loadPackage("h5py");
        await pyodide.loadPackage("pandas");
        await pyodide.loadPackage("numpy");
        await pyodide.loadPackage("scipy");
        await pyodide.loadPackage("jsonschema");
        await pyodide.loadPackage("setuptools");
        await pyodide.loadPackage("ruamel.yaml");
        await micropip.install("hdmf");
        await micropip.install("pynwb==2.2.0");
        await pyodide.runPython(`
        import hdmf
        print("HDMF Version: %s" % hdmf.__version__)

        import pynwb
        print("PyNWB Version: %s" % pynwb.__version__)

        import h5py
        print("h5py Version: %s" % h5py.__version__)
        print("h5py Drivers: %s" % str(h5py.registered_drivers()))
      `);
      }
      main();
    </script>
  </body>
</html>

Example of the above HTML running in Firefox

Comments

Installing pynwb with micropip fails when not specifying the version due to version conflicts for h5py version ranges. This is probably due to micropip getting confused by the bad/yanked 2.5.1 PyNWB release https://pypi.org/project/pynwb/2.5.1/
The ros3 driver does not appear to be installed for h5py which would be nice to have for reading files from DANDI
I had to use the dev version of Pyodide for this to work. With the current 0.21.3 release micropip for h5py did not work
The install process can probably be simplified (i.e., not all loadPackage calls listed in the example are necessary). I mainly went through the requirements.txt for HDMF and try to load the packages one-by-one to see if all the dependencies work.
The above example does not actually load any NWB data file. It would be nice to show how to actually read from an NWB file either using streaming or from the file system (file system access in the browser is limited

Do you have any interest in helping write or edit the documentation?

Yes, but I would need guidance.

Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] Have you checked the Contributing document?
[X] Have you ensured this change was not already requested?

oruebel commented 1 year ago

Just to show that it is indeed possible to read an NWB in the browser with PyNWB in pyodided, here a slightly expanded example, which includes downloading a small file from DANDI and then opening it with PyNWB in browser. I'm not sure this is actually the best way to download/access data from DANDI, but I was mainly curious to see whether opening an actual NWB file would work. This uses runPythonAsync rather than runPython to be able to use await for the downloads.

Example HTML

<html>
  <head>
    <meta charset="utf-8" />
  </head>
  <body>
    <script type="text/javascript" src="https://cdn.jsdelivr.net/pyodide/dev/full/pyodide.js"></script>
    <script type="text/javascript">
      async function main() {
        let pyodide = await loadPyodide();
        await pyodide.loadPackage("micropip");
        const micropip = pyodide.pyimport("micropip");
        await pyodide.loadPackage("h5py");
        await pyodide.loadPackage("pandas");
        await pyodide.loadPackage("numpy");
        await pyodide.loadPackage("scipy");
        await pyodide.loadPackage("jsonschema");
        await pyodide.loadPackage("setuptools");
        await pyodide.loadPackage("ruamel.yaml");
        await micropip.install("hdmf");
        await micropip.install("pynwb==2.2.0");
        await pyodide.runPythonAsync(`
        # Check HDMF version
        import hdmf
        print("HDMF Version: %s" % hdmf.__version__)

        # Check PyNWB version
        import pynwb
        print("PyNWB Version: %s" % pynwb.__version__)

        # Check h5py version
        import h5py
        print("h5py Version: %s" % h5py.__version__)
        print("h5py Drivers: %s" % str(h5py.registered_drivers()))

        # Download a small file from DANDI for testing from dandiset 000126:
        # https://dandiarchive.org/dandiset/000126/0.210813.0327/files?location= 
        from pyodide.http import pyfetch
        nwb_testfile = "test.nwb"
        response = await pyfetch("https://dandiarchive.s3.amazonaws.com/blobs/11e/c89/11ec8933-1456-4942-922b-94e5878bb991")
        if response.status == 200:
           with open(nwb_testfile, "wb") as f:
              f.write(await response.bytes())

        # Try to open the nwb file
        from pynwb import NWBHDF5IO
        with NWBHDF5IO(nwb_testfile, mode='r') as io:
           nwbfile = io.read()
           print(str(nwbfile))
       `);
      }
      main();
    </script>
  </body>
</html>

Console Output showing the print of the loaded NWB file

bendichter commented 1 year ago

The ros3 driver does not appear to be installed for h5py which would be nice to have for reading files from DANDI

I don't think this is a huge deal anymore since I'd prefer to encourage the use of fsspec instead

bendichter commented 1 year ago

@oruebel This is really cool! I'd love to try out a demo using the dandi API to select a file and using fsspec to stream it. How do I test out this code? I tried putting it into an html file and opening it chrome, but that didn't work.

oruebel commented 1 year ago

I tried putting it into an html file and opening it chrome, but that didn't work.

That is how this example works. There are no actual page elements, the code in this example is being executed when the page is loaded. To see what is happening you will have to open the developer console in the browser and reload the page to the output log.

try out a demo using the dandi API

I don't think Pyodide supports sockets yet, i.e., the requests and urlib packages are probably not working. It's worth a try, but that may be problem for ffspec and the DANDI API. However, this seems to be on the roadmap for Pyodide https://pyodide.org/en/stable/project/roadmap.html#write-http-client-in-terms-of-web-apis

bendichter commented 1 year ago

it works!

oruebel commented 1 year ago

https://jupyterlite.readthedocs.io/en/latest/ would also be of interest here (although this should probably be treated as a separate issue)

oruebel commented 1 year ago

See also https://github.com/brainsatplay/webnwb/issues/4 for some additional discussion and comparison with WebNWB

NeurodataWithoutBorders / pynwb