openclimatefix / open-source-quartz-solar-forecast

Open Source Solar Site Level Forecast
MIT License
59 stars 51 forks source link

PV forecast in browser - pyodide #48

Closed zakwatts closed 5 months ago

zakwatts commented 7 months ago

To investigate if pyodide (https://pyodide.org/en/stable/) can be used to run the a PV forecast in a browser application on WebAssembly.

zakwatts commented 7 months ago

@mrchrisadams I made this github issue for your comment. We can use this issue as place to discuss this.

mrchrisadams commented 7 months ago

Thanks @zakwatts!

I think for this to run in a browser, I think we'd need to check that the list of packages used by this quartz solar forecast would be compatible with Pyodide, and there is a list of packages pre-installed already in pyodide. you can see them, below, along with information about how to create pre-compiled wheels to make including new packages possible:

https://pyodide.org/en/stable/usage/packages-in-pyodide.html

Once something is in the browser, it should be possible to use the browser geolocation tools, to allow it to provide the latlngs. I think this would mean anyone in the world accessing the page in a modern browser would be able to perform a local solar forecast for where they are (as long as the data coverage for the necessary inputs is ok).

You can see an example of using the the browser's geolocation API in this notebook:

https://nextjournal.com/greenweb/experiments-with-the-free-marginal-carbon-intensity-from-wattime

mrchrisadams commented 7 months ago

I've added a post here on social media, to see if any further advice comes in

https://mastodon.social/@mrchrisadams/111900393206118727

mrchrisadams commented 7 months ago

Ok, I've had a bit more of a look.

The following packages are dependencies in this project:

pandas
xarray
pv-site-prediction
pydantic
huggingface_hub # only for evaluation
python-dotenv

If a package is python it works with pyodide "out of the box". If it uses some underlying C library like pandas, then there needs to be a step to compile it for WASM. Rust libraries that are consumed by python also work, like in pydantic's case, through the py03 library.

For this to be possible, you'd probably need to figure out how to make the pv-site-prediction library work in pyodide. that might be quite a mission. It's possible and the steps are outlined below for compiling in this way:

https://pyodide.org/en/stable/development/new-packages.html

However, this is definitely beyond my knowledge. If I bump into some WASM / pyodide ninja who can help, I'll update here, but I'm also aware I don't have that much time to commit to this idea, as cool as it might be.

I've made a note to close this issue at the end of the month if that helps keep this issue list easier to manage.

zakwatts commented 7 months ago

Hi @mrchrisadams! Thanks for looking into this and giving a detailed explanation of the issues with the dependencies.

for pv-site-prediction the packages used are (from the pyproject.toml: python = "^3.10,<3.12" scikit-learn = "^1.1.3,<1.2.0" pvlib = "^0.9.3" numpy = "1.23.5" xarray = "^2022.12.0" click = "^8.1.3" google-cloud-storage = "^2.7.0" zarr = "^2.13.3" dask = "^2022.12.1" h5netcdf = "^1.1.0" pyproj = "^3.4.1" fsspec = "^2022.11.0" gcsfs = "^2022.11.0" torch = {version="^1.13.1", optional=true} tqdm = "^4.64.1" ocf-blosc2 = "^0.0.2" netcdf4 = "^1.6.3" pyresample = "^1.27.1"

After doing a bit of research the issues will mostly likely be with the libraries torch, pyproj, h5netcdf, netcdf4 having heavy C or C++ dependencies (and potentially google-cloud-storage and gcsfs).

Using pytorch in pyodide has been looked into before but it seems the large amount of dependencies involved might make it a large task. Also looks like there is an open issue on pytorch for it, but looks outside the scope of the core dev team. And yes if you bump into an WASM ninjas do let us know! :)

zakwatts commented 5 months ago

Closing this issue for now

zakwatts commented 5 months ago

@all-contributors please add @mrchrisadams for ideas.

allcontributors[bot] commented 5 months ago

@zakwatts

I've put up a pull request to add @mrchrisadams! :tada: