search

qgis / QGIS-Enhancement-Proposals

QEP's (QGIS Enhancement Proposals) are used in the process of creating and discussing new enhancements for QGIS
116 stars 37 forks source link

Fix access to remote data sources in WebAssembly builds #294

Open wonder-sk opened 3 months ago

wonder-sk commented 3 months ago

QGIS Enhancement: Fix access to remote data sources in WebAssembly builds

Date 2024/03/21

Author Martin Dobias (@wonder-sk)

Contact wonder dot sk at gmail dot com

Summary

QGIS core library compiled to WebAssembly is showing a lot of promise, enabling QGIS code to be run in web browsers. At this point, it is possible to load a QGIS project and render the map entirely in the client's browser, without any server-side map rendering - see the demo website at https://qgis.github.io/qgis-js/ with some sample projects.

While many engineering challenges towards production-ready state have been solved already, there are still issues that need addressing. Among those issues are problems with data sources that require network requests from QGIS and GDAL code in order to work, such as:

Currently, QGIS projects with any of these data sources will simply fail to load. All of these data sources are commonly used in QGIS projects, so not having access to them is a significant limitation. The purpose of this QEP is to solve the problems related to network requests so that QGIS projects involving these load and render correctly.

Proposed Solution

The issues with network requests have several causes, with at least three different problems to solve:

Deliverables

Deliverables should be pull requests in qgis-js repository fixing the three issues outlined above. The work is successful when QGIS projects with vector tiles, raster tiles, WMS, WMTS or remote COGs can be loaded and visualized without errors.

Risks

This is slightly more risky project - as it can be seen from the proposed solution, some of the problems may need deeper investigation. Working with WebAssembly builds is not as smooth and polished as with native builds, and problems may be difficult to diagnose at times.

Nevertheless, it seems to be a problem worth tackling - especially as more and more traditional desktop apps are getting ported to web browser environment with the help of WebAssembly (Google Earth, Adobe Photoshop, Autocad, ...) to provide the functionality with less friction - no need to install anything!

Performance Implications

No negative performance implications are expected. On the contrary, loading of QGIS projects with WMS/WMTS layers may get faster.

Issue Tracking ID(s)

rouault commented 3 months ago

Deliverables should be pull requests fixing the three issues outlined above

In which repositories ? I'm in particular interested regarding the GDAL issues. Is the https://github.com/wonder-sk/wasm-gdal-sandbox fork a long term or interim solution? Not saying that upstreaming must be necessary a result of this work (it might be additional work to have a "web-assembly friendly curl emulation/replacement" in a state ready for upstream inclusion), but just to clarify the intended scope.

wonder-sk commented 3 months ago

In which repositories ?

I should have been clear about that, sorry - I meant pull requests in https://github.com/qgis/qgis-js repo.

Is the https://github.com/wonder-sk/wasm-gdal-sandbox fork a long term or interim solution?

I see it as an interim solution, a proof of concept, just to get things off the ground. There are many shortcomings of the current approach, so my preference would be to have curl re-implemented for WebAssembly/Emscripten, to avoid having to patch vsicurl in gdal. But of course that would be a larger undertaking (but it could benefit other projects).

vpicavet commented 3 months ago

While the topic is interesting, I do not see how it corresponds to the criteria for a QGIS Grant ?

wonder-sk commented 3 months ago

While the topic is interesting, I do not see how it corresponds to the criteria for a QGIS Grant ?

@vpicavet Could you please elaborate on why this would not correspond to the grant criteria?

The "Admissible project topics" section of grant proposal form states the following: "Proposals may be on any topic that you think is relevant and beneficial to the greater QGIS community. However, the grant programme does not accept proposals for the development of new features. Therefore, proposals should focus on improving the QGIS project infrastructure and polishing existing features."

To my best knowledge, this proposal is relevant and beneficial to QGIS community and it does not suggest development of new features...

andreasneumann commented 3 months ago

@vpicavet - I see QGIS wasm as a part of the QGIS ecosystem, just like QGIS Desktop, Server, Web Client or other parts of the (Q)GIS infrastructure. It is still in it's early days, but has a good potential and - once the basic problems are solved - it will trigger quite some investments in feature development - but it needs a bit of a "jump start" (initial aid) until it can be seen as a viable solution for future Online-GIS development. Once that is solved, I am pretty sure, that other QGIS users will chime (financially) in the future.

vpicavet commented 3 months ago

Adding COG/WMS/... support to QGIS WASM sounds like a feature implementation to me, that's why I am wondering if this would fit to a grant application.

andreasneumann commented 3 months ago

Adding COG/WMS/... support to QGIS WASM sounds like a feature implementation to me, that's why I am wondering if this would fit to a grant application.

No - the functionality is already there (from QGIS desktop). It is about fixing the issues and dead-locks and threading-issues to get this running inside wasm, and hence in the web browser, without installing it locally.

vpicavet commented 3 months ago

We probably have different views of what would be called a fix and a feature. This would need sitting around a drink to discuss more thoroughly though.

andreasneumann commented 3 months ago

I think that quite a few of the QEP proposals are sort of "edge cases" in this respect.

anitagraser commented 3 months ago

Thank you for submitting your proposal to the 2024 QGIS Grant Programme. The 2 week discussion period starts today. At the end of the discussion, the proposal author has to provide a 3-line pitch of their proposal for the voter information material. (For an example from last year check https://github.com/qgis/PSC/issues/58#issuecomment-1567892412)

nyalldawson commented 3 months ago

I'd argue that investment in stablising webassembly QGIS is a sensible risk management approach for the project. I'd personally say that there's a good chance that WebAssembly based builds of QGIS will become the dominant form of QGIS over the next decade, and I'd rather we proactively keep exploring this instead of having to scramble to catch up when the demand rises... :+1:

pathmapper commented 3 months ago

Here's a nice presentation from last week. Although it is in German, it also gives an impressive visual impression of the current status of WebAssembly QGIS:

https://media.ccc.de/v/fossgis2024-38328-qgis-js-qgis-im-browser-dank-webassembly

jjimenezshaw commented 3 months ago

There is a detail in PROJ: https://proj.org/en/9.4/apps/cs2cs.html#use-of-remote-grids PROJ can use remote grids via CURL (similar but different problem as in GDAL). I would like to develop proj in web assembly (standalone), but first I have to learn WA.

wonder-sk commented 3 months ago

There is a detail in PROJ: https://proj.org/en/9.4/apps/cs2cs.html#use-of-remote-grids PROJ can use remote grids via CURL (similar but different problem as in GDAL).

The automatic fetching of data from remote grids in PROJ is explicitly disabled in QGIS, because this could have negative performance impacts on rendering, and users are instead guided to get the whole grid downloaded.

Good point though about PROJ using CURL as well, as this could be another project to benefit from CURL port to WebAssembly... The ticket for CURL implementation in Emscripten does not show much progress so far: https://github.com/emscripten-core/emscripten/issues/3270