Use SSE Events to push realtime data from the internal server to the webmap

[granny]

Overview

This pull request introduces the integration of Server-Sent Events (SSE) into our internal webserver, enabling real-time data updates for our webmap. Using SSE we can achieve seamless real-time updates, significantly enhancing the user experience.

Details

Here are the key features and functionalities that this pull request will bring:

1. SSE provides real-time player position updates, allowing the webmap to track and display accurate player positions as they rotate and run around.
2. SSE facilitates immediate marker updates, making sure that any adjustments or additions to markers are instantly reflected on the webmap.
3. In cases where SSE connections are closed or nonexistent, the system gracefully falls back to updating through the old system, which is done through files.
4. When an SSE connection is established, the frontend bypasses file-based checks entirely.

What if I want this but I have the internal server disabled ???

You can still get the advantages of this change if you use an external server. Just re-enable the internal web server under a different port and set up a reverse proxy that points only to the /sse endpoint. This means you can gain the functionality of real-time data updates while the server is running, while also keeping the map available to view, even if the minecraft server is offline.

/sse reverse proxy Caddy example:

```css map.example.com { root * /var/www/map.example.com/ file_server reverse_proxy /sse* localhost:8084 { flush_interval -1 } @gz { path *.gz } handle @gz { header Content-Encoding gzip } } ```

/sse reverse proxy NGINX example:

```css server { server_name map.example.com; root /var/www/map.example.com/; index index.html; location ~ \.gz$ { gzip off; # Ensure nginx doesn't double-compress add_header Content-Encoding gzip; } location /sse { chunked_transfer_encoding off; proxy_buffering off; proxy_cache off; proxy_pass http://:; } } ```

API Changes

Internally, layers used to update in a per-second basis, which meant everything was configured in seconds. This PR changes that so everything runs in ticks instead. I tried to make sure that all API methods that state "seconds" correctly return or set the update interval as seconds. More specifically, I've overloaded Layer#setUpdateInterval and Scheduler#addTask(...) with an extra boolean parameter called ticks that indicates whether to return or accept the interval as ticks.

LiveDataHandler.java

A LiveDataHandler class has been added to help with pushing data to the SSE endpoints. There is one global SSE handler that sends data to /sse, and each world has it's own handler that sends data to /sse/<world>. If you hit /sse/<world> with an invalid world, you'll receive a JSON error that lists available worlds (worlds that are disabled will not appear). This is not really API, so don't expect it to be stable if it gets changed in the future.

Layer.java

The following methods were added to the Layer class. Any created layers will by default return false when isLiveUpdate() is called, which will stop the layer from being pushed through SSE. If you believe that your custom layers should be able to update through SSE, then call setLiveUpdate(true) when creating your layer.

    /**
     * Get if this layer gets pushed through sse.
     *
     * @return true if being sent through sse
     */
    public @Nullable boolean isLiveUpdate() {
        return this.liveUpdate;
    }

    /**
     * Set whether to push this layer through sse.
     *
     * @param liveUpdate true to push this layer through sse.
     * @return this layer
     */
    public @NotNull Layer setLiveUpdate(@Nullable boolean liveUpdate) {
        this.liveUpdate = liveUpdate;
        return this;
    }

---

Download the pl3xmap jar for this pull request: pl3xmap-8.zip

[granny]

A preview of what it looks like when a player is moving around:

https://github.com/granny/Pl3xMap/assets/43185817/5e73c157-3902-4079-892b-3c90d32fa093

[zndrmn]

that is actually so cool, really hope this becomes a thing eventually

[Folas1337]

I can also only express my utmost interest in this. It looks like an absolute killer feature for this plugin 😎

[Folas1337]

Wohooo more progress 😄 If you need someone to test this out, feel free to get in contact with me.

[granny]

Wohooo more progress 😄 If you need someone to test this out, feel free to get in contact with me.

I'll most likely release a test build when I think it's ready for testing before I merge it into the main project! I'll make sure to mention you when that happens :)

[granny]

This PR is now open to testing by anyone who wants to experiment with it :) The link to the build is available at the bottom of the PR description.

It's best if you delete all your configs and web/ folder beforehand for proper testing.

CC: @Folas1337

Users

A new config option is available in all the layers configs in layers/ called settings.layer.update-interval-in-ticks. If it's set to true, then it treats the update-interval option's value as ticks instead of seconds. While markers can update at any tick under SSE, files will take at least a second before updating even if you set it to below 20 ticks.

Developers

Internally, layers used to update in a per-second basis, which meant everything was configured in seconds. This PR changes that so everything runs in ticks instead. I tried to make sure that all API methods that state "seconds" correctly return or set the interval as seconds. I've overloaded these methods with an extra boolean parameter called ticks that you can set to true if you'd like to get/set one of these methods as ticks instead of seconds.

You can also send your own SSE events by using the sendSSE method in the HttpdServer class like so: Pl3xMap.api().getHttpdServer().sendSSE("eventName", "{\"key\": \"value\"}"). It's not really meant to be used by developers yet, but more work may be done in the future to allow developers more freedom to mess with the frontend.

[granny]

The plugin now reads the json files if SSE ends up taking longer than one second to send data. It will also only send data if said data has been modified since the last SSE update. This should help players who have poor internet or intermittent internet issues.

[U5B]

Would it be possible to adjust the update interval on the clientside? For some players, they might like the 1 second interval movement since it is less jittery.

[granny]

Would it be possible to adjust the update interval on the clientside? For some players, they might like the 1 second interval movement since it is less jittery.

That might end up being it's own PR where I add client-side settings that get stored in localStorage, we'll see.

[granny]

Recent changes

1. the SSE endpoint has been split up into /sse and /sse/<world>. If you hit /sse/<world> with an invalid world, you'll receive a json error that lists available worlds (worlds that are disabled will not appear).

2. isLiveUpdate and setLiveUpdate(boolean liveUpdate) methods have been added to the Layer class. Layers by default will return false to stop addons/plugins from working in SSE unless the creator specifies it themselves.

3. A new option has been added to all the built-in layer configs called settings.layer.live-update respective to the API change.

4. settings.layer.update-interval-in-ticks has been removed from built-in layers and settings.layer.update-interval has reverted back to being only in seconds (since it only really affects writing to the file and doesn't affect SSE updates).

[granny]

I'm pretty satisfied with the progress made in this PR. Pending any reported issues regarding code design or bugs resulting from these changes, this PR will be merged sometime next month. During this period, I would like some help from all of you in putting together Apache and NGINX configurations that will allow proper SSE functionality. I have already configured Caddy successfully (example in the PR description), so if anyone has a functioning setup with Apache or Nginx, please share it with me in the Discord server. (SSE prefers batch data transmission, which may hinder real-time updates.)

For those interested in conducting tests to ensure consistency, I recommend enabling the debug option in the config file and observing how the logs update between the browser and the server. If properly configured, they should both update simultaneously. Please refer to the video below for a demonstration. If the server logs update instantly while those in the browser do not, it indicates a problem with the reverse proxy configuration.

Please report any issues with this PR in the discord server.

The auto-updating link to the build with the most recent changes is available at the bottom of the PR description.

https://github.com/granny/Pl3xMap/assets/43185817/9e14c3d9-9ca6-436a-8197-6d6560ef9b03

[Folas1337]

Next month is almost over 😛 Just wanna remind you of this and even though I had to switch from nginx to caddy, it's been running super smooth ever since 😄

I hope to see this merged soon!

[granny]

Will be merged into the first build for 1.20.6 :)