xabriel / listen-to-wikipedia

Live, generative music from Wikipedia edits
http://listen.hatnote.com
Other
1 stars 1 forks source link

Use EventSource/SSE via EventStreams stream.wikimedia.org instead of obsolete RCStream #1

Open xabriel opened 1 year ago

xabriel commented 1 year ago

Use EventSource/SSE via EventStreams stream.wikimedia.org instead of obsolete RCStream

This wold mean consuming the firehose. Is it too bad for non enwiki users? Is the stream compressed?

xabriel commented 1 year ago

@tchin25's vue-rewrite branch is now using the recentchanges stream.

This solves the issue of having to deal with server side like in the original Hatnote were they have a farm of web socket servers to serve different wikis. However, this comes at a heavy cost in terms of bandwidth:

hatnote bandwidth use with just enwiki: ~3.0 KB/s ( ~11 MB/h) hatnote bandwidth use with all 40 supported wikis, including Wikidata: ~13.0 KB/s ( ~47 MB/h)

recentchanges bandwidth use with just enwiki: ~50.0 KB/s ( ~180 MB/h) recentchanges bandwidth use with all 900+ supported wikis, not including Wikidata: ~50.0 KB/s ( ~180 MB/h)

As can be seen, hatnote's approach is quite lean in terms of data usage, while the recentchanges has to consume the whole firehose regardless of use. And note that recentchanges does not include Wikidata, as it is not available in that stream. It also does not include new user creation. AFAIK, there is no current public stream with new user events.