search

jamulussoftware / jamulus

Jamulus enables musicians to perform real-time jam sessions over the internet.
https://jamulus.io
Other
1k stars 224 forks source link

Server lists getting full #875

Closed pljones closed 3 years ago

pljones commented 3 years ago

Default: 150 - every time I've checked the last three days All Genres: 150 - every time I've checked the last three days Rock: 51 - over 50 most days Jazz: 59 - over 50 most days Classical/Folk/Choir: 136 - over 120 most days... I think over half of of these are Choir or choral...

I think we need to start thinking about what happens next. The original plan was lock down would end by last October at latest and life would go back to normal... It's unlikely to be back to normal this year.

So people can't set up "Default" or "All Genres" servers - which is irritating. I'm sure many people running servers don't think of them as having a specific genre. I don't - and I really don't want my server dropping of All Genres, having lost its Default list listing already... :)

ghost commented 3 years ago

Maybe the server list could leave room for temporary servers, rather than encouraging servers to stay connected 24/7 with a Bold font after 3 days. Otherwise, you will have server squatting (maybe you can make money with this feature).

corrados commented 3 years ago

Servers with version >=3.6.1 sending the version number when registering. So one solution would be to only allow servers with version >=3.6.1 to register in the Default and All Genres lists. That would also solve the issue that so many very old Jamulus server versions are present. But some server operators may be very disappointed by that.

Opinions?

corrados commented 3 years ago

@pljones maybe you could also start a thread about this >=3.6.1 in the Facebook group to get some feedback from there.

corrados commented 3 years ago

Classical/Folk/Choir: 136 - over 120 most days... I think over half of of these are Choir or choral...

We could split this list: one list for Classical/Folk and one for Choir. Would you be able to host the new list on your server?

pljones commented 3 years ago

Yes. Classical/Folk/Choir just hit 150..! (It was 146 and then a couple of seconds later, 150!)

pljones commented 3 years ago

@pljones maybe you could also start a thread about this >=3.6.1 in the Facebook group to get some feedback from there.

The joy of herding cats... I don't expect any input from server operators running older versions as I don't expect they're paying attention.

pljones commented 3 years ago

I've set up server list servers on 22124 through 22824 on my machine now.

They seem to use around 50M each, which is tiny. Didn't check the CPU but I doubt it's anything significant as there's no audio load. Similarly, I didn't check the UDP load per server.

corrados commented 3 years ago

Thanks. So let's go with a split. But the question is: Who has to move to the new list? Choral or the other? I would wote for Choral to stay and the others get a new list. But the problem is that the server operators choose which list they want to be. So the process of moving to the new list will take long...

ann0see commented 3 years ago

Can we think of some other kind of solving the problem with full server lists? Just adding more genre lists will not solve the problem in long term since it will be full in a few months.

Could we somehow find a solution so that a server registeres to a central server which then negociates which another central server is not yet full? This idea would need a rewrite in the client too.

If we go with the current solution, there's also another issue: old client versions will not have the new central server as default.

corrados commented 3 years ago

Just adding more genre lists will not solve the problem in long term since it will be full in a few months

Rock and Jazz lists have plenty of space left. Choral will be split which gives us a lot of headroom. In the Default list, a lot of very old servers are present which we want to get rid of. I do not see a need to make mayor changes to the list operation.

If we go with the current solution, there's also another issue: old client versions will not have the new central server as default.

What do you mean by this? Do you mean that older clients will not see the new lists? That is correct. But this was also the case when we first introduced the new lists. Usually, clients get more often updated since we have a clear message shown on the main window of the version is outdated.

pljones commented 3 years ago

Usually, clients get more often updated since we have a clear message shown on the main window of the version is outdated.

...if they've got a new enough version... :)

ann0see commented 3 years ago

Do you mean that older clients will not see the new lists?

Yes.

Rock and Jazz lists have plenty of space left.

Which means that there are less Rock/Jazz musicians who use Jamulus. Servers should be evenly spread across all lists if somehow possible. Nevertheless, this shouldn't end the genre "grouping" which might help musicians of the same genre to find eachother.

corrados commented 3 years ago

As written in the Facebook group, I have just setup the All Genres server list to only allow servers to register which have a version >=3.6.1. Let's see how many complains we'll get...

Edit: I just downloaded the 3.6.0 Jamulus version and tried to register to the All Genres list. This is what I got: grafik Which is actually not what I want. We have a status message "Your server version is too old" already built into the server. This is what should be shown instead. I'll fix that in the code now...

Edit2: Done. Now it looks better :-) grafik

pljones commented 3 years ago

If you're running headless, that message gets logged, too, doesn't it?

pljones commented 3 years ago

And also, if we're adding more server lists...

  1. Rename "Default" to "All Genres 1"
  2. Rename the current "All Genres" to "All Genres 2"
  3. Add a new server list for non-genre servers, "All Genres 3" (gives another 150 capacity for general usage and also makes it clearer there's no "preferred default" as such)
  4. Do the split: rename "Classical/Folk/Choir" to "Choir" and
  5. Add a new server list for "Classical/Folk"

I think already that the "Choir" genre might need splitting further - the only prevalent sub-genre I can see being "Barbers Shop" -- but currently that can wait.

(97 servers on Default with version < 3.6.1 looking on Jamulus Explorer, so yes - it'd be very sparse to filter them off...)

corrados commented 3 years ago

Yes, I agree. We should add two new lists. What about the name "All Genres"? You are a native speaker. Is this name appropriate or should we use a better name for it?

corrados commented 3 years ago

If you're running headless, that message gets logged, too, doesn't it?

Ask the one who has implemented it ;-).

pljones commented 3 years ago

What about the name "All Genres"? You are a native speaker. Is this name appropriate or should we use a better name for it?

I find it okay, "Genre" and "Sub-genre" for music are common enough terms. However, some people take the name to mean the genre-specific lists are filtered from the non-specific lists, so may be "Any genre" rather than "All genres". I'd also be interested to hear from the translators how easy it is to express that kind of difference.

If you're running headless, that message gets logged, too, doesn't it?

Ask the one who has implemented it ;-).

It was more to point out that it's not only shown on the GUI - "I know that, you know that - but it needs saying" ;).

pljones commented 3 years ago

Another quick check on "Default" (still 97 old ones):

There are a lot of linux "from git" servers there - I'd guess installed with SimonT's install script or some variant before he updated it so it pulled the latest tagged release. Many of those, the original installer won't know how to look at the logs as the instructions they followed didn't mention looking in logs. Eventually, if the current Default server list does expel them, the owners will come looking for help.

Before Default does expel anyone, I'd suggest finding as many places as possible to make it known it will happen at least a month before. (I don't think more than a month would make a difference - it's just long enough for people to see and find out how to upgrade.)

corrados commented 3 years ago

I'd suggest finding as many places as possible to make it known it will happen at least a month before.

I agree. But if we introduce two new lists, it is not urgent to modify anything in that Default list.

pljones commented 3 years ago

I agree. But if we introduce two new lists, it is not urgent to modify anything in that Default list.

Agreed - renaming from Default will help newcomers, too.

corrados commented 3 years ago

may be "Any genre" rather than "All genres". I'd also be interested to hear from the translators how easy it is to express that kind of difference.

I don't think we have a problem translating this. I like the new name. To sum up (changes are marked with bold font):

Default -> Any Genres 1 All Genres -> Any Genres 2 Any Genres 3 (hosted by corrados) Genre Rock Genre Jazz Genre Classical/Folk/Choral -> Genre Classical/Folk Genre Choral/Barbershop (hosted by pljones)

Do you agree to these changes? @ann0see, @gilgongo could you please also comment on this?

Edit: We should start the new server lists with "-f [3.6.1]" to make sure only servers with recent version numbers are able to register.

pljones commented 3 years ago

Just to note 23 servers would get expelled from Genre Classical/Folk if -f [3.6.1] were added there, too. (Not that I'm proposing to.)

Just to confirm - which port number do you want for each server?

Also, does this count as sufficient change to bump from 3.6.x to 3.7.x? With "custom central servers", of course, it doesn't actually break 3.6.x, just makes it a bit more awkward to use.

gilgongo commented 3 years ago

Looks good to me, although perhaps it should be "Any genre" not "Any genres" for the label..

gene96817 commented 3 years ago

I agree with @pljones that there will be confusion between "Any genre" and "All genres". I believe most Americans will casually interpret "any" as meaning "all" in this context.

corrados commented 3 years ago

Thanks. Here is the updated proposal:

Any Genre 1 (jamulus.fischvolk.de:22124) Any Genre 2 (jamulusallgenres.fischvolk.de:22224) Any Genre 3 (jamulusanygenre3.fischvolk.de:22624) Genre Rock (jamulusrock.fischvolk.de:22424) Genre Jazz (jamulusjazz.fischvolk.de:22324) Genre Classical/Folk (jamulusclassical.fischvolk.de:22524) Genre Choral/Barbershop (jamuluschoral.fischvolk.de:22724)

I could introduce new URLs: jamulusanygenre1.fischvolk.de:22124 and jamulusanygenre2.fischvolk.de:22224 and keep the old ones for compatibility.

corrados commented 3 years ago

I just created the anygenre3 server and the DNS entries. So, the following URLs are already active now: jamulusanygenre1.fischvolk.de:22124 jamulusanygenre2.fischvolk.de:22224 jamulusanygenre3.fischvolk.de:22624

The DNS entry for jamuluschoral.fischvolk.de:22724 is also setup but I guess pljones has to adjust the port number of his newly created Central Server to adapt to the new address.

corrados commented 3 years ago

The changes in Jamulus are also done and ready to be reviewed: https://github.com/corrados/jamulus/pull/885

pljones commented 3 years ago

They're all there and all pretty much the same apart from the port number. I keep the welcome message and server name generic :).

I added jamuluschoral.fischvolk.de:22724 to my Jamulus Explorer custom server lists and it returns the same as jamulus.drealm.info:22724 .... nothing at all ... as expected. (If it wasn't working, it would give an error.)

EDIT Um, no... Firewall issue. Now I get the real expected reply with just the server list.

EDIT2 Oh and pinging @softins for Jamulus Explorer.

passing commented 3 years ago

Can we think of some other kind of solving the problem with full server lists? Just adding more genre lists will not solve the problem in long term since it will be full in a few months.

Could we somehow find a solution so that a server registeres to a central server which then negociates which another central server is not yet full? This idea would need a rewrite in the client too.

If we go with the current solution, there's also another issue: old client versions will not have the new central server as default.

I think right now it makes sense to just add a few more central servers. But I also think it makes sense to think about alternative (technical) ways of handling this.

I can think of 3 obvious things that a Jamulus user wants to filter on:

Servers that have a low ping.

When I get the complete list of servers from a central server right now, there's lots of servers not relevant to me as they have a high ping. If there will be 10 "All genres" central servers in the future, I'd have to check them one-by-one even on each list of up to 150 servers just the first maybe 20% are relevant to me. I would prefer to have a UI where I could just filter by continent or by multiple countries and just see servers from these locations. I could then of course filter for my own country and some nearby ones. If I don't add a filter, there could be a limit on the number of servers I get.

Servers that have users connected

Having many central servers will make finding servers with connected users more time-consuming. I'd like to just hit a checkbox to see only populated servers, independent of different central servers they may use.

Servers for specific genres

Ignoring current technical limitations, I'd just like to have a filter for genres that works independent of my client version. Having different servers that group certain genres seems like a workaround. Ideally the central server just defines a list of available genres that a server registering to it can use (just like the country ID). If there's the need for adding new genres, that should be possible at the central server without having to update clients. A client connecting to a central server would get the list of available genres and could then search for servers with specific genres.

From how I think most users would like to use Jamulus as described, I think it would make sense to have just a single central server that just doesn't give each client the full list of servers connected, but always just a subset based on filters. Not sure if that is technically realistic - maybe it would require that a server can be a dedicated central server without being a "Jamulus Server" at the same time? - or maybe if separate servers are necessary, that could be transparent for the user somehow? But at least it would be beneficial if a client just needs to know one central point to connect to (which might also return the current list of central servers, in case it is technically not possible to get rid of that)

... hoping this is useful somehow :smile:

pljones commented 3 years ago

@passing, none of the above resolve the issue here - they're all client issues.

corrados commented 3 years ago

Actually, this Issues is resolved. You can still discuss alternativ solutions but we have introduced two new lists so we have enough space free for new servers. I'll close this now.

gilgongo commented 3 years ago

@passing There have been some discussions about alternatives here. You might want to read that and then contribute some of your ideas to that. We all I think agree that the current central server solution isn't ideal but it's quite complex, and I think also involves some non-technical things too.

softins commented 3 years ago

Oh and pinging @softins for Jamulus Explorer.

Thanks, I'll update today or tomorrow...

softins commented 3 years ago

Oh and pinging @softins for Jamulus Explorer.

Thanks, I'll update today or tomorrow...

Done

marcosscriven commented 3 years ago

@corrados - sorry to bug you on here, but I'm trying to get more context on what these server lists are for. I asked over in https://sourceforge.net/p/llcon/discussion/developerforum/thread/6d73cf7c1d/ - but not had a response. Please let me know the best context to ask such questions.

gilgongo commented 3 years ago

@marcosscriven Sorry not to have replied.

I essentially want to know what the bottleneck is for the lists, why it would be a few lists of 150 rather than a single discovery server that can handle millions.

I believe it's to do with network requests/load per IP, but perhaps one of the engineers can explain.

softins commented 3 years ago

@marcosscriven Firstly, we should mention that corrados is taking a break from the Jamulus project at the moment.

The Jamulus project started around 15 years ago, but really started to take off in popularity last year with the start of lockdowns due to the pandemic. The Central Server functionality did not originally exist within Jamulus, but was added much more recently. It provides two main functions:

  1. Allow Jamulus clients to find public servers and evaluate their network distance from the client.
  2. Some magic to enable the public servers automatically to open NAT sessions so that clients can ping and connect to them.

It is also possible to run a private server and manually give its IP address or domain name to invited guests, but in that case, the magic mentioned in 2 above does not happen, and it is necessary for the server operator to set up the necessary inbound port forwarding in any firewall or router that is in front of the private server.

The Jamulus protocol is currently purely UDP-based, and communication with Central Servers is performed using this protocol. There is a practical limit on the size of server lists, which is why a number of central servers are provided, distinguished by genre. Even with this limit, a list of servers requires a large UDP message, which needs to be split into fragments at the IP layer and reassembled at the receiving end. Although this is perfectly valid IP protocol, some routers or router configurations do not pass fragmented IP packets, resulting in empty lists for a small minority of clients.

Some of us would like to reimplement the server list/directory feature using TCP to overcome these limitations, but this is not a trivial undertaking, and has compatibility implications for existing versions of Jamulus.

pljones commented 3 years ago

The changelog comment needs updating. See https://github.com/jamulussoftware/jamulus/pull/885/files#r576413889

gene96817 commented 3 years ago

I don't know enough to anticipate the potential backward compatibility problems. 1- The server can send the server list as a series of UDP messages to eliminate the need for UDP fragmentation. 2- since all the servers carry a genre tag, why doesn't the server simply manage one (large) list and let the client decide which genre tag to use for displaying the subset of servers.

pljones commented 3 years ago

Re-opened unnecessarily - it's been dealt with already.

softins commented 3 years ago

2- since all the servers carry a genre tag, why doesn't the server simply manage one (large) list and let the client decide which genre tag to use for displaying the subset of servers.

@gene96817 The servers don't carry a genre tag. If public, they just register to a central server of the operator's choice. Genre is just a way of classifying the public Central Servers. Each Central Server is independent of the others.

gene96817 commented 3 years ago

The servers register (or self-declare) with a genre. Isn't this functionally equivalent to having a tag bound to the server?

softins commented 3 years ago

The servers register (or self-declare) with a genre. Isn't this functionally equivalent to having a tag bound to the server?

No, not really. The server just registers to a central server. Some public central servers are designated as being intended for a specific genre, but that’s all. Some aren’t, and it’s also possible for people to run and register to their own central servers, without any concept of genre.

gene96817 commented 3 years ago

You are being too literal. The server administrator (owner?) determines the genre. "without any concept of genre" = any Some human determines the mapping. We can move the locality of that information to make list management easier.

marcosscriven commented 3 years ago

@softins - Thanks so much for the comprehensive response. To paraphrase, list sizes are limited by UDP packet sizes and consumer-level router support?

As a user, what's a little confusing for me is if you're not really interested in the lists at all. I only used 'public' server option for the hole-punching aspect because I couldn't be bothered faffing with port forwarding and firewall tweaks. In that sense I neither cared about genres, or even other servers. Once I finished my session, I shut down both the client and server, running on my laptop.

Moving lists/discovery to TCP sounds like a good plan. Have you heard of ZeroTier lib? https://github.com/zerotier/libzt It's worked really well for me in the past.

While I'm here, an idea/suggestion (and again, I feel bad about hijacking a dev thread, so LMK if there's a better place): Upon reading about Jamulus a bit more from user's perspective, there seems to be consistent feedback that it works really well, but you need your local friendly geek to help you out.

Could the whole server/client thing be entirely hidden for smaller groups? Google tells me the average upload speed in UK is 6mbit, which looking at https://jamulus.io/wiki/Network-Requirements would seem to be enough for one person in the group to be running a server alongside their client, that could handle ~9 other users (10 including themselves) with medium stereo/buffer. I've heard of whole choirs/orchestras using this, but is that common? If every client also silently ran a server, used something like zerotier to find each other, and then automatically chose whomever had the highest upload speed to be the server, I think that could simplify usage.

gilgongo commented 3 years ago

@marcosscriven @softins

If every client also silently ran a server, used something like zerotier to find each other, and then automatically chose whomever had the highest upload speed to be the server, I think that could simplify usage.

Interesting sort of quasi-p2p idea! I'l leave it to engineers to comment though...

I couldn't be bothered faffing with port forwarding and firewall tweaks.

This reminds me that the whole server/client scale slots thing with Jamlus is complicated, and not just for technical reasons.

I think, but I do not know, that unless people are purely in the mood for jamming with strangers (a minority interest), they will seek to set up either a private server or try running a temporary public one. The latter of course risks the slot limit issue (although there is a 3rd party "disposable sever" service - can't remember its name...). If they hit the limit, they'll probably try harder to set up a private one. So there's a possible vicious cycle there.

There is also the (perhaps bigger?) issue of empty servers: nobody wants to join one of those, so seeking to allow more servers promotes a ghost town punctuated by very small numbers of full or nearly-full servers. This doesn't make Jamulus look good. So another vicious cycle there!

People have suggested some formula for booting idle servers off the lists. I can't remember what the counter-argument to that was exactly but it feels sort of wrong. There might also be the idea of gate-keeping public servers in some way. So we'd use --listfilter and only let known good operators to list. This would also allow us to manage numbers per city so that they don't promote the ghost town effect, as well as allowing us to be less interested in backward compatibility issues if we can rely on public server operators to do upgrades, etc.

The whole genre thing was just a hack to spread the demand for slots across servers. It wasn't a great scheme, but at least players and servers could be "matched" in some way that also solved a (minor) problem: what if I want to jam with rockers but others want to play folk music? But now, with the introduction of "Any Genre 1", "Any Genre 2" and "Any Genre 3" (in the forthcoming release), we are back to that problem again. Players have to switch between those lists for (apparently to them) no reason.

softins commented 3 years ago

softins - Thanks so much for the comprehensive response. To paraphrase, list sizes are limited by UDP packet sizes and consumer-level router support?

Basically, yes, I guess. And also because when a client requests a list, the central server also tells every server in that list to send an empty message to that client, to open the NAT session if required so that the client can ping each server. If you got to thousands in the list, that is a lot of requests! Not quite a DDoS, but hey :)

As a user, what's a little confusing for me is if you're not really interested in the lists at all. I only used 'public' server option for the hole-punching aspect because I couldn't be bothered faffing with port forwarding and firewall tweaks. In that sense I neither cared about genres, or even other servers. Once I finished my session, I shut down both the client and server, running on my laptop.

Yes, I understand that use case. I am actually interested in investigating whether we could add UPnP to the server, so it could automatically tell the router to open an inbound port. For private servers, this would lower the bar to entry (provided the user's router had UPnP supported and enabled).

Moving lists/discovery to TCP sounds like a good plan. Have you heard of ZeroTier lib? https://github.com/zerotier/libzt It's worked really well for me in the past.

No, never heard if it. Thanks for the link, I'll check it out.

While I'm here, an idea/suggestion (and again, I feel bad about hijacking a dev thread, so LMK if there's a better place):

If you want to bounce ideas around in general, rather than report a bug or request a feature, it's probably best to post it to the Discussions area https://github.com/jamulussoftware/jamulus/discussions

Upon reading about Jamulus a bit more from user's perspective, there seems to be consistent feedback that it works really well, but you need your local friendly geek to help you out.

I think that's true, and is likely to continue. Although once set up, many non-geeks have continued to use it without problems.

Could the whole server/client thing be entirely hidden for smaller groups? Google tells me the average upload speed in UK is 6mbit, which looking at https://jamulus.io/wiki/Network-Requirements would seem to be enough for one person in the group to be running a server alongside their client, that could handle ~9 other users (10 including themselves) with medium stereo/buffer. I've heard of whole choirs/orchestras using this, but is that common? If every client also silently ran a server, used something like zerotier to find each other, and then automatically chose whomever had the highest upload speed to be the server, I think that could simplify usage.

For small private groups, I think it's easier just for the person with the best connection and the right router setup to run up the server specifically. The server has its own GUI quite different from the clients'. For larger groups, the overwhelming consensus is to use a cloud-based server.

gene96817 commented 3 years ago

Could the whole server/client thing be entirely hidden for smaller groups? Google tells me the average upload speed in UK is 6mbit, which looking at https://jamulus.io/wiki/Network-Requirements would seem to be enough for one person in the group to be running a server alongside their client, that could handle ~9 other users (10 including themselves) with medium stereo/buffer. I've heard of whole choirs/orchestras using this, but is that common? If every client also silently ran a server, used something like zerotier to find each other, and then automatically chose whomever had the highest upload speed to be the server, I think that could simplify usage.

I think this will makes it harder to solve server performance problems because the server problem will be moving around. Getting new users running could be more difficult as this adds another possible failure mode to the user scenario. This could bring the scalability problem of JamKazam into Jamulus. At present, a small group of new users can just "borrow" an idle server. There is no need for them to invite server issues to their experience.