Implement a Server-Database

[JohnnyCrazy]

Hey,

As I opened rooms.json I already thought my browser crashed :stuck_out_tongue_winking_eye:

It needs some kind of Database, I would probably go for Redis since it has a fast access time and would fit into the project. (Although it's not good to use if the host-systems RAM is not high enough, alternative for Redis would be MongoDB)

If you accept PRs, maybe I can look into it and implement some stuff.

Greetings!

[pdaddyo]

Hi there,

Thanks for your feedback, and yes that JSON file is not for browsing of course :) The server loads it on first launch, and keeps all data in memory, writing a history file to disk every 10 minutes and saving state when it's closed or crashes.

This was a deliberate choice to keep things simple during development (I'm new to Linux, Node, Express, React, libspotify, Github etc, so didn't want to be fighting DB performance / write / sync issues on top of an unfamiliar stack).

I agree it seems "obvious" we'd need a database - but to be honest I'm not sure we do, for now. Happy to be convinced otherwise, but if we continue to grow in next few months I don't want more technology just for the sake of it - what would the advantages be for a system as dynamic as Soundbounce (I'm talking specifically about the data currently in rooms.json)?

Any database will be slower, and use more RAM than keeping all data in-process (unless the db has to go to disk to get the data, in which case the speed penalty will be very high, especially on our EBS-backed AWS instance), so I've not looked at this yet since what we've got is incredibly fast, and won't run out of the paltry 1GB RAM even with 10k users and 1k rooms (which is a pipe dream).

FYI we currently have almost 100 rooms, with around 7000 tracks in them, nearly 300 users and total data size is under 10MB.

However, having said all that, 2 aspects that I think where we could use redis in near future are: user sessions (since there's already a redis adapter for express), and logging actions (logins, track plays, chats etc) so we can do things like #61

I'll definitely accept PRs (the Android version was my first ever PR!), let's discuss this further and make Soundbounce as great as we can, all input is appreciated since I'm new to most of this tech anyway ;)

Looking forward to discussing further.

Thanks,

Paul

[JohnnyCrazy]

While I totaly aggree with you about the current setup, I would see a huge potential in the future.

I agree it seems "obvious" we'd need a database - but to be honest I'm not sure we do, for now. Happy to be convinced otherwise, but if we continue to grow in next few months I don't want more technology just for the sake of it - what would the advantages be for a system as dynamic as Soundbounce (I'm talking specifically about the data currently in rooms.json)?

For now, yes of course, it's reliable and fast. I don't know what your plans are, but maybe at sometime there will be a website(with login etc.) and then we would need to implement some kind of database-storage. But I guess you're right, it will take some time until we get to this point.

[Westie]

You always need a database. Never, ever use plain old flat files. Not because it's "not modern enough" but, just urgh.

[Jmaharman]

You always need a database. Never, ever use plain old flat files. Not because it's "not modern enough" but, just urgh.

Don't worry @Westie, @pdaddyo knows what he is doing, it's actually his years of experience that has lead him to avoid the database for as long as possible, and to keep things simple.

10 years ago you would probably be forgiven for such a statement, as there was only one or two choices. There are so many databases, and types of databases, to choose from these days you are better off delaying that sort of decision until you know as much as possible. Especially when iterating early on in a project, of which this project is only about a month and a half old.

I'm sure pull requests would be welcomed for a database implementation should you wish to fill the void.

Keep up the great work @pdaddyo !

[pdaddyo]

Thanks @Jmaharman

I'm still not hearing a single answer for why we need a database now apart from "urgh" and "you always need a database". This is the first system I've ever built without a database from the get-go, and it's surprised me how effective it's been. You guys haven't stated an actual advantage to moving away from this setup yet - as I said I'm still happy to be convinced otherwise, but through technical merit, not dogma.

@JohnnyCrazy the system effectively is a website as it stands, and supports logging in, social features, realtime chat and of course synchronising music across the connected clients. This has been stable across thousands of sessions since we launched a few weeks ago.

As stated we will accept pull requests, so please get involved!

[Westie]

I was having a discussion with Paul in Indie ETC. and uh, he actually convinced me.

I still thinking that having a proper DB in there, either NoSQL or SQL based, would be good for stuff like API access (which, would be cool) but, after him showing me some stats, I'm quite amazed.

There, I said it. I've done a kind of u-turn.

[JohnnyCrazy]

Think we can all agree onto the following: Soundbounce currently doesn't need a Database, but 'maybe' sometimes in the future (for APIs etc.)?

[pdaddyo]

Thanks David, your bribe is in the post! Definitely agree about a DB for some future features, we'll look to community for input here (I'm thinking redis for user sessions, and to log every play, chat, vote, add, etc so we can do some processing "offline" like Top lists, stats page and so on)

On Fri Jan 02 2015 at 14:44:58 David Weston notifications@github.com wrote:

I was having a discussion with Paul in Indie ETC. and uh, he actually convinced me.

I still thinking that having a proper DB in there, either NoSQL or SQL based, would be good for stuff like API access (which, would be cool) but, after him showing me some stats, I'm quite amazed.

There, I said it. I've done a kind of u-turn.

— Reply to this email directly or view it on GitHub https://github.com/pdaddyo/soundbounce/issues/137#issuecomment-68530907.

[asabil]

You don't need a database, that would introduce too much latency without any real benefits. I would suggest splitting out the rooms state into separate files, basically 1 file per room, you load the file in memory when the room is accessed and you flush the state to disk every n minutes.

The Soundrop servers were implemented in Erlang, I would recommend looking at it. You could then model a room as a process and a user session as a process as well and keep things smooth, fast and very simple.

Hope this helps.

[pdaddyo]

Thanks for your input @asabil. When the homepage is loaded, all rooms need to be in memory in order to have the correct "now playing" track artwork show up, so lazy-loading rooms isn't an option for us I don't think.

Interesting to hear Soundrop backend was developed using Erlang - is there any chance that code could be open-sourced now that the service has closed?

I'm closing this issue for now, will re-open if we look at DBs again in future.