Online Projects/Presets/Samples "Collaborato"

[tresf]

Closed due to lack of resources.

A feature initially developed by @tobydox that never made it to stable but recently resurrected as a topic via mailing list by "oe ai" (2014-04-28) is an enhancement request to have Project/Presets/Samples (et.al) downloadable/uploadable via the LMMS interface.

This bug report is a placeholder for this feature to track feasibility, initial efforts, major thoughts/concerns.

Note, the LSP rewrite should take special consideration as to accommodate this "Collaborato" feature.

Picture of pre-1.0.0 implementation provided for conceptual purposes only.

---

@diizy said on lmms-devel 2015-01-21

A reminder: before we add any online features to LMMS, we need to remember that any time something is connected online, it brings with it a whole bunch of new issues to worry about.

As long as LMMS is a purely local application, we don't need to worry about such things. The moment we allow LMMS to connect online, we accept a responsibility of handling the user's data over networks, and all the privacy & security concerns that come with it. Simply adding an online feature while ignoring those concerns is unacceptable.

So we really need a cost/benefit analysis here...

[tresf]

Amazon S3

If we leverage cloud storage for this, Amazon S3 offers a "Free Tier". Limits are listed below.

Anyone with an Amazon account can set up an Amazon Web Services (AWS) account and start trying the API, which leverages their own QueryAPI (SOAP is now listed as deprecated, REST or QueryAPI is recommended instead).

---

QT

According to Amazon's API listing, they do NOT offer direct API support for the C++ language (requiring manual REST/WebService bindings), however libaws, a 3rd party C++ project may simplify it: http://sourceforge.net/projects/libaws/

---

SilverStripe/CMS

A *nix web server can mount the AWS remote filesystem as if it is local using s3fs. Get s3fs here. Although s3fs may help gaining direct access to the files, Amazon's PHP/python APIs might prove more useful long term, depending on the design of the LSP.

---

Amazon Free Tier Details/Limits:

• 5GB Storage
• 15GB Transfer Out / Month
• Unlimited Transfers In
• 20,000 Get Requests / Month
• 2,000 Put Requests / Month

[eagles051387]

Taking another collaborative idea into account here. If need be I will file a seperate enhancement request for this. What do you guys think of allowing multiple people lets say you and a friend to work on the same project at the same time. Kind of like google docs does it?

On Wed, Apr 30, 2014 at 4:44 AM, Tres Finocchiaro notifications@github.comwrote:

A feature initially developed by @tobydox https://github.com/tobydoxthat never made it to stable but recently (2014-04-28) resurrected via mailing list by "oe ai" is an enhancement request to have Project/Presets/Samples (et.al) downloadable via the LMMS interface.

This bug report is a placeholder for this feature to track feasibility, initial efforts, major thoughts/concerns.

Note, the LSP rewrite should take special consideration as to accommodate this "Collaborato" feature.

Picture provided for concept only. [image: lmms-goes-online]https://cloud.githubusercontent.com/assets/6345473/2837149/e6089c14-d00f-11e3-898a-470978930f35.png

— Reply to this email directly or view it on GitHubhttps://github.com/LMMS/lmms/issues/661 .

Jonathan Aquilina

[Sti2nd]

That would be amazing, and probably impossible to implement without one of the best developers in the world?

[tresf]

That would be amazing, and probably impossible to implement without one of the best developers in the world?

Complexity and latency aside, the cloud's IO would cost thousands to fund if every knob twist or piano key was written back to the cloud (spiking from a reasonable dozens of events/day/client to an unreasonable thousands of events/day/client) . This "live collab" is how Google Docs works (you can see the collaborator typing in real-time) but Google Docs runs on Google Cloud Servers, so the data is from Google Cloud to Google Cloud. We don't have the luxury of this since our software does not (and likely won't - run in the cloud.... well... until LMMS is rewritten in JavaScript --- kidding).

The complexity of what Jonathan is describing (and bringing this conversation WAY off topic) can just as easily be implemented without the cloud. This "collaborato" feature is about sharing projects/presets/samples, etc. What Jonathan is describing is completely different as it would be live DAW collaboration. This would require queuing events and distributing them to clients via a predefined mechanism (probably direct socket communication to start). A basic server-client model could be written to support this (i.e. on a LAN) and if users demand cloud access, then THEY COULD BUY THEIR OWN :). Our funding would never allow this amount of traffic.

It's not an unreasonable goal from a technology perspective, but this also isn't the first time Jonathan has mentioned this in an unrelated thread.

Instead what I would like from Jonathan is collaboration on closing bug reports that involve his SilverStripe software, not blowing up bug reports with "conversation bombs" from his wish list. :)

[eagles051387]

I wasnt trying to take this thread off topic, Hope to continue working on those bugs this evening tres.

On Wed, Apr 30, 2014 at 5:07 PM, Tres Finocchiaro notifications@github.comwrote:

That would be amazing, and probably impossible to implement without one of the best developers in the world?

Complexity and latency aside, the cloud's IO would cost thousands to fund if every knob twist or piano key was written back to the cloud (from dozens of events/day/client to thousands/day/client) . This is how Google Docs works (you can see the collaborator typing in real-time) but Google Docs runs on Google Servers, so the IO is from Google to Google, which we don't have the luxury of doing (until LMMS is rewritten in JavaScript --- kidding).

The complexity in what Jonathan is describing (and bringing this conversation WAY off topic) can just as easily be implemented without the cloud. It would require queuing events and distributing them to clients via any mechanism (probably direct socket communication to start). A basic server-client model could be written to support this (i.e. on a LAN) and interested cloud clients could buy their own proxy to distribute this through a web service as our funding would never allow this amount of traffic.

It's not an unreasonable goal from a technology perspective, but this also isn't the first time Jonathan has mentioned this in an unrelated thread.

Instead what I would like from Jonathan is collaboration on closing bug reports that involve his SilverStripe software, not blowing up bug reports with wish lists. :)

— Reply to this email directly or view it on GitHubhttps://github.com/LMMS/lmms/issues/661#issuecomment-41807875 .

Jonathan Aquilina

[tresf]

I found a stellar article on Amazon Web Services which goes into detail about Amazon S3 Cloud storage. @eagles051387, please read this when you have time. http://www.smashingmagazine.com/2011/12/01/integrating-amazon-s3-wordpress/

This starts off with the management interface, but quickly dives into php as well.

The good news is the API is very straight forward and includes capabilities to store meta-data about files, which I think could be used for star ratings, genres, licensing, etc, however I fear this will require a lot of manual coding from our end, so please provide feedback in terms of direction.

[eagles051387]

Sadly at this stage, the manual coding is an unnecessary evil as its a new feature of LSP the manual coding is an unnecessary evil. I will read the article and revert shortly.

On Fri, May 2, 2014 at 2:25 AM, Tres Finocchiaro notifications@github.comwrote:

I found a stellar article on Amazon Web Services which goes into detail about Amazon S3 Cloud storage. @eagles051387https://github.com/eagles051387, please read this when you have time. http://www.smashingmagazine.com/2011/12/01/integrating-amazon-s3-wordpress/

This starts off with the management interface, but quickly dives into php as well.

The good news is the API is very straight forward and includes capabilities to store meta-data about files, which I think could be used for star ratings, genres, licensing, etc, however I fear this will require a lot of manual coding from our end, so please provide feedback in terms of direction.

— Reply to this email directly or view it on GitHubhttps://github.com/LMMS/lmms/issues/661#issuecomment-41972031 .

Jonathan Aquilina

[eagles051387]

I think from reading the article it sums up S3 very nicely. I have one fear though what is this going to cost in the long term. Do we start a fundraising campaign? I understand that we have 12 months free, but I think now wouldnt be an ideal time to start those 12 months seeing as LSP is still being reworked. I think once we have lsp2 up and running we can consider forging ahead and take advantage of that free year.

On Fri, May 2, 2014 at 2:25 AM, Tres Finocchiaro notifications@github.comwrote:

I found a stellar article on Amazon Web Services which goes into detail about Amazon S3 Cloud storage. @eagles051387https://github.com/eagles051387, please read this when you have time. http://www.smashingmagazine.com/2011/12/01/integrating-amazon-s3-wordpress/

This starts off with the management interface, but quickly dives into php as well.

The good news is the API is very straight forward and includes capabilities to store meta-data about files, which I think could be used for star ratings, genres, licensing, etc, however I fear this will require a lot of manual coding from our end, so please provide feedback in terms of direction.

— Reply to this email directly or view it on GitHubhttps://github.com/LMMS/lmms/issues/661#issuecomment-41972031 .

Jonathan Aquilina

[tresf]

I think once we have lsp2 up and running we can consider forging ahead and take advantage of that free year.

I understand that we have 12 months free, but I think now wouldnt be an ideal time to start those 12 months seeing as LSP is still being reworked.

You've lost me. Are we not re-writing it with cloud access in mind? What do you mean "still being reworked?" who is reworking it and why would we put time towards it if it would all change for AWS compatibility? What exactly are you proposing?

[eagles051387]

I think one thing that isnt 100% clear is this.

LSP database and all that is going to be migrated to the cloud correct.

[tresf]

I think one thing that isnt 100% clear is this.

LSP database and all that is going to be migrated to the cloud correct.

This depends on how far we get with a "Hello World". AWS supports meta data. What we need to determine (and we can't until we try out a cloud storage API) is how much of the database can simply be meta data around the files -- OR -- how much needs to reside in a relational database.

If you think of Collaborato as a feature within LMMS, we can't afford database connections back to MySQL to lookup this stuff, but would rather need the cloud to do as much as possible as to not DOS our website. :)

-Tres

[tresf]

I'm closing this enhancement task out as a "won't fix" due to lack of coordination and direction from a dedicated web team. Please feel free to re-open or reference this in the future when the proper resources are available. There's no reason for it to clutter the tracker at this time.

[tresf]

Cloud comparisons:

So I'm looking into a proper cloud storage for "collaborato" :) and I've come across a few options (please list any/all others).

• Google Cloud Storage
• Amazon S3
• EMC Atmos

Feel free to read the details below, but my initial analysis is to use Amazon since they offer a "Free Tier" which will cost nothing to startup and our bandwidth, storage and file operations will be capped per month.

From an API perspective, REST/SOAP seem to be defacto. Google provides a JSON API although .NET, Java and Python seem to be standard implementations. All APIs use similar terminology "DELETE/INSERT/LIST/ETC", and refer to data as "BUCKETS/OBJECTS/ETC"

Here's the pricing details. EMCs wasn't readily available.

Pricing:

• Google https://developers.google.com/storage/pricing; Storage: About $.02/GB/Mo (500GB would be $10) Bandwidth: DOWN: ~$0.10 - $0.20/GB (Varies based on region) UP: FREE File Operations: $0.01 per 10,000/1,000 (operating depending)
• Amazon http://aws.amazon.com/s3/pricing/; Storage: About $.03/GB/Mo (500GB would be $15) First 5GB Free using "Free Tier" Bandwidth: DOWN: ~$0.10/GB First 15GB/Mo Free using "Free Tier" UP: FREE File Operations: $0.005 per 10,000/1,000 (operation depending) First 20,000/2,000 free using "Free Tier"
• EMC http://www.emc.com/storage/atmos/atmos.htm; (hides its pricing behind an inquiry screen)

APIs:

• Google Java/JSON
• Amazon REST/SOAP
• EMC REST/SOAP Claims to also support NFS/CIFS

-Tres

[Wallacoloo]

I think this is a GREAT opportunity to play around with IPFS. It's more-or-less one giant, globally accessible distributed file system that combines a bunch of technologies developed in version control systems and torrent / peer-to-peer communications.

All that would have to be done is on the server side, install the IPFS daemon and mount IPFS to some directory (it behaves as an ordinary file system). Whenever something is uploaded to the LSP, place the public files in the IPFS directory. Then on the user side, they can also mount the IPFS directory to one of the LMMS directories. Then they can browse LSP samples, projects, etc, from within LMMS and we don't have to modify any code within LMMS. Because of the peer-to-peer nature, we shouldn't have any bandwidth issues - the user will cache everything they request and new requests are distributed to everyone else who has downloaded those files, much like torrenting.

[tresf]

@Wallacoloo,

Cool concept. The only issue is that currently our LSP is managed by a database, so the filesystem isn't really what most would consider a filesystem since the files are arbitrarily named and receive their actual file name only at download time.

-Tres

[Wallacoloo]

@tresf if I were to add IPFS support to the LSP, would it be welcomed? And if so, what would be the preferred way to implement it? I could implement it as a separate service which downloads newly uploaded files through the http interface and stores them in an IPFS folder (thus acquiring their correct name in the process), or I could add a hook into the upload process that copies the files to the file system before storing them in the database. The first is probably better since it has fewer dependencies - it could conceivably be modified to run on a different host even.

I have the week after next off, so I could probably implement this at some point. But I'd like to know if the devs would actually consider merging a feature like this in advance so that I don't end up wasting my free time. I assume this feature would be welcomed, based on the fact that this thread had a lot of discussion, but I've been wrong before so I'm a bit weary.

[tresf]

Good question. I think an LSP rewrite would be a better use of time, personally. The technology is attractive, but I'm not sure what type of demand we'd have for it. From an accessibility perspective, I think leveraging a webservice which lists the projects and presets directly in the software is much more robust and a better delivery to the user over a 3rd party plugin, personally. It's not much different than putting a copy of our files on DropBox or Google Drive, the clients still need to use the 3rd party service to receive them. The one you posted only helps with the cloud bandwidth problem -- i.e. cost -- which may not even be a problem.

Currently @Lukas-W @Umcaruje and myself maintain the website, so we'd be the ones approving. I think it's a lot of work that could be better spent improving something more fundamentally broken, personally. :)

[tresf]

That said, if you decided to integrate and convert our LSP into a content-management-type front end (perhaps one that tied into our forums logins) that would be a nice win. Our LSP is in dire need of maintenance.

Also, if you do jump into the LSP database, we have some procedures for setting it up here:

https://github.com/LMMS/lmms.io/wiki/LSP-Installation

[Umcaruje]

I agree that this is a really cool idea, but I think that LSP should have a rewrite (something angular,js based maybe?), rather than patching the old php code, since using an existing framework would probably make maintaining a bit easier. That said, I realize there isn't a lot of manpower for that, so maybe patching the php code is the best that can happen atm.

[tresf]

@Umcaruje, note that @Lukas-W has converted the majority of the website to use some newer technologies which can greatly simplify the LSP code if we do a rewrite. He uses a php package manager called "composer" and it does some heavy lifting in terms of using 3rd party APIs, templating, custom URL routing, etc.

Someone with some time to learn these technologies could shrink the LSP code-base to 1/10th of its size and then start adding in features. :)