Portable Project files

[grindhold]

Sometimes i have wished for the ability to create portable projects. By this i mean a file format which the user can export to, that embeds all the project's 3rd party dependencies (samples, presets, instruments), so I can easily share a project I am working on with a friend. So my friend does not have to manually download stuff from the interwebs.

Of course there are some things to think about here:

• File size can get huge
• Crossplatforming of the embedded plugins
• Licensing of embedded plugins
• Plugins mean executable code which is being shipped to another persons computer. It may compromise that system's security and give opportunity for malicious use.

How do you see that? Would you want such a feature in the first place? How could the security problems be adressed?

[diizy]

On 11/01/2014 02:33 PM, grindhold wrote:

Sometimes i have wished for the ability to create portable projects. By this i mean a file format which the user can export to, that embeds all the project's 3rd party dependencies (samples, presets, instruments), so I can easily share a project I am working on with a friend. So my friend does not have to manually download stuff from the interwebs.

Of course there are some things to think about here:

• File size can get huge
• Crossplatforming of the embedded plugins
• Licensing of embedded plugins
• Plugins mean executable code which is being shipped to another persons computer. It may compromise that system's security and give opportunity for malicious use.

How do you see that? Would you want such a feature in the first place? How could the security problems be adressed?

Plugins shouldn't be embedded. That should be left up to the users to ensure they both have the right plugins. It'll just be trouble with plugins.

Embedding samples is another thing. I think what we need is a widget that lists all samples in the project, then gives the user the option to either embed them in the project or to save them as an external file.

When all the samples (and soundfonts?) are embedded, then the project file is basically portable.

[tresf]

We should combine feedback from #441 into this one. And close one or the other (I'd vote to keep this open, since @grindhold is more likely to start coding the feature)

As far as plugins, we'd need architecture and OS differentiation (with still no guarantee they will import) but I'd be a big fan of at least carrying over win32 VSTs, since they tend to be more portable than other plugin formats.

The major issue with any native code (i.e. DLL, SO, DYLIB) in a project file is that it can bring with it malicious code. For this reason, if we allow the packaging of said files, we should purposely disable the import of packaged plugins without deliberate user override. (To the likes of "Warning! This project contains native plugins which have been disabled.")

[tresf]

And in regards to SF2 files, I think this is a must, but it is a double edged sword. SF2s are so large that they are likely to tremendously inflate the mmp7z file size.

They are also more likely to be pirated via those means. Having a file size limitation on the LSP with admin-type override would probably mitigate this.

[diizy]

On 11/01/2014 03:41 PM, Tres Finocchiaro wrote:

We should combine feedback from #441 https://github.com/LMMS/lmms/issues/441 into this one. And close one or the other (I'd vote to keep this open, since @grindhold https://github.com/grindhold is more likely to start coding the feature)

As far as plugins, we'd need architecture and OS differentiation (with still no guarantee they will import) but I'd be a big fan of at least carrying over win32 VSTs, since they tend to be more portable than other plugin formats.

Not always though. Some VST's require installation and use specific data directories.

The major issue with any native code (i.e. DLL, SO, DYLIB) in a project file is that it can bring with it malicious code.

Yes.

I have to say I'm against allowing any kind of custom executable code in project files. Even if they come with a warning, that's not going to actually do any good... people, especially non-computer-savvy people, have been conditioned to ignore all warnings, leave all terms and conditions unread, and blindly click on all "next" buttons. Executable code in project files is exactly the kind of thing that has given Microsoft the reputation of unsecurity... I feel this is a can of worms that is best left unopened.

Users can easily sort out which plugins they need. It's less easy for them to sort out the 50 samples/soundfonts used in a project file, and their bundling does not create any kind of risk for the user, so I think only bundling samples and other non-executable data files would be the best course of action.

As for plugins, maybe we could see if it's feasible to identify the plugins by their name or serial number or whatever, instead of file/pathname. This way, if a project uses eg. "synth1" and the other person has "synth1" on their computer, the plugin will be found regardless of it being in a different directory. AFAIK our LADSPA plugins already behave this way...

[diizy]

On 11/01/2014 03:47 PM, Tres Finocchiaro wrote:

And in regards to SF2 files, I think this is a must, but it is a double edged sword. SF2s are so large that they are likely to tremendously inflate the mmp7z file size.

They are also more likely to be pirated via those means. Having a file size limitation on the LSP with admin-type override would probably mitigate this.

Let's not go that route either - it's not our job to police our users. I don't think pirated SF2-files are ever going to be an issue - people want to distribute them, they have much, much, much better and easier ways to do that than using LMMS project files. Plus I feel like at this point, most SF2-files are pretty much abandonware...

File size shouldn't be a concern anyway - people are going to have to somehow distribute those soundfonts anyway, if they want to collaborate with them, so it doesn't really matter if there's one big file or lots of small files, except that one big file is much more convenient to share.

Here's a comparsion... Adobe Indesign has an option to save a project as a "package", which creates a directory with the project file + all the dependencies. There's no size limit for the dependencies, which can pile up pretty big in a big project: think of a 300-page book with lots of print-quality bitmaps... what Indesign does is, when you save as this kind of package, it also bundles in all the fonts you used, and it displays an optional warning (one you can disable with a "don't show again" checkbox) that says some legal mumbo-jumbo about fonts and how you need to follow the relevant copyright laws etc.

I feel even this might be a bit overkill in our situation, but if the copyright mafia ever becomes a concern, that's something we can at least do to sort of cover our own ass, legal-wise.

[Sti2nd]

As for plugins, maybe we could see if it's feasible to identify the plugins by their name or serial number or whatever, instead of file/pathname. This way, if a project uses eg. "synth1" and the other person has "synth1" on their computer, the plugin will be found regardless of it being in a different directory.

Yes, I think plugins is just as equally important as the samples! (I like the new Kicker :zap: )

[Umcaruje]

Actually, a great python utility for making zip files out of projects has been around for some time. Its @grejppi's LmmsPackager. It supports only samples in the ~/lmms/samples directory, but it does the job good.

[musikBear]

I have to say I'm against allowing any kind of custom executable code in project files.

100 :+1: agree
Should never be done!

[unfa]

I think embeding samples and soundfonts into an .mmpp (MultiMedia Project Package/Portable) file would be a next step for LSP, as people would start using samples not shipped with LMMS. I think it would be great for cooperative projects.

As for shipping instruments - I think it would just mean trouble as Vesa wrote. Security and compatibility being the top killer issues IMO. 01-11-2014 13:33, "grindhold" notifications@github.com napisał(a):

Sometimes i have wished for the ability to create portable projects. By this i mean a file format which the user can export to, that embeds all the project's 3rd party dependencies (samples, presets, instruments), so I can easily share a project I am working on with a friend. So my friend does not have to manually download stuff from the interwebs.

Of course there are some things to think about here:

• File size can get huge
• Crossplatforming of the embedded plugins
• Licensing of embedded plugins
• Plugins mean executable code which is being shipped to another persons computer. It may compromise that system's security and give opportunity for malicious use.

How do you see that? Would you want such a feature in the first place? How could the security problems be adressed?

— Reply to this email directly or view it on GitHub https://github.com/LMMS/lmms/issues/1257.

[grindhold]

@unfa i had the same worries from the start. nevertheless i wanted to put it to discussion.

[Sti2nd]

Unless we have unlimited space for free probably not too realistic to make the lsp take any project bundles (unless they are under 1mb of course).

This discussion is about project sharing, right? And portable projects is one solution. That would be great, as it would solve the problem of the missing samples and sf2 files (and .pat?), but not for the plugins because of understandable reasons mentioned. So if the goal is to make a project file fully portable it would be smart thinking out how to automatically find the correct plugin? One can't add several paths for one sample, sf2, dll as of now, right?

[tresf]

They are also more likely to be pirated via those means. Having a file size limitation on the LSP with admin-type override would probably mitigate this.

Let's not go that route either - it's not our job to police our users. I don't think pirated SF2-files are ever going to be an issue - people want to distribute them, they have much, much, much better and easier ways to do that than using LMMS project files. Plus I feel like at this point, most SF2-files are pretty much abandonware...

I agree it is impossible to police from copyrighted material, however I find it unreasonable to allow 100MB projects uploaded to the LSP without some form of override in place.

In terms of SF2-files being abandonware, I'm not sure if that is accurate. From my experiences, it is still quite a popular format for professional musicians and more and more are being released (some paid, some free) every day: http://the-filmmusic-group.deviantart.com/journal/Free-HQ-Orchestra-Soundfonts-224143947

I think as more composing is moving to synthesis, the demand for soundfonts will only go up, and from experience, they tend to outperform the VST-wrapped equivalents in terms of performance and compatibility in DAWs.

I have to say I'm against allowing any kind of custom executable code in project files.

I don't disagree the dangers in hosting 3rd party executable code, but who says we have to host it? The convenience is there and to your point, it is not our job to police this. Perhaps you believe the risks outweigh the benefits of this.... If we ever gain the ability to bundle projects as well as read mmpz via the LSP, we should be able to blacklist them from our sharing platform, but bundling them as a fundamental feature has its merits from a sharing perspective. It is tremendously useful, even if we disable it by default. I would be a big fan of having this an option.

we'd need architecture and OS differentiation (with still no guarantee they will import) but I'd be a big fan of at least carrying over win32 VSTs, since they tend to be more portable than other plugin formats.

Not always though. Some VST's require installation and use specific data directories.

I'm not sure what you are arguing here. I said "tend to be", which is qualitative and implies "not always". If you look at LADSPA for example, they tend to be different binaries for each platform. VSTs -- specifically 32-bit VSTs -- do have a high rate of compatibility.

And in regards to the installation and specific data directories, that depends on the type of VST. VST effects don't recurse subfolders and there's no file browser to add it manually, so the ones that work well with LMMS currently tend to be those that are standalone DLLs. I personally won't use a VST unless it is available as a standalone DLL. I have about 30 of them currently and none of them were installed.

The larger -- more involved -- VST installations would certainly fall into the "go and install this first" category, but I wouldn't consider that the norm, and I would certainly hope that's not one of the major arguments for stripping it out of the bundle feature.

[tresf]

This discussion is about project sharing, right?

Well, that is the question behind the question, yes, but "portability" really is the feature we're talking about. This doesn't necessarily mean sharing, but it's probably the most common use of a portable project. Simply backing up a project (for recovery purposes, or to bring to another computer) is another good purpose for having it portable as well.

[diizy]

On 11/02/2014 11:18 PM, Tres Finocchiaro wrote:

I agree it is impossible to police from copyrighted material, however I find it unreasonable to allow 100MB projects uploaded to the LSP without some form of override in place.

Sorry, I misread your earlier post somehow... Yeah, for the LSP a size limit makes sense.

I don't disagree, but the convenience is there. If we ever gain the ability to bundle projects as well as read mmpz, we should be able to blacklist them from our sharing platform, but bundling them has its merits from a sharing perspective. It is tremendously useful, even if we disable it by default. I would be a big fan of having this an option.

Eh, I still think it's probably more trouble than it's worth.

The main issue I see is that even if the feature is only enabled "on-demand", someone could still use that feature to create malicious .mmpz files which could then be opened by an unsuspecting victim... and sure we could show warning screens but again, those aren't really all that useful. I just don't see a way to implement this feature in a way that would be both convenient and safe.

[follower]

Perhaps an interim step could be in creating a list of "dependencies" (like a Python/pip requirements.txt file) which could then be installed (either from files within a project package or from an existing site--using a similar process to something like Homebrew).

(And if the original site no longer exists perhaps it could prompt the person to "Make a noise like 'errrrr-wwwssshh-tnnnng'... Great! Now make a sound like an Cello playing an A1#..." and automatically record them. :D )

[ghost]

I have a proposal. What about creating a option Save as Portable Project which does the following:

1. Open a dialogue that request a PATH and DIRECTORY_NAME.
2. Instead of creating a .mmpz file or alike directly, it creates a directory within the given PATH and with the DIRECTORY_NAME.
3. It copies all dependants (samples and soundfonts) into that directory (or subdirectories of that: DIRECTORY_NAME/samples and DIRECTORY_NAME/soundfonts).
4. It stores the normally created .mmpz in that directory as DIRECTORY_NAME.mmpz with relative paths to the samples and soundfonts.

We also need to add a entry to the QDomDocument to define that it’s a portable project so it knows if we add new samples/soundfonts that they should be stored relative also. Same goes for saving a portable project it should purge removed samples/soundfonts.

Duplicate filename scenario:

Used samples:

a/kick.ogg
a/snare.ogg
b/kick.ogg
b/snare.ogg

Would become

samples/kick.ogg
samples/snare.ogg
samples/kick.1.ogg
samples/snare.1.ogg

[musikBear]

@CYBERDEV Maby this could be simplified. If we ask the question: When is Portable Projects useful The plausible answer would be : when people want to share So the recipient are getting a service! In that light, The recipient should be willing to do some manual stuff, in order to get the service. Imo the tasks LMMS need to do, is the

• file-localisation
• file-copy
• files-to-folder-packing These are all done on sender-side -Eg providing the service is as easy as to click your mouse, zip the folder and upload! The recipient of the upload, then takes responsibility for unpacking and moving the various elements of the provided folder, into all the correct places in his LMMS installation. That is done manually. If there are naming conflicts, the recipient has to solve these manually! This scheme would imo be sufficient for a Project-export feature.

[tresf]

Let's not over-talk this topic please. The feature is well defined. I see no argument for conflict resolution. If portable becomes a format, it looks in the following order: 1. Portable location 2. User location. 3. System location.

No popups, no manual steps, just a reliable exported project with all of its dependencies. Even the built-in samples should be bundled because the future of LMMS is not guaranteed to contain the same exact built-in samples.

[Gumichan01]

Hello guys,

I have developed a little program that basically does what it is expected in this issue (more or less).

It takes your project (.mmp, or .mmpz), retrieves the samples and the soundfont files used by the project and packaqes them in a .mmpk file (it is just a Zip file). It can also extract the package and configure the imported project file. The package can be shared, or used as a backup..

Some elements may be missing and some improvements can be done. I did not make this program as a potential module that tend to be integrated into the main project, but this could be a little Proof of concept I guess. Don't hesitate to provide feedback :-).

[tresf]

@Gumichan01 thanks however I'd recommend providing precompiled binaries if you want users to test and provide feedback on your utility. Shall you ever submit this upstream to LMMS, we'd be happy to work with you to make it a permanent feature. If the utility is easy enough to snag in precompiled form, we can even link it as a stop-gap solution. Sharing projects has a high demand and users would be happy to have such a stop-gap until this feature is developed upstream.

[IanCaio]

Doesn't #5735 covers this issue? I didn't realize there was an open issue about this so I didn't link it there

[tresf]

Doesn't #5735 covers this issue? I didn't realize there was an open issue about this so I didn't link it there

Good call. Perhaps @Gumichan01 would care to collaborate.

[Gumichan01]

@Gumichan01 thanks however I'd recommend providing precompiled binaries if you want users to test and provide feedback on your utility. Shall you ever submit this upstream to LMMS, we'd be happy to work with you to make it a permanent feature. If the utility is easy enough to snag in precompiled form, we can even link it as a stop-gap solution.

Ok, I will provide an .exe for Windows and an AppImage for Linux distributions.

Doesn't #5735 covers this issue? I didn't realize there was an open issue about this so I didn't link it there

Good call. Perhaps @Gumichan01 would care to collaborate.

I just read the PR. This bundle idea seems very interesting. I will test it.

[Gumichan01]

Hey,

@Gumichan01 thanks however I'd recommend providing precompiled binaries if you want users to test and provide feedback on your utility. Shall you ever submit this upstream to LMMS, we'd be happy to work with you to make it a permanent feature. If the utility is easy enough to snag in precompiled form, we can even link it as a stop-gap solution.

Ok, I will provide an .exe for Windows and an AppImage for Linux distributions.

It's done. You can find those binaries here. I hope this can help.
I also took a look on #5735 and left a comment.