search

LMMS / lmms

Cross-platform music production software
https://lmms.io
GNU General Public License v2.0
8.08k stars 1.01k forks source link

JSON project format #3981

Open tresf opened 6 years ago

tresf commented 6 years ago

Proposal to add a new JSON-backed, ZIP-encapsulated project format.

The Problem

Possible Solutions

Mockup image

image

Benefits

Before

<?xml version="1.0"?>
<!DOCTYPE lmms-project>
<lmms-project version="1.0" creator="LMMS" creatorversion="1.1.3" type="song">
  <head timesig_numerator="4" mastervol="100" timesig_denominator="4" bpm="140" masterpitch="0"/>
  <song>
    <trackcontainer width="600" x="5" y="5" maximized="0" height="300" visible="1" type="song" minimized="0">
      <track muted="0" type="0" name="TripleOscillator" solo="0">

After (mockup)

{
  lmms-project: {
    version: 1.0,
    creator: "LMMS",
    creatorversion: "1.1.3",
    type: "song",
    head: {
      timesig_numerator: 4,
      mastervol: 100,
      timesig_denominator: 4,
      bpm: 140,
      masterpitch: 0
    },
    song: {
      trackcontainer: {
        width: 600,
        x: 5,
        y: 5,
        maximized: 0,
        height: 300,
        visible: 1,
        type: "song",
        minimized: 0,
        track: [
          {
            muted: 0,
            type: 0,
            name: "TripleOscillator",
            solo: 0,
JohannesLorenz commented 5 years ago

Did you have plans how to handle external resources? Is this specific point different between XML and json?

tresf commented 5 years ago

@JohannesLorenz it would just mimic the existing folder layout (samples, etc). I'd even go as far as to say presets should be stashed as separate files (instead of embedding them) so that they can be easily shared with others.

vlad0337187 commented 5 years ago

Good idea. Also I'd suggest to use YAML or CSON for this =)

tresf commented 5 years ago

From what I've read YAML is less common for data and more common for configuration. YAML becomes valuable if we have a lot of self-references.

Quoting j2y:

YAML vs JSON YAML is best suited for configuration where JSON is better as a serialization format or serving up data for your APIs.

YAML is by no means a holy grail or a replacement for JSON - you should use the data format that makes the most sense for what you are trying to accomplish.

But in some cases, YAML has a couple of big advantages over JSON, including the ability to self reference, support for complex datatypes, embedded block literals, comments, and more.

Write you configuration files in YAML format where you have the opportunity - it is designed to be readable and editable by humans.

JSON, in contrast, is only designed to be human readable - intentionally lacking features to support editing. Lets start with lack of comment support - this is intentionally left out of the JSON spec because its not what the format was designed for.

A big win for YAML is that it does support comments. This is very useful especially when you use it for configuration. For data interchange, many of YAMLs features lose their appeal.

YAML parsers are younger and have been known to be less secure.

JSON vs YAML JSON wins as a serialization format. It is more explicit and more suitable for data interchange between your apis.

JSON ships with a far simpler spec than YAML. You can learn JSON a lot faster than you can learn YAML, because it is not nearly as robust in its feature set.

YAML is a superset of JSON, which means you can parse JSON with a YAML parser.

Try mixing JSON and YAML in the same document: [..., ..] for annotating arrays and { "foo" : "bar"} for objects.

vlad0337187 commented 5 years ago

@tresf , I thought it could be human-readable =)

ExperiBass commented 4 years ago

Related thing: .lmmsrc.json

mnini commented 4 years ago

Yaml is best suitable, if you want to edit something or whatever, I suggest comments in projects files with ranges of instruments...

It's easier to script by keyboard, without playing audio, than tunin stuff in engine, because mouse clics are too far away to work with at certain parts of interface and entering numbers by hand trough lmms takes a lot of time...

I suggest you to not use JSON, because JSON is more horrific to edit then xml, t. javascript programmer When you delete list there's some comma issue and stuff like that, it's really horryfying, yaml is clear without any wrongness.

Veratil commented 4 years ago

If we go with YAML, yaml.org points to libfyaml being a spec 1.2 compliant (with 1.3 features it says). It's MIT so it's compatible with GPL. It uses CMake and gives instructions of how to include it.

Cons listed:

  1. Windows - libfyaml is not supporting windows yet.
  2. Unicode - libfyaml only supports UTF8 and has no support for wide character input.
musikBear commented 4 years ago

beside the technical considerations between formats of files, would this be a finite break with backward compatibility?

tresf commented 4 years ago

@musikBear the general consensus is to preserve backwards compatibility at all costs for the foreseeable future for any feature; mmpz would be supported still.

ExperiBass commented 4 years ago

so, if we move to yaml, will there be a way to convert existing project files/presets to yaml? imo keeping support for xml longer than a few versions would be a bit of a hassle, especially if something changed in how the file is structured. Maybe LMMS could automatically convert on project save?

vlad0337187 commented 4 years ago

I think it could auto-convert on project open. So there will be no need to save .mmpz any more.

Veratil commented 4 years ago

Maybe LMMS could automatically convert on project save?

I think it could auto-convert on project open. So there will be no need to save .mmpz any more.

Since we're also considering a file extension change (I assume from @tresf possible solutions and screenshot example), it probably wouldn't be an auto-convert on save, but only when using save as to explicitly specify the new format. Using the same extension is dangerous since we don't really have any header identifier in the file to say that it's this or that format. My opinion is to have a new file format which will be default on save as, but if using the older mmp format then it will still fall back to using that.

This means our save pipeline needs to be modified which will need to be tested in depth.

musikBear commented 4 years ago

@Veratil wrote:

My opinion is to have a new file format which will be default on save as, but if using the older mmp format then it will still fall back to using that. 💯

Absolutely 100% spot ON

ExperiBass commented 4 years ago

Using the same extension is dangerous since we don't really have any header identifier in the file to say that it's this or that format.

one way to avoid that is to attempt to read as yaml, and if it fails it falls back to xml and then converts on save. Would that not be a good way, and why not?

tresf commented 4 years ago

We aren't treating a single file extension as two separate formats. It's not worth talking about.

DomClark commented 4 years ago

I suggest comments in projects files with ranges of instruments

(I assume this means annotating all values, not just literal pitch ranges, given the objection in the comment to "entering numbers by hand through lmms".)

I'm against this.

I do like the idea of providing meta-information about the project file format though. I think JSON schema would be suitable here: a schema can be referenced just once at the root of a project file, we can keep a schema up to date as LMMS is developed, it can describe all possible aspects of a project file, and there are libraries for most common languages to use schemas and validate files against them.

When you delete list there's some comma issue and stuff like that, it's really horryfying

I agree that forbidding trailing commas is irritating (C++ has that issue with member initialiser lists which I don't like), but I think "really horrifying" is an exaggeration and this is a rather weak reason not to use JSON. It's not like YAML doesn't have nasty gotchas either: unquoted strings can end up being interpreted as different types in rather unintuitive ways; it's insecure (YAML libraries for multiple languages have or have had arbitrary code execution problems or even features) and we don't want to propagate malware; etc. Not to mention that it's considerably more complex than JSON.

I don't see the point of overcomplicating this; it's primarily meant to be a serialisation format after all, and JSON is far from illegible.

ExperiBass commented 4 years ago

about the trailing commas, doesnt the JSON library for C/C++ handle that? so you wouldn't need to worry about trailing commas anyways

musikBear commented 4 years ago

comments in projects files with ranges of instruments

I have thought about a kind of 'report-feature' Its just some loose thoughts.. The short explanation would be in the line of: Right-click a connection/ controller/ preset & (?) Select option 'To-journal' LMMS would then write a simple text-explanation of the item set-up. That could be done in project-notes, or it could be done in a new .Jour text-document

ExperiBass commented 4 years ago

@musikBear sooo... the help feature?

tresf commented 4 years ago

@GingkathFox @musikBear can we please stop speculative rhetoric? JSON (or YAML) is meant to improve the experience. It's NOT meant to make a project file a human editable text document. If a UI feature is needed (such as link-specific annotations) track that in a feature. It'll make it just fine into the project file regardless of the format.

I don't see the point of overcomplicating this

👍

I think "really horrifying" is an exaggeration and this is a rather weak reason not to use JSON

💯

mnini commented 4 years ago

Project files doesn't have to break backward compatibility. @Veratil I suggest that opening filetype can go opening by extension, changing new extension to .lp (lmms project) and make project file be folder of actual project, from which you can choose on and can be saved to another projects, and projects folders like these could be renderable whole at the time into multiple files. Each of those project folders could contain samples used included in zip, so everything to re-render on another machine would come with a project file, which is nowadays issue. Even if I download something from .lmms portal I have problem opening files.

@musikBear just notes file in zip directory with project is what I would suggest, and projects could be folders, remember some people have to transfer samples as individual files and this could help that.

@GingkathFox trialing commas are standard of JSON since it changed for saving one line change of github of JSON, however JSON from C++ library is not beautified and I would keep the white space, therefore I would use YAML, maybe some extended YAML, that would allow comments that are not project file, but "range" of value in the line, because then you could just write project file to make music, which would be iconic. Project files over 5MB renders for hours on normal machine, therefore I really think there's enough space to keep white space, it's literally nothing, even on slow DSL internet, and file size doesn't have to be hyper optimalized if it would be negative in one way.

DomClark commented 4 years ago

however JSON from C++ library is not beautified and I would keep the white space

Qt certainly can (https://doc.qt.io/qt-5/qjsondocument.html#JsonFormat-enum), as can RapidJSON (https://rapidjson.org/classrapidjson_1_1_pretty_writer.html). I'm sure the same goes for most other libraries.

mnini commented 4 years ago

Sorry my bad. But how you comment inside JSON?

Veratil commented 4 years ago 
{
  "comment": "This is a comment"
}

😁

mnini commented 4 years ago

Yes, but if you want to specify range for every variable it looks utterly retarded.

musikBear commented 4 years ago

.... 😞 Nooooooo Not U2 2 Github! Nooot U @mnini sigh Like all sites online, gitHub has complied to PC. A remark where the perfectly fine word re**d appears, will be marked as not PC, and then it will be blocked ....by or as a result of (yesssssss You guessed it..) PC.... That is the new reality enforced by Global Karen You see.... Some persons / individuals have taken it on themself, to police all other peoples posts, to make sure that no one say a 'word', that could offend Global Karen's very delicate persona . . :€ Aaaaand the last piece in this puzzling puzzle comes here: Github has be bought by .....Microsoft!

tresf commented 4 years ago

The moderation is done by humans. Let's keep this on-topic and civil please.

tresf commented 4 years ago

Yes, but if you want to specify range for every variable it looks utterly [****].

In regards to project formats being a home for in-depth documentation (such as descriptions and value ranges), the project file is currently not the place for it.

I agree that polluting the document for commenting purposes is a gross implementation, but that's not the purpose of a project file. Range is part of the instruments, knobs sliders and plugins and managed as such in code. Tracking this information in the project file is not currently a requirement in the software and likewise would need a strong argument for this to keep it in the design.

ExperiBass commented 4 years ago

i mean, i dont get all this about comments and range, a project would be literally this:

{
    "name": "This is totally a real project!",
    "instruments: [
        "tripleoscillator": {
            "triposcsetting1": "triposcsetting1value"
        }
    ]
}

obviously this is oversimplified, but thats beside the point. the point is you wont need comments. if youre poking at the project file manually, i think we can safely assume you know where to find everything. So heres what i think LMMS should do with project files:

  1. LMMS should check for legal values. We dont want the master fader to be +500 and completely destroy whatever is playing the project. Also, the project file shouldn't have to worry about ranges. The project files purpose is to save data. It doesn't (and shouldn't) care about what are legal values, and what the range of say, the w/d knob on a effect is.
  2. LMMS should not worry about commenting and prettifying. If the end user wants to edit it they can easily use VSCode and a plugin to make it human-readable. But LMMS should name each key something that explains itself, like, say, TripleOscillatorMasterVolume for the Vol knob or TripleOscillatorResoAttack for the reso filter.

will edit with more as i think of them

tresf commented 4 years ago

@GingkathFox agree with everything except prettifying. I always prettify by default and as long as there's no major file size penalty, will be my recommendation. Case and point, you just did it to prove your point. 😉 Minified adds a small amount of storage space for -- what I consider -- drastic degradation in readability.

messmerd commented 1 year ago

Here's something worth looking into: https://github.com/bitwig/dawproject

It's an open exchange format for DAWs, and allows project files that follow the format to be opened in any DAW that supports it (currently Bitwig Studio and PreSonus Studio One). It's XML in a ZIP container.