We must stop using "Location" tags to define relationships between items in our files

Issue type

File corruption

Description with steps to reproduce

This goes hand in hand with my previous ~~rant~~ issue about default offsets as part of the wider infrastructural changes that we need to undertake moving forward.

The problem

The Location class in our codebase is what we use to represent a certain location in the score, as defined by a staff, a voice, a time tick, etc. Nothing wrong with that, except that using it in our files to represent univoque one-to-one relationships between items at different points is a very bad solution which keeps causing a very large amount of problems. These are mainly in two areas:

Spanners (i.e. all those elements like slurs, hairpins, lines etc. which "span" a given duration)
Score/part links

Spanners

In our files, the start point and end point of spanners are written separately, and we tag the start with a <next> Location which indicates where the spanner will end, end the end with a <prev> Location which indicates where the spanner started. Essentially, we first write "here starts a spanner which will end (x,y,z) ahead of here", then go on writing whatever is in between, then write "here ends a spanner which started (-x,-y,-z) before here".

Reading a spanner from the file means reading the start point, storing it somewhere, until the end point is found, at which point we check if the <next> and <prev> tag match each other, in which case we can successfully reconnect the spanner and add it to the score. I find this to be quite a mad approach. Here are some of the problems arising from this:

It is extremely easy to mess something up and cause the locations to not match, in which case we fail to reconnect the spanner. That is the main culprit for the dozens of issue I've seen in the past of spanners disappearing when reopening a file.
We are effectively writing the same (x,y,z) information twice: once at the start once at the end, and we are doing so independently, which doubles the possibility of errors.
When we read the start of a spanner, we need to create and manage a temporary state that waits until we find its end. This forces us to have an entire infrastructure around this (namely the ConnectorInfoReader class) which is again very prone to bugs and mistakes.
Spanners that fail to be properly reconnected need to be deleted, and that has often caused nuclear explosions if something else had already stored references to them, which then become invalidated.

Score/parts

We write the score and each part in their own separate file. Items in the score (let's call them master items) are marked with the <linkedMain> tag. Items in the parts (let's call them part items) are marked with <linked> tags.

The fundamental problem is that we store no information in our files that lets us univoquely retrieve the master-item for any given part-item. What the <linked> tag does is simply tell our system that this part-item should be linked to a master-item in the score, and the way we find that link is by looking in the master score for an item of the same type and at the same Location as this. Again, this is not a good approach, because

We don't know that we are connecting the correct items, we are just guessing it based on their location, which can easily lead to errors.
It is very easy for a small mistake in one single Location tag to cause disastrous corruptions. See for example #23681

Solutions

Both cases share a common root cause: we are using the Location tag as a strange workaround way of defining one-to-one relationships between items (the relationship between the start and end point of a spanner, or the relationship between a part-item and its master-item).

There is a much simpler way of doing that: give items an individual ID number and use that when we need to cross-reference them. We already write items IDs now so this will not be too much work.

For spanners: just given the spanner an ID and say "here starts spanner 1234" and "here ends spanner 1234". This already allows us to get rid of the next and prev tags. (There's more to say about spanner writing but I won't do it here).
For master-part-items: just write in the part-item the ID of its master-item, that's it.

What is the latest version of MuseScore Studio where this issue is present?

This is an all time issue

This is a non-technical reply, but I wanted to add that this bizarreness affects the user throughout the editing process. I have encountered a broad range of bizarre events, and the longer the file, the more likely these bizarre events are to occur. Example: joining two measures (among other as-yet undetermined operations) in the early part of a long score can lead to all slurs, hairpins, ottavas, voltas being shifted by an arbitrary amount. Slurs move to magic empty spaces where they reside until you attempt to edit them (or close and reopen the file), and they vanish. This has happened to me numerous times in my most recent score project. I only caught it once. The other times, I either saved before I scrolled forward and realized what had happened, or assumed the orphaned slurs were just grace note slurs that had lost their minds, which happens all the time. Events like this cannot be undone (you must both catch the event, and have the presence of mind to close without saving, and reopen from the last save point) and virtually impossible to pin down for reports as bugs. The effects are catastrophic.

But on a daily level, the behavior of these objects under editing is counterintuitive and numbingly time-consuming to manage. take hand-adjusted slurs. When the score layout changes in any way in the region of such a slur, it is either reset and "auto place" is turned back on, or it is moved around as if for some reason its position and dimensions as fixed offsets on the page must be preserved. From day 1, I have not understood why positions of objects cannot be defined in relation to each other. When a user hand-places a symbol, s/he knows exactly what is meant. But this information cannot be communicated to Musescore. I can't say, "please attach this slur endpoint to the top front of the notehead instead of above the staccato dot. " Or, "on top of the accent mark, which should be directly above the notehead". Musescore has no idea why I place anything, and workflow of a professional quality engraving must be managed from start to finish with an understanding of the arbitrary effects of changes in measure flow on symbols with a custom length.

It is as if, instead of symbols being defined as distinct objects that can then -by virtue of that very fact- be put into relation to one another, they are just phantoms in space being pushed around haplessly.

.... and I have just read exactly that above. It wouldn't just make the code easier to manage, it would make Musescore vastly more powerful.

The problem with joining measures has been fixed (#23030) for 4.4.2, which is to be released in the next few days. Your other remarks are also good points. Technically, they are probably more related to https://github.com/musescore/MuseScore/issues/22622 than to this issue, but nevertheless thanks for highlighting these things.

well, that's definitely related, from a user point of view. Autoplace becomes increasingly unmanageable as scores become more complex, and the offset system IS baffling. But to me it has always felt like there was a core issue beneath all this. I do maths in a little app made in JavaScript, where everything is a global variable, and so making objects aware of both each other and the state of the applet is tricky. Even when I save state values in dummy variables, sometimes I still need to know exactly what order objects are updated, to know if data retrieved from the dummies is current. Data just slushes about unsafely because I can't create code objects to hide it, and I can't durably relate user-manipulable objects because data is just slushing about unsafely.

Obviously what you do here is much much more complicated, but to me the Object ID is the key out of prison. And then, on the user end, not only will it help us with more stable software, I think advanced users will be more than happy to supply additional dependency and parent information for free, and consider it a feature.

Perhaps I have misunderstood the thrust of this fix.

From day 1, I have not understood why positions of objects cannot be defined in relation to each other. When a user hand-places a symbol, s/he knows exactly what is meant. But this information cannot be communicated to Musescore. I can't say, "please attach this slur endpoint to the top front of the notehead instead of above the staccato dot. "

Tangentially, a similar desire has led me to propose #24504, to allow any line to attach to a note head. It would actually be a very powerful paradigm for the user to be able to make the anchor of any object another object, and to keep the offsets in relation to this new anchor. When you turn on formatting marks in MS Word, you can move the anchors around like this from paragraph to paragraph, and it makes some layouts much easier to achieve (and more resilient to reflowing text). Diagramming tools generally allow you to attach lines to other objects either by centre, or at one of eight points around the edge (some give much more control than that) and then wherever the objects are placed, the attachment follows. I could see this making score layout more accessible to new engravers, and a time saver for old hands. This would be a big chunk of work, but could have a great payoff and differentiate MuseScore from other packages. It would need fleshing out in a different issue or system, though 😄

We already write items IDs now so this will not be too much work.

I hope this is not about <eid>, because eid is not unique to an element and its value is copied during copy/paste.

musescore / MuseScore