Preferences reordered alphabetically and comment lines removed

[mattst]

Note: Using Sublime Text 3 (Build 3059) on Linux Mint 13 LTS.

When Sublime Text changes a setting in the user Preferences.sublime-settings file the settings are all reordered alphabetically and comment lines are removed. This is, I believe, behaviour by design and not a bug, although it might be an unintended oversight.

~/.config/sublime-text-3/Packages/User/Preferences.sublime-settings

It can be reproduced very easily by:

1. Edit the user Preferences.sublime-settings file and move some settings around and add some comment lines.
2. Change the font size using the appropriate keys or using the Preferences menu.
3. The Preferences.sublime-settings file will be reordered alphabetically and any comment lines will be removed.

This is triggered by a variety of actions:

• Changing the font size.
• Changing the color scheme.
• Changing the dictionary.
• Any key or palette command which changes a user's preferences.
• Installing or removing any package using the widely used 'Package Control' package (which needs to disable and re-enable packages when a user is installing or removing a package).
• Any installed package which facilitates changing a user's preferences, e.g. 'ColorSublime'.

This is undesirable behaviour because:

• Comment lines, to remind a user what a particular setting does, get removed.
• Temporarily disabling a particular setting by commenting it out is not reliable.
• Settings grouped together for clarity and later ease of removal (e.g. settings for a plugin) are no longer grouped together — see example below.
• It forces some users to maintain a separate Preferences.sublime-settings file so that notes about settings can be kept — see same example below.
• It is disrespectful of the user's ability to maintain their own config files.

For example keeping a section like the one below is not possible / reliable.

// Set theme and related settings. Note that 'tabs_medium' and
// 'findreplace_small' are Predawn settings and not ST settings.
"theme": "predawn.sublime-theme",
"tabs_medium": true,
"findreplace_small": true,

I now maintain two Preferences.sublime-settings files, the one in my ST config directory and another located elsewhere which contains my groupings and comment lines. That way I can be sure to remove redundant settings when uninstalling a package or theme and provide myself with appropriate reminders in comment lines.

Please note that there are bug reports elsewhere concerning this behaviour. These have been created by users who have mistakenly blamed a particular package for this activity. There are 3 such issues listed in the Sublime Package Control github issue pages which I found having made this assumption myself.

I hope this is of interest and helps in some way. I completely adore ST and hope that this gets solved at some point. :)

[IanLee1521]

One additional thing that seems like it is probably connected is that the Preferences.sublime-settings seems to not retain the leading spaces when I save it and re-open it. It always reverts back to tab stops, despite the settings I have (namely "translate_tabs_to_spaces": true).

[mattst]

Yes, I've noticed that too.

...and blank lines should also be left in place so that users can maintain neat groupings around their settings and comments.

[eugenesvk]

mattst, thanks for such a perfect description of this bug! It's a pity it's still not resolved after almost one year!!! I'm new to ST3 and put comments to all the settings I've imported from somewhere else only to realize all were gone after I changed the font size :(. It indeed limits my ability to learn about possible settings by playing with the config file.

[rublev]

this is so brutal, how could something so simple not work? now i have to waste another half hour reconfiguring all my stuff and backing it up for whatever reason.

[ghost]

Nervewrecking issue.

[felixsanz]

+1, so annoying

[seanCodes]

+1, it would be really nice to see this fixed.

[Maartyl]

+1, it would be very much appreciated

[tzhouhc]

+1, would appreciate!

[alextegelid]

:+1:

[diegobit]

+1

[gschenck]

jps wrote (https://forum.sublimetext.com/t/dev-build-3101/17027/48) about this:

"Updates don't rewrite User Preferences, but any automatic changes, either from the UI (e.g., adding a word to the dictionary) or from the API (e.g., a plugin calling Settings.set() on the User Preferences) will cause them to be rewritten.

One work around here is to place your hand edited Preferences.sublime-settings file in a separate package (ZUser perhaps). This way the file will never be programmatically modified."

[titoBouzout]

I think this can be closed, as is expected that the editor could save own preferences. Still looks kinda a workaround, idk how to improve this.. maybe requesting to save the preferences in a kind of middle space idk

[eugenesvk]

@titoBouzout No one is arguing the editor should not be able to save the preferences. We are arguing it shouldn't delete comments and reorder all the settings during that process.

[titoBouzout]

Yes I understand that part, but the parser and thingy that the editor is using to read AND write to that file makes the other bits to disappears. Which is kind of expected. So the workaround is to use a ZUser folder package to add your preferences there and let the editor manage the User/Preferences, my suggestion was to open a new bug, for saving the "autogenerated" or "saved" preferences in another folder than User.. maybe DefaultAuto/

[eugenesvk]

@titoBouzout Re. the workaround with ZUser - would you please clarify how to make sure that the preferences file in ZUser is loaded after the one in /User? I'm a bit confused by this. When I just move the preferences file to the ZUser folder, it doesn't get parsed by Sublime, instead I get the default view. Likely I'm missing something simple in how to set up a ZUser package with a preferences file, but not sure what it is. Also a pity to lose the convenience of "ctrl-," to open preferences.

[titoBouzout]

If you have

• /Packages/User/Preferences.sublime-settings

you should have

• /Packages/ZUser/Preferences.sublime-settings

to open the preferences with the command, you can add (main menu - Preferences - Keybindings user ) the following:

{ "keys": ["ctrl+whatever"], "command": "open_file", "args": { "file": "${packages}/ZUser/Preferences.sublime-settings" } }

using the CTRL modifier you want.

[eugenesvk]

@titoBouzout that's what I have, but the ZUser file is not being picked up (it's a copy of the User for with only font size changed so I can easily see which one is bring loaded) From http://docs.sublimetext.info/en/latest/extensibility/packages.html#installing-packages I understood that the User folder is always read last, so adding a ZUser doing make a difference. How does it work for you?

[titoBouzout]

Works just fine, font size setting added in User but not in ZUser, all my ZUser preferences are there with comments, and these are picked. You may need to restart ST after doing the change of folder.

[eugenesvk]

@titoBouzout well, what you describe is the same — User has a higher priority than ZUser, so the Z is a rather confusing naming (I assumed that this would allow ZUser to be loaded AFTER the automatically handled settings in User, which isn't the case). So this workaround will not work for e.g. font size, because even if I remove the font_size setting item from User, add it in ZUser, but then change the size inside Sublime, the User pref file will get a respective font_size line added and that line will override the ZUser preference file. But thanks, it's clear now what this workaround means. Though the extra confusion of settings that can and can't be automatically written by Sublime is too much for me. I really wish the parser would just leave comments aside — they are not meant for it!

[nathany]

Something other than JSON for config seems like a good solution to me. Maybe CSON or TOML? I haven't evaluated the alternatives.

It's a big request, but not unlike adding a sublime-syntax alternative to TextMate bundles. The requirements of a different format IMO:

• Allowing and maintaining comments as present in the default preferences
• Never reordering the file, if people want to group preferences nicely it sticks
• Allowing a trailing comma or not having commas so as to have clean diffs when adding a config
• Also the editor should have pretty printing to a standard format with indentation cleanup on save so that machine inserted preferences and human edited preferences coexist nicely

A custom JSON superset could be a possibility if not introducing a second format is a requirement.

[ethanbeyer]

Just chiming in to say I'd love to be able to keep my settings commented. I understand that the parser is doing this, and it's not an easy fix, but since the Default settings have comments and spacing, it just seems like something that should be allowable.

[MrGurns]

Would love to see this fixed. I would fix it myself if I could, sounds like a super simple issue.

[wbond]

To fix this, a solution would need to:

• Annotate all parsed settings in memory with relevant comments placed above, trailing or after
• Remember the order in which all settings were loaded
• Handle reflowing values onto multiple lines as they grow (think lists of strings), preserving comments related to individual items in the list
• Preserve blank lines attached to individual values

I think the biggest issue is determining the exact rules for how comments and blank lines are associated with values, how values are reflowed and sub-values with comments are preserved. If anyone has examples of parsers and emitters for JSON or YAML that do things like preserve comments and ordering when emitting, that could be something to investigate. A quick google search brought this up: https://kev.inburke.com/kevin/more-comment-preserving-configuration-parsers/.

I've looked into other formats. For settings, YAML is likely overcomplicated. TOML is funky once you get to "tables". CSON wouldn't really solve any issues, and doesn't seem to have a C++ implementation to work from.

The final aspect is coming up with a solution Jon can get behind.

[mattst]

When I created this issue back in August 2014 I had only been using ST for a few months, was yet to write any non-trivial plugins, and saw the reordering of settings and removal of groupings and comment lines as an annoyance and went so far as to call it "disrespectful of the user's ability to maintain their own config files".

For quite some time I've realized that fixing these things would involve a great deal of work which frankly I'd prefer to go into other aspects of ST development.

From a technical standpoint a configuration file format that integrates comments into the structure of the language, rather than allowing comments to coexist within it, would seem to be required - maybe someone will modify JSON, TOML, etc., to allow this. You could certainly already do it with a XML DTD to result in something like this:

<setting>
    <comment>Saves buffer when it loses the focus.</comment>
    <name>save_on_focus_lost</name>
    <value>true</value>
</setting>

Fine for parsing, horribly unsatisfying and nasty to manually edit. It would fail to resolve the grouping of settings together without adding a further level above <setting>, i.e. <section>.

AFAIAC I no longer particularly need comments in my preferences and no longer maintain a separate Preferences.sublime-settings file, instead I have a ST notes file to which I add things that I might forget or want a quick reference for, opened in ST with Ctrl+F1, its settings section is currently empty. :)

[nathany]

@wbond Members of the Go community were considering comment preservation for the dependency manager manifest, but eventually punted on it. It's certainly not a simple thing to solve. https://github.com/golang/dep/issues/119

Maybe rewriting the user preferences file could be avoided in some cases, such as by moving dictionary words to another file.

[m147]

Well it's been three years since the issue has been brought up and seems there is no official solutions to this. Which is a pity because this is a really frustrating. I guess I'll just have to use the workaround which is less than ideal but I suppose it's the only way until the devs of ST resolve it, which seems unlikely. Would be great if it were solved!

[jrappen]

@wbond PackageDev shows tooltips for settings now, perhaps you could re-use that code as a start?

[BarbzYHOOL]

this is... horrible

any solution?

[mrmeszaros]

A solution might be to keep the contents of the file in a buffer as-is, and build the AST from the contents with the ranges representing the specific node (this is a common practice in parsers AFAIK). From there any change would just replace the range of the buffer the old value represented - that would keep everything else intact.

Please note:

• this is just an idea - extra thought is needed how it would impact on performance
• this might need tweaks to also include indentation fixing
• I would be very glad to submit a fix to this as I feel the pain
  • I am rather a newbie to developing sublime-plugins, so tutoring/links are welcome

[codethief]

I have a slightly different (and probably simpler) proposal: Have two files for user preferences, one where entirely manual, user-authored changes go (file A), including comments and so on, and another file B which is generated by Sublime when the user changes preferences through the UI.

Upon changing a setting through the UI, Sublime would change the second file B accordingly and verify that the new setting in file B doesn't get overridden by any setting in file A (which takes precedence over B). Otherwise it will notify the user that she will have to edit file A manually.

Note that the current workflow, where both Sublime and the user modify the same file (i.e. file), would still be possible in this approach as users could simply choose to leave file A empty.

[mike-usa]

TL;DR

• comments are even more useful as time passes
• comments are not removed in other similar files (default preferences, user key bindings, etc)
• using a .cache directory for post-processed files seems like a quickfix
• per-project preference management could be helpful

---

I have been following this thread for some time, without commenting. I figured the issue of overwriting the preferences is a nuisance, but wasn't too debilitating as they're often a set-it and forget-it feature.

What @codethief has suggested seems reasonable and something I thought SublimeText would have implemented internally, by now -- I assumed that ST would have a .cache where it stores post-processes files that are used during its bootstrap and updates.

Being involved in different projects, especially as the years fly by, the need to recall reasons for certain preferences has become increasingly useful. For instance, Jekyll projects have their own working directories, which means you'd want to ignore the /html directory for them, but not for non-jekyll projects. In other cases /.tmp is used, which without any context (via comments), a couple years from now I may remove, without realizing that it is used by Sails.js exclusively -- a comment would help.

Furthermore, the user experience is a little inconsistent. I see comments being available in the Default file, but that they are removed when saving in the user file. This behavior is not immediately apparent, nor is it expected from the user's point of view. Not to mention, it may be frustrating that some considerable time/effort has been supplied to formatting a file (spacing, comments/wording, etc) only to have them be removed without warning. A poorly formatted JSON file will bark with error/warning, but it won't if it will remove custom comments/whitespace, or provide time to copy it and move it to another file.

I will add, that I haven't explored much in the way of existing packages that manage user preferences. Perhaps there is a package that renders this issue useless? In a quick browse, I didn't see one. On that note, I would propose if some consideration is given to this issue in the future, one thing that may benefit ST is the ability to manage per-project preferences. By default ST automatically stores settings, but I think those are dynamically generated and not part of a global ST manager. There are some linters (e.g., SublimeLinter) that help enforce formatting rules, but that doesn't help the case state above, which might want to exclude html directories for Jekyll, or other directories for other custom projects.

[keith-hall]

Note that sublime-project files do support preferences, see https://www.sublimetext.com/docs/3/projects.html - I believe support for some more keys were added since this issue was opened. (ofc, many plugins don't honor project specific settings, but that is a separate concern)

I wonder whether a simple solution could be, for when writing to the preferences file, to parse the file using the JSON.sublime-syntax, to determine the offsets in which to replace/insert the new preference key/value, and how much performance would suffer compared to the current implementation... edit: I see this is similar to what someone else proposed, if only I had scrolled up earlier ;)

[dreamcat4]

The solution proposed by @mikeblume seems to make the most sense to me. Just have a cached copy of the file that is sorted and uniqued / etc. held in ~/.cache/ somewhere.

[ednykissyou]

Official? If the official sees it and is willing to solve it, it should not be difficult.

[jrappen]

Fixed in ST4069, cannot reproduce on macOS.

---

Sublime Text:
  build: 4069
  channel: dev
    portable: false
  platform: osx x64

System:
  hardware: MBP 15" 2018, i9 2.9GHz, 32GB, 1TB, Radeon Pro 560X
  software: macOS Catalina 10.15.4 (19E266)
    open_gl:
      gl_api_version: 4.1 INTEL-14.5.22
      glsl_version: 4.10
      vendor: Intel Inc.
      renderer: Intel(R) UHD Graphics 630

[deathaxe]

Don't see it on Windows as well.

[BarbzYHOOL]

fixed without a related commit

magically fixed maybe