Standard Package Structure

[pradyunsg]

This is really a broader version of #8, extending it to the whole package. The discussion there was headed this way and this is really a topic that deserves a separate issue of it's own. So, here it is.

The idea is that Packages in Sublime Text should be arranged in a standard way which may be optionally/partially enforced, like in other text-editors like TextMate 2 or Atom.

Since having too many files in the root directory of the Package creates a lot of noise, the various types of files should be moved into sub-directories in the package. While reducing noise in the root directory, it also makes the Package arrangement feel more organized.

Things worth more discussion (IMO): Some initial questions regarding the structure:

• Is there a need for a "/tests" directory?
  Can tests for the Python code be placed in "/lib/tests" and syntax tests be placed in "/syntaxes/tests" instead?
• How many .py files should be there in the root package directory?
  Should it be only one, with the same name as Package ({name}/{name}.py) and it should include everything that it wishes Sublime Text (and other plugins?) to know about...
• Naming scheme for the sub-folders.
  Should they be CamelCase or snake_case or hypenated-names?

/cc @jappen @FichteFoll @aziz @jbrooksuk @gerardroche @sublimehq

---

Initial Draft

(Based on @jrappen's original suggestion)

    /build
        /linux
            *
        /osx
            *
        /windows
            *.exe
        *.sublime-build
    /color-schemes
        *.tmTheme
        *.hidden-tmTheme
    /commands
        *.sublime-commands
    /keymaps
        *.sublime-keymap
    /lib
        *.py
    /macros
        *.sublime-macro
        *.tmMacro
    /menus
        *.sublime-menu
    /messages
        install.txt
        *.txt
    /metadata
        *.tmPreferences
    /mousemaps
        *.sublime-mousemap
    /settings
        *.sublime-settings
    /snippets
        *.sublime-completions
        *.sublime-snippet
    /syntaxes
        *.tmLanguage
        *.hidden-tmLanguage
        *.sublime-syntax
    /tests
        *.*
    /templates
        *.*
    /themes
        *.sublime-theme
    .gitignore
    .gitattributes
    .no-sublime-package
    *.py
    CHANGELOG.md
    CONTRIBUTING.md
    LICENSE.md
    README.md

[gerardroche]

• Tests all go in /tests the package won't be split into sub packages so littering tests across the package holds no value
• You can have as many py files as you like in the project root
• Where possible, sub folders should be lowercase dash separated like color-schemes

In the latest build 3093 all snippets have been moved to /Snippets. Sure wish they had been moved to /snippets.

[warmwaffles]

I actually like the idea of this structure

[pradyunsg]

@gerardroche (2) is for backwards compatibility?

[jrappen]

Backwards compatibility doesn't make sense here cause this repo is only for Build 3092 and later.

[jrappen]

@pradyunsg I updated my draft.

[gerardroche]

@gerardroche (2) is for backwards compatibility?

Backwards compat. is not really the issue. Though, ST commands will only load from the root folder. I just don't think this needs to be a rule. A guideline maybe.

Plugin guidelines and best practices is a supplementary information to the standard structure.

[pradyunsg]

That makes sense... It is indeed more apt as guideline.

[FichteFoll]

I'm mostly fine with the strucutre, but I have a few questions and things to consider:

1. The "lib" folder is, to my knowledge, mostly used for binary resources and not for source code, which goes into "src". However, due to the compiled-interpreted nature of Python it can also load "libraries" (modules) from source directly, so I suppose this is what you were thinking? I'm also still not too convinced of the look of from .lib import module, it just seems ugly. But there's hardly a way around that (no, sys.path will not be touched!). The only other option would be to use TitleCase for basic resource folders and snake_case for Python modules.
2. Sublime Text only reloads .py plugins if they are directly in the package root, and does not reload the modules it imports (also quite obvious). I suppose one could hack something in a way that does not require to write a reloader a la Package Control, which would be a lot easier to use. You must still reload the base file that's importing the command classes for example however ... or I suppose you could write a build system that takes care of the reload. It's quite hacky but would probabably be managable. I'll see if I can come up with something.
3. Instead of bundling binaries for building in build/<os> I would place them in bin/, along with potential other binary files.
4. What about "any other file" that does not fall under one of these directories, for example a .json resource that is loaded dynamically? I suppose you somewhat intended to use templates/ for that, but that seems a bit too arbitrarily specific imo.
5. I suppose *.tmDragCommand and *.tmCommand are only listed because this structure is supposed to be compatible with TextMate too? I'm opposed to that. Sublime Text basically has a unique file format for each resource except metadata now and proper backwards compatability can hardly be achieved like this. Furthermore, TextMate has its own package structure convention which only starts to differ from this by using capitalized folder names.

Regarding your question list in OP:

1. Personally, I prefer tests for decoupled components to be in their own directories. Syntax definitions and plugins are 100% disjoint and won't have anything in common in the foreseeable future. Thus, I would use color-schemes/tests and lib/tests.
2. Until there is a proper way to solve the issues I raised in my 2. point, people are free to place as many .py files in the root as they like. However, they must not be placed in the root if they don't directly define an interface via sublime_plugin. That means, subclassing any of the classes provided by sublime_plugin which are intended to be loaded directly by ST (and not mixins, whch should definitely not be exposed to ST directly). This includes single-file modules that should go into <modulename>/__init__.py instead. Note: Importing mixins into the base module scope will cause ST to load them as well since it looks into dir(module). By defining commands in their own files and then importing commands into the "root" file, mixins will not be exposed.
3. I generally prefer snake_case, unless Python modules were to be placed in root sub-folders. In the case I would suggest TitleCase for general resources and snake_case for the Python modules (see my 1.).

@jrappen, for chronological reasons it would have been better if you did not modify your first draft in place but rather posted it here as a comment (which is also more on topic).

[gerardroche]

1. The lib folder is not mandatory and will only be used by plugins with larger codebases e.g. package control. Yes, there's no need to modify the sys path. Unless there is a better way I think the import rules are ok. Any rules for the lib folder structure can be guidelines.
2. Unless ST changes the way that plugins are loaded then I think it best not to apply hacks.

1. I prefer tests all go in tests/. Also, all non release files should be added as export-ignore rules .gitattributes file so that they are not distributed in releases. See https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes#Exporting-Your-Repository So the ignore rule for all tests will be tests/ export-ignore instead of lots of ignore rules.
2. These rules have the potential to confuse. If they are to be rules rather than guidelines then they need to be explicit. I think they are better as guidelines.
3. I'd prefer any rules over the lib folder to be guidelines.

This structure when done could go in the wiki.

[FichteFoll]

Regarding python rules: I guess I worded this a bit strongly. We can not enforce any rules in regard to this because authors can do pretty much whatever. Similarly with the entire structure guidelines. Unless ST wanted to break compatibility with all (!) packages, these need to stay guidelines.

Also, I guess I was more talking about template classes than mixins because mixins wouldn't be directly subclassing commands.

[jrappen]

@FichteFoll fair enough, will do. I simply was opposed to the idea of reposting walls of text for people to spot the difference.

---

Maybe adding a GUIDELINES.STRUCTURE.md file via pr would be a start? Maybe with a milestone as target?

As of right now this discussion lacks feedback from Jon and which would be a great help moving forward.

[pradyunsg]

@FichteFoll Response to your list:

1. CamelCase seems like the best option for the reasons you mention...
2. It would be nice if something for this makes it's way into AAAPackageDev. While this would most likely need generating a module dependency graph for detemining which order the modules need to be loaded in, it should be do-able... Not that it's impossible... Also, there's modulefinder in the standard library...
3. :+1: Then build should be renamed to build-systems (or Build Systems).
4. templates should probably be renamed to extras or something else.
5. So, those .tm* files (except .tmPreferences should not be there)? Or move those things into a TextMate folder?

---

@jrappen I meant backwards compatibility with existing packages...

Also, We could post something like a diff for the marking (<- need a better word) changes... I'll change the draft heading in the OP to the "Initial Draft" if that sounds good...

Edit: And change it to the initial draft we had on this post.

Just an example of the last update:

@@ -11,9 +11,6 @@
         *.hidden-tmTheme
     /commands
         *.sublime-commands
+        *.tmCommand
+    /drag-commands
+        *.tmDragCommand
     /keymaps
         *.sublime-keymap
     /lib 

---

@gerardroche Putting the conclusions into the Wiki sounds good... The wiki should be linked to from Package Control Docs as well, for easier referencing.

Edit: On second thought, that's obvious... :sweat_smile:

---

This structure will not be enforced and shall remain only a guideline for near future. These may be enforced at some future date (with a future version of ST possibly) if seen appropriate by the relevant people (one could, in theory enforce this on packages in Package Control). Would that be a reasonable assumption?

Should mousemaps and keymaps into be merged into a single inputmaps folder (name up for discussion)? They fall into the same basic category of mappings of input to commands. There probably won't be more than 4 (one generic, 3 platform specific) of either in a single package... So, merging seems logical to me.

Probably Jon is the best person to answer this next one: How important is backwards compatibility with this? Do we only care about plugins that work with 3092+ or some other release? Or maybe a case-by-case decision?

[gerardroche]

FichteFoll is right, these are all guidelines. ST will never enforce these on users.

I've been using words like "rules", "mandatory" and "guidelines" which I probably shouldn't have.

The way I see it, all default packages should conform to the standard directory structure, whereas it is only a recommended structure for third party packages.

[jrappen]

@@ -30,6 +30,8 @@
     /mousemaps
         *.sublime-mousemap
+    /projects
+        *.sublime-project
     /settings
         *.sublime-settings

[FichteFoll]

Disagree with .sublime-project. These things are supposed to be in the "project root", by convention, and I refuse to not follow that - especially because these aren't auto-loaded resources and you won't have more than one per project/package.

In that sense, you might as well make folders for /readmes and /licenses.

[pradyunsg]

I agree with @FichteFoll, a project should have exactly one .sublime-project file. So, it won't make sense to add a folder for a single file.

Carrying from his comment, there should be exactly one README and ideally only one LICENSE in a project. So, the same reasoning applies.

[pradyunsg]

Should mousemaps and keymaps into be merged into a single inputmaps folder (name up for discussion)? They fall into the same basic category of mappings of input to commands. There probably won't be more than 4 (one generic, 3 platform specific) of either in a single package... So, merging seems logical to me.

@@ -11,8 +11,9 @@
 /commands
     *.sublime-commands
-/keymaps
+/inputmaps
     *.sublime-keymap
+    *.sublime-mousemap
 /lib
     *.py
@@ -26,7 +27,6 @@
 /metadata
     *.tmPreferences
- /mousemaps
-    *.sublime-mousemap
 /settings
     *.sublime-settings

[FichteFoll]

@pradyunsg :+1:

[FichteFoll]

To reiterate some of my earlier concerns, I would:

1. Remove *.tmCommand (and *.tmDragCommand, *.tmMacro). They are not interpreted by ST and should not be included. Package structure of ST and TM differs to much to create a common structure.
2. lib/ should imo be accompanied by src/, where your own package code goes into src/ and third-party code (i.e. other modules you import) should go to lib/.
3. Binaries from build/ should be in bin/.
4. templates/ seems pretty arbitrary and should probably not be considered part of the basic structure. Instead, we should suggest usage of a catch-all directory for any remaining files, e.g. other/ which can then be sub-foldered however you like.
5. *tests/ may appear in any sub-directory, notably color-schemes/tests/ and src/tests/.

Regarding plugins, I imagine the following:

• Preferrably there should be only one Python file in the project root, which imports all classes interfacing with the sublime_plugin API. Example main.py: (note: __init__.py can't be used because it's currently double-loaded by ST, bug)

  from .src import *  # or write names explicitly

  and in src/__init__.py:

  __all__ = ('MyListener', 'ACommand', 'BCommand', 'plugin_loaded')  # optional
  
  from .src.listener import MyListener
  from .src.commands import ACommand, BCommand
  from .src.commands import ACommand, BCommand
  
  def plugin_loaded():
    ...

Other concerns:

1. completions and snippets are similar, but neither is strictly the other. Unless we can come up with a good name to bundle these, they should go in separate directories.

---

This leaves us with the following structure (not diffed because formatting change):

bin/
    *.exe
    *.sh
    *.com
    *.bat
    *.*
build/
    *.sublime-build
color-schemes/
    *.tmTheme
    *.hidden-tmTheme
commands/
    *.sublime-commands
completions/
    *.sublime-completions
inputmaps/
    *.sublime-keymap
    *.sublime-mousemap
lib/
    *.py
    *.pyc
macros/
    *.sublime-macro
menus/
    *.sublime-menu
messages/
    *.txt
    *.md
metadata/
    *.tmPreferences
other/
    *.*
settings/
    *.sublime-settings
snippets/
    *.sublime-snippet
src/
    tests/
        *.py
    __init__.py
    *.py
syntaxes/
    tests/
        *.*
    *.tmLanguage
    *.hidden-tmLanguage
    *.sublime-syntax
themes/
    *.sublime-theme
.gitignore           # git
.gitattributes       # git
.no-sublime-package  # Package Control
dependencies.json    # Package Control
main.py
messages.json        # Package Control
CHANGELOG.md
CONTRIBUTING.md
LICENSE.md
README.md

[pradyunsg]

There's a small issue with this arrangement. For selecting Color Schemes arranged according this arrangement, one has to navigate through an extra level of menus, if you try to select one through the Preferences menu-option.

[FichteFoll]

True. I also don't know if themes can actually be in any subfolder and still be picked up.

We could make exceptions for packages that only contain color schemes and themes (with these as the main purpose). Other packages that implement the color schemes on a programmatical basis, or via setting of some custom syntaxes, they could still be in the subfolder since they hardly get selected manually. On the other hand, this kind of defeats a unique package structure.

[pradyunsg]

Themes work in sub-directories. (Tested with Soda moved into a sub-folder of another temporary package; renamed one theme-file, changed tab-height and applied the renamed theme.) It's just Colour Schemes that are a problem/an exception.

So, I'm thinking:

• Any *.tmTheme files should be added to the root of the package.
• Any other *.hidden-tmTheme files should also go into other/. (the generated files should be *.hidden-tmTheme files, right?)

Diff from @FichteFoll 's version

@@ -6,9 +6,6 @@
     *.*
 build/
     *.sublime-build
-color-schemes/
-    *.tmTheme
-    *.hidden-tmTheme
 commands/
     *.sublime-commands
 completions/
@@ -30,6 +27,7 @@
     *.tmPreferences
 other/
     *.*
+    *.hidden-tmTheme    # Auto-generated
 settings/
     *.sublime-settings
 snippets/
@@ -47,6 +45,7 @@
     *.sublime-syntax
 themes/
     *.sublime-theme
+*.tmTheme            # Non auto-generated
 .gitignore           # git
 .gitattributes       # git
 .no-sublime-package  # Package Control

Deviating slightly from the topic, on the "core"/closed-source part, I think there should be a change to make the loading of colour-schemes more consistent with the rest of the things like settings and themes. (And while he's at it, Jon might make a new, nicer syntax for defining colour-schemes :wink:)

/cc @sublimehq

[FichteFoll]

For nicer syntax for color schemes, I found CSS (and their pre-processor languages SCSS/Stylus) to be rather convenient.

Other than that I have to agree that tmTheme files should probably be added to the package root.

[pradyunsg]

@FichteFoll I argee agree, overwhelmingly! I was half-way through with writing of my own for something like the package you linked. Thanks for linking me to it. :)

[wbond]

From a usability standpoint, I don't think adding a bunch of folders adds anything, but instead tends to make it harder to understand what a package contains. Browsing around various Atom, Textmate and VS Code packages, I find it weird to have to drill down into subfolders to see a single syntax or completion file.

There are certainly circumstances where subfolders are a good idea. Snippets are a specific case where it definitely helps. We've recently moved the rest of the snippets into subfolders in this repo.

I've considered a folder for tests, however I found value in having single large tests files as opposed to a lot of little ones. It can help catch bugs that won't be found when just testing all numbers, or all string variations. Those contexts tend to be fairly easy to get right anyway. It is the larger, meta scopes that tend to have more edge cases. So maybe at some point we'll move the syntax test files into a subfolder.

I think some of these ideas could be really helpful for some packages. My recommendation would be to use them as you see fit. I don't see us ever enforcing this since it would break backwards compatibility.

With these comments, I don't see there being any other action item, and in the interest of whittling down the list of issues to actionable things, I am going to close this.

[jrappen]

(JSON-based) *.sublime-color-scheme (and *.hidden-color-scheme) files are now a thing for Build 3149 or later.