Support nested folder hierarchies for projections

[glittershark]

Some tools like Zend Framework have a folder hierarchy something like the following:

application/
|   modules/
|   |   mod1/
|   |   |   controllers/
|   |   |   |   MyController.php
|   |   |   |   ...
|   |   mod2/
|   |   |   controllers/
|   |   |   |   MyController.php
|   |   |   |   ...
|   |   ...
|   ...

It'd be nice in projectile if I could define a projection like:

{
    "application/modules/*/controllers/*Controller.php": {
        "command": "controller"
    }
}

so I could EController mod1_my

[tpope]

I also recently realized two Clojure conventions, test/foo/bar/test/baz.clj and test/foo/bar/t_baz.clj would stand to benefit from something like this as well. I think the right solution is going to involve a {} expression in the key rather than *, but I have only the vaguest idea of how that might work.

Your proposed solution has an underscore that sprang out of nowhere.

[glittershark]

There would need to be some (maybe user defined) way to separate the path components, I would think. Maybe I'm wrong about that though.

What about named regex groups for the file glob? Something like:

{
    "application/modules/(*)/controllers/(*)Controller.php": {
        "command": "controller",
        "name": "{1}_{2}"
    }
}

The named groups, and the "name" option could be optional so as to not break existing .projections.json configuration.

[tpope]

Maybe if you drop the parentheses, but that's still a whole lot of added complexity for a relatively dumb problem. I'd sooner just force a join with / or smoosh them together. I'm not sure I buy why it needs to be an underscore here at any rate.

[glittershark]

Simplicity is fine. I like the sound of putting a / in there

[qstrahl]

Seconding the motion to force a join with /

On 11 April 2014 12:11, Griffin Smith notifications@github.com wrote:

Simplicity is fine. I like the sound of putting a / in there

Reply to this email directly or view it on GitHubhttps://github.com/tpope/vim-projectile/issues/14#issuecomment-40221125 .

[tpope]

Note that smooshing just means you'd have to write "application/modules/*controllers/*Controller.php", which is way more consistent and opens up some other possibilities (at the cost of possible but unlikely false positives, DWIM, and just plain looking weird).

[tpope]

Can someone confirm that both of these return the expected results?

:echo glob("application/modules/**/*controllers/**/*Controller.php")
:echo glob("application/modules/**/*/controllers/**/*Controller.php")

[glittershark]

Looks right to me

[tpope]

@glittershark I don't know what you mean.

[glittershark]

If you're referring to my other comment, I deleted it because I didn't know what I meant either :)

[tpope]

Was leaning towards smooshing then realized false positives for test/*t_*.clj wouldn't be all that unlikely. So I think / joining will win, but probably with a constraint there has to be a / after the first *.

[qstrahl]

Seems sound to me. Looking forward to support for this!

[tommcdo]

I feel like `(space) would be a good delimiter. I mean, each wildcard along the way could contain/`'s and it might just be ridiculous to disambiguate. It's probably (hopefully) far less likely that there will be spaces in the filenames of a code project.

With this style, each wildcard is kind of like a command argument, each with dedicated completion. It would be nice if completion would allow wildcard positions to be skipped (auto-filled), but I can see that getting complicated.

How would multiple wildcards affect the language used for alternates and templates? Would {} be replaced with {1} (and so {underscore} with {1|underscore}, etc)? Or could some other assumptions be made?

[qstrahl]

each wildcard along the way could contain /'s and it might just be ridiculous to disambiguate. It's probably (hopefully) far less likely that there will be spaces in the filenames of a code project.

Uh, could it? Last I checked, you can't have directory separators in file names on any sane system.

[glittershark]

What about the ** glob?

[tommcdo]

The * gets interpreted by projectile as a recursive glob (**)

[qstrahl]

I might not be thinking very hard, but I can't come up with an ambiguous case.

[tommcdo]

Let's say you have

foo/*/bar/*.baz

as your pattern. Then you have these files:

foo/a/bar/b/c.baz
foo/a/b/bar/c.baz

Using / would mean they both match a/b/c.

[qstrahl]

There it is. Yeah, that's no good. =\

[tpope]

It gets ambiguous the second you match */dir/* against foo/dir/bar/dir/baz. It cancels out if we just combine everything together with slashes. This alone is an argument against combining with spaces or {1} or any of these other contrivances.

I'd actually like to switch from * to {}, because it would give us symmetry with the expansions and clear up people mistaking * for a file glob in one fell swoop, plus free up * to use as a more conventional glob. But I think {}/dir/{} is even more confusing in the context of my most recent proposed solution.

[tommcdo]

Correct me if I'm wrong, but for a case like that, `is just as good as/. You'd have the following arguments with/`:

foo/bar/dir/baz
foo/dir/bar/baz

versus the following with :

foo bar/dir/baz
foo/dir/bar baz

Both are cleared up in this case.

But / doesn't address the type of ambiguity I pointed out in my last comment, which arises from more than one match having the same smooshed value (e.g. a + b/c; and a/b + c).

[glittershark]

I feel like there's a case to be made for splitting apart the behavior of * and ** in projectile, to get rid of these kinds of ambiguities.

[tpope]

You're right. I guess the first * will be greedy, or we could maybe get more clever, go back to smooshing, and make {}dir/{} match lazily but greedily otherwise. (This is consistent with how you'd expect test/{}t_{}.clj to work. I think.) In the PHP example, what does a nested path look like?

Spaces are arguable in the title case but absolutely ridiculous in the context of my Clojure examples.

[tpope]

@glittershark I'm trying to move away from the globbing syntax not double down on it.

[tpope]

Though if I can't switch to {} it might be better than nothing.

[tommcdo]

What about creating files? How would you infer where to unsmoosh?

[tpope]

I guess greedy/lazy actually means "all of the nesting"/"no nesting".

[tommcdo]

Sounds kind of limiting. I'm picking up that we might be thinking of this differently. I'm thinking that each submatch would be accessible independently when it comes to alternates and templates. I don't know what the notation would look like, but a possible example is this:

"foo/{}/bar/{}.baz": {
    "command": "foo",
    "alternate": "foo/{1}/bar/{2}-test.baz",
    "template": [
        "class {2|underscore}"
    ]
}

[tpope]

I want some pattern for test/foo/bar/t_baz.clj that lets {} expand to foo/bar/baz. Numbered expansions aren't out of the big picture but for shit this basic they're a cop out.

[tpope]

I'm also thinking forward to potential abstractions like "alternate": "test", which will fall flat on its face if there's not a singular canonical expansion.

[tpope]

Let me make my earlier question more concrete. test/**/t_*.clj is the correct glob for my example. Would the PHP example look like application/modules/*/controllers/**/*Controller.php? Is nesting possible in the second slot? And only in the second slot?

[glittershark]

Not for my use case - there's no nesting prior to *Controller.php, so it'd just be application/modules/*/controllers/*Controller.php assuming strict glob syntax

[glittershark]

See the example file hierarchy in the original issue description

[tommcdo]

In the case of Zend, I think a module path could include some nesting. While it's probably almost always going to be in a single directory, the module path can be configured arbitrarily, so it's not unreasonable that some file could be at application/modules/foo/bar/controllers/FooController.php. A similar directory structure is common in Kohana, and in that case it's more likely that both slots could contain nesting, e.g. modules/foo/bar/classes/Controller/Foo/Bar.php.

I'm not sure where you're going with the "alternate": "test" point. Why would it fail if there's no expansion?

[tpope]

Gonna think on it a while; mistakes here could be costly in terms of both implementation time and backwards compatibility. In the meantime if anyone finds any real world examples that couldn't be handled by .../**/...*... let me know.

[tpope]

@tommcdo the idea is that that Clojure file could point back at the source command, and then we can just try plugging foo/bar/baz into src/{}.clj and any other patterns in turn. And in the reverse case, we try plugging into test/{}t_{}.clj by applying dirname/basename semantics. This is one reason I want to distance us from "globbing", because globbing is just one implementation detail among other uses.

[tpope]

Ugh just saw the Kohana example. Surely it would have the same disambiguation problem we do?

[tommcdo]

Yeah, it would.

I'm not saying it's common (heck, Kohana is no longer supported), but it still seems like a viable use case to me.

[tpope]

Does anyone even know how well projectile works on Windows? Looking at the code and first order of business will be making sure we're doing the right thing with slashes.

[tommcdo]

I think Windows doesn't care whether you use forward or backward slashes. That said, I haven't tried it.

[tpope]

Yes you can pass either slash to glob but globbing is just one piece of the puzzle. We need to take it into account in equality checks for example. I've think I've been diligent about that but without any testing it's hard to say.

[tpope]

After sleeping on it, I felt that given that ** actually maps to the problem pretty cleanly (Kohona notwithstanding), it would probably be best to just support a limited version of it, rather than force contrived curly brace semantics. I've pushed a prototype to the glob branch. I expect there are breakages around the empty ** case, at the very least. Let me know what you find.

We will probably also need to add an expansion like {dirname} that also includes the trailing slash. This might be a good time to revisit that "leading" idea you had, @tommcdo.

[tpope]

Well this issue got quiet. I iterated a few times, and I'm pretty satisfied with the result.

[glittershark]

This is pretty sweet. Thanks!

[monokrome]

Just wanting to provide a use case which I don't believe is covered by the current solution.

This same convention used by Kohana is used by some AngularJS projects. Off the top of my head, some other frameworks that provide this structure as convention include Django and Symfony. Many Marionette.JS projects also take this approach. I'm pretty sure that Rails projects are similar, but it's scaffolding makes this less of an issue.

An example of the project that I'm working on is quite similar to Kohana with the following structure:

src:
  - common:
    - module.coffee
    - partials:
      - index.jade
      - other.jade
    - controllers:
      - index.coffee
      - other.coffee
    - directives:
      - other.coffee
  - specific:
    - module.coffee
    - partials:
      - index.jade
      - some.jade
    - controllers:
      - index.coffee
      - some.coffee
    - directives:
      - some.coffee

It would be nice to have a more powerful way of doing this, because I would like to be able to create a new files using the template functionality from projectile. For instance, a command like :EController common index might create a file called src/common/controllers/index.coffee.

This file would ideally use the already-existing expansions from projectile to mock out the following initial template:

angular.module 'common'
  .service 'common.controllers.index', [
    '$scope'

  ].concat ($scope) ->
    angular.extend $scope, {}

The module is the name of the first glob. The service name is essentially the path from the first to last glob replacing slashes with dots. It's a fairly common thing for Angular modules to map to the filesystem in this way.

Right now, I am needing to create specific keys for every module - but it's a bit of a task because there are quite a few of them. The results are similar to this:

{
  "src/common/directives/*.coffee": {"command": "CommonDirective"},
  "src/common/services/*.coffee": {"command": "CommonService"},
  "src/common/controllers/*.coffee": {"command": "CommonService"},
  "src/*/module.coffee": {"command": "Module"}
}

Most of this functionality exists already in the project, but being able to glob and get references to each one is the missing component for these types of file structures is the single missing piece

[tpope]

Links to real world projects would be helpful.

Those CamelCase commands are irking me. Makes me wonder if I should be enforcing some sort of normalization.

[monokrome]

The majority of Django apps follow this pattern, where there's a set of core files in either the project root or in a specific core module. This same concept translate to the other aforementioned frameworks to varying degrees of popularity.

Some examples:

• Django CMS
• A Popular AngularJS Skeleton

Related articles:

• Modular AngularJS App Design
• How to Structure Large AngularJS Applications

The camel case commands seemed a bit odd to me, also. I felt like it was more conventional to type :ESource instead of Esource and it doesn't sacrifice an extra key stroke since E already requires shift be held.

[tpope]

Having thought about it a few days, I think the next logical step on the path I've set us down is to allow multiple *s but only a single **. This should avoid ambiguity.

Not finding either of those repos or blog posts to contain a concrete example of nesting that can't be handled by ** and * (although perhaps there's something hiding in the massive Django one). So I'm still unconvinced this is anything but the lowest priority.

The official projectionist convention for commands is lowercase, squished together. This is already partially enforced by stripping out punctuation, and a future version might add a tolower() to the mix as well.

[monokrome]

This might just be a misunderstanding in how projectionist works with the \ glob, but I don't see how to reference it as a pattern (in a template or alternate, for example) or how to use it for creating new files.

Are these not supported use cases?

[tpope]

The ** and * combine together. Pull them back apart with {dirname} and {basename} if you need to.