Major grammar improvements

[jtbandes]

Lots of stuff!

• Cleanup/improvement/modernizations of keywords
• More thorough support for numeric literals, including invalid.illegal for malformed literals
• Scopes for operator/precedencegroup
• Improvements to function parameter lists
• Improvements to generic parameter lists
• More thorough support for class/struct/enum definitions (scopes!), with generic params and constraints, and inheritance/conformance clauses
• Support for conditional compilation blocks (#if/#elseif/#endif) and platform checks supported therein

Resolves #20.

cc @infininight, @natecook1000, @sorbits

#if os(macOS) && swift(>=3.0) && !arch(arm64)
import AppKit
#elseif false
Don't even think about parsing this
#endif

// Swift 2 stuff
infix operator ++ { associativity left precedence 10 assignment }
class Foo<Wrapped where Wrapped: SequenceType>: Bar<String, (Int, Any)>, Baz {
  func foo<T where Wrapped.Generator.Element == T>(x y: Int)
    rethrows -> (@escaping () -> Self) -> Int {}

  init?<T where T == T>(param: protocol<Foo, Bar>) throws {}
}
@noreturn func bar(arg: Int = __LINE__) {
  [0x1p+2 ... 0b42 + 0x1.5].map { self.foo($0) }
  fatalError()
}

// Swift 3 stuff
typealias Foo<K> = AnySequence<K> where K: Sequence

infix operator ++ : ExamplePrecedence
precedencegroup ExamplePrecedence {
  higherThan: LogicalConjunctionPrecedence
  lowerThan: ExponentiationPrecedence
  associativity: left
  assignment: true
}

class Foo<Wrapped>: Bar<String, (Int, Any)>, Baz where Wrapped: SequenceType { 
  func foo<T>(x y: Int) rethrows -> (@escaping () -> Self) -> Int
    where Wrapped.Iterator.Element == T {}

  init?<T>(param: Foo & Bar) throws where T == T {}
}
func bar(arg: Int = #line) throws -> Never { fatalError() }

/**/ func thisIsNotAComment() {}

[natecook1000]

This looks really good! We'll still need to update the "built-in" names to include the Swift 3 standard library renamings—things like Sequence and Iterator should be in there now, though we should also keep the Swift 2 names.

[jtbandes]

Yeah, I was thinking maybe we should keep them separate in the grammar, so at some point in the future we can remove the Swift 2 names. Perhaps even provide separate Swift 2 and Swift 3 grammars which include the appropriate common stuff?

[jtbandes]

I think this is now in decent shape, and it's getting pretty large. I'd be happy for this to go through review now — any further features can be added in a follow-up PR.

[patrickrgaffney]

I have been using this for a little over a week now and it's really great.

But is there a reason why there is no function call scope? It isn't in the current language grammar either, I'm just assuming this PR is going to get merged.

Thanks again @jtbandes!

[jtbandes]

The latest work (still in progress) is actually at https://github.com/infininight/swift.tmbundle/commits/master

Is there something you want to use function call scope for?

[patrickrgaffney]

Gotcha, I had actually cloned your repo.

For an example:

func printSomething() {
    print("something")
}

/// This call to printSomething() is not highlighted.
printSomething()

Most of the grammar's implement these calls in the meta.function-call.language scope. I know C and Python do off the top of my head.

[jtbandes]

Ok, I imagine it's not too hard to match simple cases of this. In fact, probably wouldn't want to match more complicated cases (likeThis)().

[jtbandes]

@patrickrgaffney I've added a basic version of function call scopes to https://github.com/infininight/swift.tmbundle/pull/4. It doesn't handle initializers like [Int](repeating: count: ) and Array<Int>(...) and other such nontrivial expressions though.

[patrickrgaffney]

Good stuff @jtbandes!

Are you against having the initializers highlighted, or just didn't implement that in that PR?

Also, it might be a good idea to offer the same function-call scope for methods, a la instance.methodName(parameters). I haven't looked at the PR yet so I'm not sure if they fit the function-call scope or not.

[jtbandes]

I'm not against it, just not sure how to do it well. These things are hard because they can be ambiguous (is Foo(x) a function call or initializer? though perhaps there should be no difference in highlighting) and complicated to match (since Foo1<Foo2<Int>, [Foo3: (Foo4, Foo5)]>(with: x) is a valid initializer expression). We might be able to get away with highlighting all argument labels inside tuples, actually.

[patrickrgaffney]

Never mind about the method calls, your entity.name.function.swift scope handle these well.

As for the initializers — the builtin types work pretty well already, at least from the user's perspective. The type names themselves are picked up in something like Array<Int>(...) and for a single name — Array(...) it just uses the new entity.name.function.swift scope.

The only reasonable way I can think of to discern between function calls and initializers is to highlight according to the naming conventions. Camel-case that begins with a capitalized letter would be an initializer, with everything else defaulting to entity.name.function.swift. There is a bit of precedence for this, for example, in the C bundle the grammar highlights POSIX type names — names that end in _t as in wchar_t — as support.type.posix-reserved.c.

It also does some highlighting common C constant's, ex: kConstantVal.

But I'll leave this decision up to you. 😎

[jtbandes]

@patrickrgaffney Please pull and try again — I just updated this to try and handle more arbitrary function calls, initializers, and subscripts with labels too (and just any tuple).

[patrickrgaffney]

Now that I am going through some code with your new grammar, I think it could be useful to use the naming conventions.

For example, associated values in enums aren't really function calls.

enum Shape {
    case Square(Double)
    ....

    func area() -> Double {
        switch self {
        case let .Square(side):
            return side * side
        }
    }
}

The Square case should probably get a highlight, but I'm not sure it should be the same as a function call (maybe a type?). And the .Case syntax could be a red-flag that this is an enumeration — as opposed to a type or function.

[jtbandes]

I was considering trying to handle «whitespace» .foo differently from «something else».foo, which might help with that.

BTW, latest conventions say that enum cases should be lowercase, so upper/lower isn't enough to distinguish enum cases from other functions.

[jtbandes]

And it's a tricky distinction anyway, because enum constructors are function calls, e.g. let f: (Int) -> Int? = Optional.some, and yet they can also be used as discriminators in patterns, e.g. if case Optional.some(let x) = y where they aren't "acting" like functions but mostly just providing structure.

[patrickrgaffney]

Haha I really need to start reading the SE Proposals.

Yeah I see what you mean. This is very tricky. I'll think on it some more and see what I can come up with. If only we could index the files...

[infininight]

@patrickrgaffney We basically have everything imported and put together on my fork:

https://github.com/infininight/swift.tmbundle/commits/master

Will be pushing this live in ~48 hours unless we see any issues, feel free to look it over. :)

[patrickrgaffney]

Any reason this hasn't been merged yet? Just checking in... 😎

[infininight]

Just got sidetracked a bit, pushed a fix for one last bug tonight and have now deployed the changes. Thanks everyone. :)