[RFC] Discussion on finding common scopes for import export related keywords.

[UltraInstinct05]

I have mainly opened this as an RFC issue to initiate some discussion on import export related keywords in various languages & to find (maybe ?) some common scopes.

This was mainly due to the fact that in JavaScript, import & export are both scoped as keyword.control.import-export (as well as any other keyword(s) like except, as, from in the said statements), which doesn't feel correct since one can explicitly mark an import keyword as keyword.control.import (|||ly for export) rather than having the current scope.

Some of the other references I have been able to find are :-

1. D -> keyword.control.import
2. Erlang -> keyword.control.directive.import
3. Go -> keyword.other.import
4. Haskel -> keyword.control.import
5. Java -> keyword.control.import
6. Python -> keyword.control.import

[Thom1729]

I don't have a strong opinion about this, except that JavaScript's keyword.control.import-export is not great and I would welcome a new scope.

keyword.control.import seems to be the clear leader. I don't particularly care for control, because I think that's supposed to be used for flow control keywords like if, switch, and for, but I wouldn't be upset if that's where we landed.

Import syntax varies wildly across different languages. Off the top of my head, I might mentally divide it into three cases:

Dedicated syntax

In these cases, there's probably a clearly identifiable import keyword to scope.

JavaScript:

import React from "react";
import "./style.css";

Python:

import os.path
from os.path import join

Just another function

When imports are done with a function or with function-like syntax, any of variable.function, support.function, and/or keyword may be appropriate.

JavaScript:

const fs = require("fs"); // Literally just a function call. Currently has a `support` scope.
import("./style.css"); // Technically special syntax, but works almost exactly like a function. Currently scoped `keyword.import`.

Some kind of directive

Languages that have multiple directives, including an import directive, should probably scope directives consistently, but we might add a keyword scope to the name of an import directive.

Erlang:

-import(lists, [map/2,foldl/3,foldr/3]).

C:

#include <stdio.h>

---

Exports might be more diverse. A lot of languages have an explicit export statement. Some implicitly export everything. Some are weird:

JavaScript (CommonJS):

function foo () {}
module.exports = { foo };

Python:

__all__ = ["foo"]

def foo():
    pass

Probably we can't hope to find a perfect solution that will handle every weird export system, but the well-behaved ones should be pretty easy.

[deathaxe]

I agree keyword.control should really be dedicated to control flow keywords. We already introduced keyword.declaration for keywords like class or struct which declare/define new user defined data types or functions (def, ...) in order to resolve conflicts with storage.type.

It would be consistent to also have dedicated scope for keywords which denote statements which are evaluated at "compile-time" or "import-time". Import/export keywords are only one type of such pre-runtime stuff.

I can remember of suggestions to introduce keyword.import for import-time keywords, but I personally find import a bit unlucky as it may also is used by many keywords (e.g.: python's import would need to be scoped as keyword.import.import then).

This discussion may therefore be related with https://github.com/sublimehq/Packages/issues/1860, which also makes a suggestion how to handle include/import keywords as keyword.directive.import.

Use of keyword.control.import is mainly for historical reason and even seems to go back to TextMate. That's the reason for stacking meta.preprocessor keyword.control.directive in Erlang.

[FichteFoll]

I can remember of suggestions to introduce keyword.import for import-time keywords, but I personally find import a bit unlucky as it may also is used by many keywords (e.g.: python's import would need to be scoped as keyword.import.import then).

That would have been https://github.com/sublimehq/Packages/issues/1228#issuecomment-337072664 and the latest update on that seems to be https://github.com/sublimehq/Packages/issues/1860#issuecomment-549451136. Generally, we're unhappy with keyword.control but are undecided whether we would want keyword.import or a different more general 2nd level scope like directive. I think this is our best shot currently.

[mitranim]

Imports tend to be declarations with side effects. They introduce names into scope (declaration), and include another module into your module graph (side effect). Consider JS:

import 'pkg'
import A from 'pkg'
import * as A from 'pkg'
import {A, B, C} from 'pkg'

require('pkg')
const A = require('pkg')
const {A, B, C} = require('pkg')

Logically, this means keywords that import should be scoped like const and other keywords that declare variables, constants, functions, types, etc.: keyword.declaration.

Meanwhile, names declared by imports should be scoped somewhat like variable declarations, with appropriate symbol indexing. In most languages it would be local symbol index (only current buffer). This would be easy to decide if we had a generic scope for all declarations including variables, functions, types, imports, etc.

[deathaxe]

Please note that a main intend is to distinguish preprocessor like (compile / import time) statements/keywords from normal runtime code. In C/C++ for instance we might want to highlight all kinds of #blablab in a dedicated color, which includes conditional keywords such as #ifdef as well as imports like #import.

Hence the idea is to use a common (2nd-level) scope such as keyword.directive, which could propably be specialized as

• keyword.directive.conditional
• keyword.directive.import
• keyword.directive.export
• ...

The question may be whether we want different colors for things like #include vs. using in C/C++.

[Thom1729]

The rub is that in some languages (like C), imports are preprocessor-like, and in others (like Python and JavaScript) they are not. If we do choose a common scope for import keywords across diverse languages, then it can't be a directive scope for the same reason that it can't be a function scope — because that would be wrong for many languages.

Given the diversity of import syntaxes, would it make sense to have more than one standard scope? We could have keyword.directive.import for languages where imports are directives, keyword.declaration.import for languages where imports are declaratory statements, and maybe even support.function.import or something for languages where imports are library functions or function-like syntax. JavaScript would use the latter two:

import foo from 'bar';
^^^^^^ keyword.declaration.import

// Dynamic imports are function-like, but technically special syntax.
const foo = await import('bar');
                  ^^^^^^ support.function.import

// In Node.js, require is a bona fide function.
const foo = require('bar');
            ^^^^^^^ support.function.import

// FYI, JavaScript does have true directive syntax, though they're not currently scoped.
function f() {
  'use strict';
   ^^^^^^^^^^ keyword.directive
}

I know the whole point was to standardize on a single scope, but if we do that, then it would have to be something almost completely generic, like keyword.import, or it would be wrong for some of the most common languages. Moreover, we'd probably want to stack it with an existing scope in a lot of cases (e.g. keyword.directive keyword.import), which has downsides and I think would meet resistance.

Personally, I could live with keyword.declaration.import for everything. But I admit that it might not be quite right for C-like languages, and as I haven't written any substantial C code in ten years, I will yield to the objections of C users. If we were to standardize multiple scopes for diverse syntax, then C might look like so:

#include <stdio.h>
^^^^^^^^ keyword.directive.import

using namespace Foo;
^^^^^ keyword.declaration.import

Thoughts? Is this the best of all worlds, or is it splitting the baby?

[mitranim]

A few more cases for consideration.

Go's import can do everything:

import _ "unicode/utf8" // Only side effect: add package to dependency DAG.
import   "unicode/utf8" // Implicitly declare `utf8` (namespace).
import u "unicode/utf8" // Alias: declare namespace `u` but not `utf8`.
import . "unicode/utf8" // Include: declare all public symbols but not `utf8`.

var _ = utf8.UTFMax
var _ = u.UTFMax
var _ = UTFMax

Swift's import simultaneously declares the package, which can be used as a namespace, but also includes, declaring everything exported by that package in the current scope:

import Foundation          // Declares `Foundation` and everything from it.
var a: Calendar            // Comes from `Foundation`.
var b: Foundation.Calendar // Also allowed.

---

I agree with Thom about schizo-scopes.

Declaration keywords and compiler directives can be seen as intersecting concepts that overlap partially. All imports executed at compile time (which includes JS bundling) can be seen as compiler directives, but in languages without a distinct concept of compiler directives, it doesn't make sense to introduce that concept just for imports. Imports executed at runtime (like in Python) are most certainly not compiler directives. On the other hand, imports explicitly implemented as compiler directives, like C #include, really are compiler directives, and should be scoped as such. This naturally concludes that we might have to use different scopes.

[deathaxe]

As the main intention behind keyword.directive is to enable dedicated colors for all kinds of preprocessor macros it makes absolutely sense to have those two kinds of keywords for imports. It solves the #include vs. using issue in C/C++ and seems to adapt well for import / export statements in various scripting languages.

It would look like:

keyword
  directive
    conditional   // #if , #else, ...
    declaration   // #define, #pragma, ...
    import        // #include, ...
    other
    ...
  declaration
    export       // export
    import       // import, from, require, using, ...

[mitranim]

Makes sense. One correction: the Node function require, as Thom pointed out, really is a function. It doesn't have any special syntax and is not a reserved identifier. If supported at all, it should probably be scoped as variable.function when called (see #2649), with an optional additive scope like support.function.

Having a symmetric scope for export style keywords seems tempting, especially for JS. In JS, export isn't part of declarations the way public is. It makes its own statement, "taking" a declaration statement or anonymous expression as its "input": https://tc39.es/ecma262/#prod-ExportDeclaration. The spec even calls it "export declaration".

In many languages, things are "exported" by making them public via modifier keywords such as public or pub, which conventionally receive the storage.modifier scope. However, in the JS example above, from the perspective of the compiler / language spec, export doesn't act as a modifier inside something else; it's a keyword that makes its own statement.

Personally I would lean towards being precise with the semantics of individual keywords, which means storage.modifier for public modifiers and probably keyword.declaration.export (tentative) for export statements. It's more complicated but more precise. 🤷‍♀️

[mitranim]

One more to consider: in Haskell, one statement declares the current module and its exports, where submodules are exposed by using the same keyword:

module CurrentModule (module SubModule, someFunction) where

import SubModule

-- the rest of the code

This makes module a double declaration keyword: for namespace and for exports, but the same keyword inside the export list does not declare the exported submodules in the current scope (import declares them) and should be scoped and handled differently.

[deathaxe]

What I had in mind when writing down require was Perl's require statement. You are absolutely right with it being a function in JS.

Same with export. Erlang has those -export statemens and other syntaxes may have, too. If import is scoped keyword.declaration.import, exports should receive a scope which feels naturally familiar with imports.

I wouldn't ever have considdered visibility modifiers such as public as export like statements. They are and should keep using storage.modifiier.