microsoft / TypeScript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
https://www.typescriptlang.org
Apache License 2.0
100.34k stars 12.4k forks source link

Proposal: Preprocessor Directives #4691

Closed yortus closed 7 years ago

yortus commented 9 years ago

This proposal is based on a working implementation at: https://github.com/yortus/TypeScript/tree/preprocessor-directives

Problem Scenario

Whilst the TypeScript compiler has some options to control what to emit for a particular source file, it currently has limited support for controlling what is scanned into the compiler from a particular source file. A source file is either included in its entirety, or not at all.

This makes some scenarios difficult. For instance, there are two default lib files, lib.d.ts, and lib.es6.d.ts. A program may be compiled against one, or the other, or neither. If only some ES6 typings are desired, then either they must all be taken (using lib.es6.d.ts), or a custom set of core typings must be maintained with the project.

Even if the core lib files were further subdivided into smaller modular files that could be selectively included in a build, problems would remain. For instance, consider an ES6 built-in, WeakMap, which has members that use the ES6 Symbol spec and the ES6 Iterable spec. How many files must the WeakMap definition be broken into to keep the lib files feature-modular?

Related scenarios have been discussed in other issues:

This proposal focuses on the lib.d.ts modularity problem, since that was the core requirement for the related proposal (#4692) that motivated the working implementation.

Workarounds

With regards to compiling core typings for only some ES6 features, some workarounds are:

For other scenarios, such as supporting DEBUG builds or IOS builds etc, a common practice is to use a single codebase with conditional execution to differentiate behaviour in different environments. This generally works well, except if conditional require(...)s are needed, as these can be a problem for some module systems that statically analyse module dependencies.

Proposed Solution

This proposal adds a new kind of syntax for preprocessor directives.

Preprocessor Directive Syntax

A preprocesor directive:

Valid:

    #if DEBUG
#endif // end of debug section

Invalid:

}  #if DEBUG
#if X  foo(); #endif 

Directives with Arguments

A preprocessor directive may take an argument. If so, the argument appears after the directive identifier on the same line. The directive identifier and its argument must be separated by at least one whitespace character.

Under this proposal, only #if takes an argument, which must be an identifier. An extended proposal may expand argument syntax to include preprocessor symbol expressions.

Contextual Interpretation

If a syntactically valid preprocessor directive appears inside a multiline comment or a multiline string, it is not considered a preprocessor directive. It remains a normal part of the enclosing comment or string.

/*
   The next line is NOT a preprocessor directive
   #if XYZ
*/

Preprocessor Symbols

A preprocessor symbol is an identifier used with some directives (only #if under this proposal). Preprocessor symbols have no values, they are simply defined or undefined. Under this proposal, the only way to define a preprocessor symbol is using the define compiler option (see below).

Preprocessor symbols are in a completely separate space to all other identifiers in source code; they may have the same names as source code identifiers, but they never clash with them.

#if DEBUG
#if SomeFeature
#if __condition
#if TARGET_HAS_ITERABLES

#if and #endif

The #if and #endif preprocessor directives signal the start and end of a block of conditionally compiled source code. #if must be given a preprocessor symbol as an argument. #endif takes no argument. Each #if in a file must have a matching #endif on a subsequent line in that file.

When the TypeScript compiler encounters an #if directive, it evaluates its preprocessor symbol against a list of defined symbols. If the symbol is defined, then the TypeScript scanner continues scanning the source file normally, as if the directive was not present. If the symbol is not defined, then the compiler skips all the source code down to the matching #endif without compiling it.

#if...#endif blocks may be nested. Inner blocks will be unconditionally skipped if their outer block is being skipped.

#if HAS_FOO
foo();
#endif

#if HAS_FOO
foo();
#if HAS_BAR
foo() + bar();
#endif
#endif

The define Compiler Option

Preprocessor symbols may be defined at compile time using the define compiler option, which takes a comma-separated list of identifiers.

tsc --define FOO,bar

{
    "target": "es6",
    "define": "DEBUG,__foo,ABC"
}

Possible Extensions

This proposal is limited to a small set of features that are useful on their own, but may be expanded.

In particular, #if and #endif alone are sufficient to address the lib.d.ts problem described above, as evidenced in the working implementation of #4692. The ability to nest #if...#endif blocks effectively allows logical ANDing of preprocesor symbols.

Possible extensions include:

The working implementation implements preprocessor directives in the TypeScript scanner, since they are really a filter on incoming tokens. This works fairly well for this limited proposal, but questions arise if extensions were added:

weswigham commented 9 years ago

I, personally, don't like the idea of adding a preprocessor to TS. #if is almost a language unto itself, and embedding a DSL into the compiler is usually something which should avoided. If you really want a c-style preprocessor, you don't need to integrate it into TS, IMO. (It is just string manipulation, after all.)

But a preprocessor like this will never be in ECMAscript, and it follows no ECMAscript patterns or semantics - so there's no real hope that support for it would broaden in the JS community in the future. While preprocessor directives give great flexibility and power, there's no runtime JS equivalent for them - you're just using TS as a... well... JS preprocessor.

The most common argument I've seen an argument for why people "need" a preprocessor is when they refuse to do dependency injection for dependencies which change with compilation target (test primitives, etc) - using them as a shortcut, a hack, for avoiding refactoring their code - so maybe I'm just a bit bitter at all the bad code I've seen.

@mhegazy mentions here that TS would be more likely to take a [Conditional(bool)] style change, and I can see why - it follow proposed semantics for an ES feature (decorators) and, conceptually, can be desugared into a runtime check, but also be used to indicate to the compiler that it can perform additional typesystem optimizations/removals/additions.

danquirk commented 9 years ago

Historically we have been very resistant to preprocessor directives, especially control flow type directives (in contrast to a more declarative thing like #Deprecated). That said, it's clear that we do need to investigate better ways for people to make the compiler aware of which subset of their runtime environment they're targettng at design time.

yortus commented 9 years ago

Since this proposal is really aimed just at supporting more granular lib.d.ts typings (as proposed in #4692), it could be limited to just that case - i.e. an internal detail of the compiler that only affects how the default lib is scanned during compilation. The syntax would then also be an internal implementation detail, and could be changed to a ///<...> style or [Conditional(...)] style for example.

@weswigham would a [Conditional(bool)] / decorators style approach work with in purely ambient source file like lib.es6.d.ts? I understood it's a runtime mechanism, but the problem here is to be conditional about types rather than values. Or are you suggesting it could be extended for that purpose?

I agree that preprocessor directives have very weak appeal in a JavaScript environment, which has adequate alternatives. However with regard to conditional inclusion of core types to match real-world mixed ES5/6/7 targets (like in #4692), there are no alternative language-level mechanisms. The only other way is to split the core typings into many small files and work out which ones to pass to the compiler, which is just conditional compilation by another means.

weswigham commented 9 years ago

@weswigham would a [Conditional(bool)] / decorators style approach work with in purely ambient source file like lib.es6.d.ts? I understood it's a runtime mechanism, but the problem here is to be conditional about types rather than values. Or are you suggesting it could be extended for that purpose?

With some dead code elimination, yes. If you enable babel's dead code elimination alongside decorators it... kinda does the right thing right now. (Some bits get too aggressively culled while others aren't pruned as much as they could. Both features are experimental, so I'm not expecting perfection.)

For example, you can do this with babel right now:

var debug = true;
function LogSetter(descriptor) {
  if (debug) {
    if (descriptor.set) {
      let oldSet = descriptor.set;
      descriptor.set = function() { console.log(...arguments); oldSet.call(this, arguments); }
    }
  }
}

class Foo {
  @LogSetter
  set name(value) {
    this._name = value;
  }
}

(new Foo()).name = "yes";

When debug = true, your decorator is emitted like so:

function LogSetter(descriptor) {
  if (descriptor.set) {
    (function () {
      var oldSet = descriptor.set;
      descriptor.set = function () {
        console.log.apply(console, arguments);oldSet.call(this, arguments);
      };
    })();
  }
}

and false:

function LogSetter(descriptor) {}

With a little bit more intelligence/fixup it could know to omit the decorator entirely. But anyways, what I'm getting at is that using normal JS constants to control your runtime changes and some good dead code elimination with actual JS conditions can cover "conditional compilation" most if not all of the time. And the best part about it is that you can disable dead code elimination and inspect what options caused what branch eliminations at runtime, making it much easier to debug than a tangle of #if pragmas. TS has a couple extra bits with the extra things we can decorate which need a bit more intelligence on the TS side (For example, if a property decorator always returns {} effectively nullifying the property, does the property need to stay on the type at compile time for type checking? Probably not.), but it is by and large the same concept.

yortus commented 9 years ago

But anyways, what I'm getting at is that using normal JS constants to control your runtime changes and some good dead code elimination with actual JS conditions can cover "conditional compilation" most if not all of the time.

Right, however the problem scenario presented in this proposal (selectively choosing parts of lib.d.ts files) seems to be one of those times that JS constants and dead code elimination won't help.

yortus commented 9 years ago

@weswigham to add to my above comment, as long as other solutions can be found to 'selectively choosing parts of lib.d.ts files', that don't require adding a proprocessor to tsc, I agree that preprocessor directives are unlikely to have other compelling use cases. The other scenarios all seem to have adequate alternatives either at runtime or as you mention through things like dead code elimination.

tinganho commented 9 years ago

@weswigham the purpose of dead code elimination is to optimize code bases by eliminating dead code. Debug code for me isn't dead. Dead code for me is an unused public method in a library. And the trick you describe is being branded as an ugly hack, people began to use the method you described along with different minifiers a long time ago. Dead code elimination has no directives. So when programmers see that code, how do they know it will be eliminated? I think programmers wants a distinct syntax to handle preprocessing.

I still think preprocessor directives makes a lot of sense. JS has the most widest platform usage of all programming languages, and when we programmers now program code using compilers instead of vanilla JS. I think a feature like preprocessor directives makes a lot of sense to target different platforms. This feature will never likely be implemented in JS so it fits TS also.

rhencke commented 9 years ago

D did this very well without preprocessor directives, using what they called 'static if'.

It has roughly the same semantics as 'if', but can be used in type definitions, and is evaluated at compile-time. It cannot slice through arbitrary text, like preprocessor macros can.

See: http://dlang.org/version.html#staticif

weswigham commented 9 years ago

Some languages can accomplish that with hygienic macros. I'm not sure we'd want hygienic macros, though.

rhencke commented 9 years ago

That's true. I was thinking something simpler than most hygienic macros are. I suppose what I'm suggesting is something more like this hypothetical compile-time if:

// ambient
interface App {
    quit();

    if (false) {
        eraseHardDrive();
    }

    if (Platform === "OS X") {
        setApplicationMenu(menu: Menu);
        getApplicationMenu(): Menu;
    }

    if (Version >= 3) {
        magicVersion3Function();
    }
}

(In this case, Platform and Version are identifiers whose values are known at compile-time, through hand-wavy magic.)

I don't have an exact proposal in mind - my hope in bringing this up is more that if this feature does make it in some form, it can be done leveraging the syntax and concepts already present in TypeScript.

RichiCoder1 commented 9 years ago

@rhencke reading that, I'm vaguely reminded of Dlang's static conditions (http://dlang.org/version.html).

Could have something like:

interface App {
    quit();

    static if (false) {
        eraseHardDrive();
    }

    version(OS_X) {
        setApplicationMenu(menu: Menu);
        getApplicationMenu(): Menu;
    }

    version(3) {
        magicVersion3Function();
    }
}
rhencke commented 9 years ago

@RichiCoder1 Not a coincidence. ;) See: https://github.com/Microsoft/TypeScript/issues/4691#issuecomment-140945854

RichiCoder1 commented 9 years ago

I need to learn how to scroll apparently haha.

omidkrad commented 9 years ago

I have a suggestion for this. I think we can accomplish this without introducing new syntax into the language. We can do this through conventions plus compiler options.

Let's say the convention for a conditional compilation symbol is like: const ALLCAPS. So with the following code:

const DEBUG = false;

if (!DEBUG) {
    console.log("release");
} else {
    console.log("debug");
}

It is compiled normally by default keeping both condition blocks in the emit. Now if we compile with a special compiler option like -define DEBUG, we get an output where DEBUG is set to true with all the code blocks where DEBUG == false scraped off:

var DEBUG = true;

if (!DEBUG) {
    // removed
} else {
    console.log("debug");
}

That would be the simplest form, but we could even optimize it further to totally remove the DEBUG and if statement so that the output is only:

{
    console.log("debug");
}

Notice the curly braces and therefore the scope of the block is preserved.

For the negative condition the compiler option would be -define DEBUG=false or maybe -undefine DEBUG.

yortus commented 9 years ago

@omidkrad if statements can work for dead code elimination scenarios, but what about the problem scenario described in this proposal? Selectively compiling parts of lib.d.ts. Those are purely ambient declarations that don't (presently) allow control flow statements.

omidkrad commented 9 years ago

I'm not in favor of supporting #if and #endif in the tsc compiler. I think it's best to make an external tool such as a shell script to do the pre-processing of source files before passing them to tsc.

gregoryagu commented 8 years ago

I am in favor of support #if and #endif. It works well in c#, and having moved from c# to ts, find the need arises for the same reasons. I use VS 15 to compile ts. An external tool would have to run every time I make a change to a ts file. Often, I make a small change, save, and refresh the browser. I don't know how an external tool would be included in such a workflow. If I could just switch from DEBUG to RELEASE like it possible in c#, that would be very helpful.

omidkrad commented 8 years ago

@gregoryagu introducing #if and #endif would be a major diverging from current ECMAScript proposals and for that many people may not like it. VS 15 integrates Gulp builds in Web projects and provided that the external tool can watch files for changes, it is pretty easy to include that into the build pipeline. Supporting that in the IDE is a different story.

jbrantly commented 8 years ago

But anyways, what I'm getting at is that using normal JS constants to control your runtime changes and some good dead code elimination with actual JS conditions can cover "conditional compilation" most if not all of the time.

Another scenario that could potentially benefit from this which is not covered by the above are import statements and tripleslash references. Specifically this could be useful for manipulating test files on DefinitelyTyped.

jbrantly commented 8 years ago

Another thought occurred to me: this could also be extremely useful for solving some of the issues in #4337. The "wrappings" around definitions (declare namespace vs declare module, and export = vs export default) could be changed based on flags without needing to duplicate the definitions or resort to other weird measures.

Jameskmonger commented 8 years ago

I agree, this would be a useful feature. It would also be useful if they could be set at an 'environment' level, so for example the DEBUG variable could be set for that compilation rather than having to be defined per file.

Pajn commented 8 years ago

I strongly oppose this in ts files preprocessor directives makes the file much harder to statically understand which would be bad for tools as well as programmers. It's also a sign of bad code and can the need can always be solved by a refactoring. What happens in dts files is less interesting though, no programmer would need to read them and they don't encourage bad code in there.

bgever commented 8 years ago

I have a need for this also, since I have code that is only necessary in a dev/test environment and not in production.

It would be great if I could write something like this:

var result = obj.alwaysCall('foo')
#if !PROD
.callWhenNotInProduction('bar')
#endif
;
newtack commented 8 years ago

I second this as well. We want code by contract checks during dev time but not during production. Some code will be specific to platform (web, ios, Android, Windows).

Pajn commented 8 years ago

@bgever What is wrong with

var result = obj.alwaysCall('foo');
if (process.env.NODE_ENV !== 'production') {
  result = result.callWhenNotInProduction('bar');
}

Manipulating code as strings will create huge problems and bugs as well as making it very hard for tools (and humans) to understand.

fis-cz commented 8 years ago

I am missing multiple things with the TypeScript. These things would really help in the TypeScript code development. And I am pretty sure I am not alone.

a) References in Visual Studio. They are useless in the TS project now. I would welcome it would work as in C# project where I can reference other "Lib" projects in the same solution or external libs.

b) Preprocessor. It does not need to be included in the compiler itself. Moreover, it would be better to have it separate as a first stage of the build process. On other hand, it should understand the code so isn't it better to have it directly in the compiler?

c) Namespace / Module / Library merger. Everything what is in the same namespace / module / library should be merged to a single file before compilation. What the hell is reason to need of exporting the class in the same namespace in order to be possible to refer it in another file but the same namespace? Also, the result code for splitted namespaces / modules is not ideal. If I will have 100 classes in the same namespace but splitted to 100 files, the result code will contain 100 closures representing this namespace.

d) Minifier. I am not talking about just removing whitespaces. I am talking about something what google closure compiler is doing when advanced options are switched on. Before the TS is compiled it should be possible to minify it. At least for the production builds. Additionally, I would welcome the minifier is able to recognize string references to object properties and minifies also these strings. I would say this can be done using some annotations (not decorators as somebody recommended) or the preprocessor directives.

e) The transpilation should be the same as it is now, but it would be necessary to do one addtitional step after compilation. Refresh map files to match the original code before preprocessing, merging and minification.

This would not require any changes to ES, runtime support, neither to the TS transpiler itself. Everything would be still the same as before, except some additional features how to control the build process and look of the final JS output.

devluz commented 8 years ago

Would love this. I have a project that doesn't use modules and now I reuse parts of the code for a project using modules. I would use an export statement at the end of my file like this:

#if USE_MODULES
    export {ClassA, ClassB, ClassC};
#endif
wilfrem commented 8 years ago

:+1: I want use this feature for switching dev/production constants.

gzhebrunov commented 8 years ago

As developer with C++/C# experience would be happy to see this feature in TS.

JohnWeisz commented 8 years ago

How about -- instead of having to come up with new syntax -- just simply recognizing compiler directives embedded in comments, similar to the Web Essentials region directive?

//#if DEBUG
console.log("debug mode");
//#endif

This should be absolutely trivial to implement on top of existing compilation logic, as no additional syntax is required, only simple comment parsing.

nippur72 commented 8 years ago

as a proof of concept, I've written a webpack loader (ifdef-loader) that does conditional compilation with the syntax of compiler directives in // comments

As a bonus, it works for normal JavaScript files too, not only TypeScript.

That BTW raises the following problem: imagine to have a typescript library supporting a conditional compile (e.g.: target="android" | "ios"). How do you ship it as pure .js npm package and still preserve the conditional compile? The only way I see is via comments.

acrazing commented 8 years ago

The function is useful, and I think there are two ways to resolve this:

The first one method is elegant and thorough. The second method simplifies problem, and could resolve the second point in #10490 by plugin self, but could not cover all scenarios.

niieani commented 8 years ago

I think doing this in comments, perhaps triple-slash ones to differentiate from normal comments is the best option. You don't need to add any special non-ECMA syntax, yet can still benefit from the feature.

/// #if DEBUG
console.log("hello!");
/// #endif
nippur72 commented 8 years ago

@niieani got your suggestion and turned to triple slashes in ifdef-loader.

My first impression from using in production is that directives in comments are rather dangerous... Also, it's not always possible to write clean code that validates in the IDE, often the directives do conflict (e.g. declaring two vars, etc..).

niieani commented 8 years ago

@nippur72 indeed, good points. I guess TypeScript would have to support this natively so that IDE can understand all the different possibilities. Not so easy. Why do you think using comments is dangerous?

JanMinar commented 7 years ago

I'm really hoping for this (or something similiar) to be added to TypeScript. Currently we're using a self-built precompiler in our company that strips away parts of the code based on set precompiler flags. Unfortunately this results in a lot of error messages in IDEs that support TypeScript but not our precompiler syntax.

By now we have quite the extensive in-house library that we house client and server applications. The part of the code that is shared between the client and server is about 95% of all code. Most of the time it is small methods and variables that have to work / be set differently on the server and the client side. Using precompiler flags for conditional compiling helps us to prevent a large amount of (unnecessary) code duplication.

nippur72 commented 7 years ago

I stopped using a preprocessor (the above mentioned ifdef-loader) in favor of simple if statements that can be easily erased in production. Tools like uglify-js can detect unreachable code and remove it from the bundle. It's very convenient and doesn't get into a fight with TypeScript syntax.

if(DEBUG) {
   console.log("this won't be included in production!");
}

Of course not everything is wrappable around an if, e.g. you can't totally erase a method, but at least you can make its body to be empty:

class SomeClass {
   someMethod() {
      if(DEBUG) {
         console.log("this won't be included in production!");
      }
   }
}
JanMinar commented 7 years ago

@nippur72 , that's a nice idea, but unfortunately it will produce a lot of compiler errors if you have modules, libraries and classes that aren't shared across client- and server-side.

A (simplified) example:

#ifdef CLIENT
/// <reference path="ajax.ts"/>
#elseif SERVER
/// <reference path="mysql.ts"/>
#endif

export class User
{
  // 2000 lines of code that are exactly the same on client and server

  public getAccountData(callback:(data:AccountData) => void):void
  {
#ifdef CLIENT
    Ajax.callService("getAccountData", this.id, callback);
#elseif SERVER
    Sql.query("SELECT * FROM users WHERE id = ?", this.id, callback);
#endif
  }
}
RyanCavanaugh commented 7 years ago

Points on this:

Overall, there are scenarios (statement-level ifdefs, etc) that are already well-supported by existing tools, and other scenarios (parse-time ifdefs that could fundamentally mess with the AST) that we really don't want to support due to wanting to avoid a C preprocessor nightmare.

There are some new opportunities to inject phases into the emit pipeline and people can try those out if they want to try to take existing JS patterns of #ifdef / etc and put them into the TS emitter. But we don't intend to support anything at this time that would need to be understood by the parser or checker.

fis-cz commented 7 years ago

Just to note...

There are already multiple JS build tools to handle this, as one would expect; none of them interfere with TS in big ways

Yes, but its the same as if you say there are plenty minifiers or whatever other tools. Nothing can do better than compiler itself. Same as with minification.

Mixing-and-match compilation units by using tsconfig file inheritance works pretty well as a file-level solution if you need conditional declarations (rather than conditional expressions)

No. It does not. If I have references in the file I am lost. So conditional compilation sucks here. See #15417

Making the "open a TS file and things work in an editor" scenario work with conditional compilation is basically a disaster

Agree. A lot of work with doubtful results.

JohnWeisz commented 7 years ago

@JanMinar

"The part of the code that is shared between the client and server is about 95% of all code. Most of the time it is small methods and variables that have to work / be set differently on the server and the client side. Using precompiler flags for conditional compiling helps us to prevent a large amount of (unnecessary) code duplication."

Honestly, I wouldn't really consider this as a valid use-case of precompiler directives.

For this task, dependency injection could be much better instead, with a clever outsourcing of the non-common parts (we do this on web/desktop/mobile builds using a similar common core code-base). This is only my opinion of course, but still, I wouldn't rely on precompilers here.

JanMinar commented 7 years ago

@JohnWeisz

For this task, dependency injection could be much better instead, with a clever outsourcing of the non-common parts (we do this on web/desktop/mobile builds using a similar common core code-base). This is only my opinion of course, but still, I wouldn't rely on precompilers here.

I'm not quite sure I understand your approach. Doesn't this just move the problem from the actual class to the service class? I'd still need some way to load and instantiate a different service class depending on the target environment.

JohnWeisz commented 7 years ago

@JanMinar

I'm not quite sure I understand your approach. Doesn't this just move the problem from the actual class to the service class? I'd still need some way to load and instantiate a different service class depending on the target environment.

No, the whole point of dependency injection here would be that you inject a different "service class" instance into the core of your application, depending on whether you build for server or client (with your common code-base not knowing and not caring about the actual service implementation, as long as it has the required interface).

Your app core would only define the required methods (and any properties) in the form of interfaces, and it would be then up to a platform-specific implementation to actually ship these interface implementations.

For example, we are building an application for web, Electron, and PhoneGap, and we are shipping a single app core to all 3 platforms. However, all 3 platforms require completely separate logic for opening, reading, writing, and saving files (e.g. we use the Node.js FileSystem API on Electron, and a virtual filesystem on web). The application core does not care how this file handling logic is done, we simply inject an implementation for file handling and that's it.

This does not require precompilation, as the application core is imported into the platform specific wrapper project.