line number mapping for debug

pmuellr commented 14 years ago

There's been some email chit-chat regarding support coffeescript in the existing browser debuggers. The basic idea is to include information in the generated JavaScript files that the debuggers can make use of to display the original coffeescript source, and deal with line mapping issues between the CoffeeScript source and generated JavaScript.

Thought I'd go ahead and open up an issue for discussion.

pmuellr commented 14 years ago

So some thoughts:

CoffeeScript source location - somehow we need the coffeescript source. See Issue #557 as one possible solution. The debugger would then be responsible for finding the source given that "URL". Java current just uses the basename() of the file as the file name in it's debug info, but of course it also has the full class name available to it as well. Not sure how we might get some "relative" name in there, so it seems like basename() or abspath() of the input file are the choices. abspath() exposes more information than it should, basename() will cause naming conflicts for same-named files in different directories
line number information - The debugger will eventually want this in hash tables, both directions. Go from a cs line # to js line # when setting breakpoints, and js line # to cs line # when the JS vm is communicating to the debugger (eg, "stopped at line 47"). Presumably the actual data for the line number data could be in JSON format (eg, an object literal mapping cs #'s to an array of js #'s).

mrblonde commented 14 years ago

Might be worth looking at this also: http://nex-3.com/posts/92-firesass-bridges-the-gap-between-sass-and-firebug

Maybe this same type of strategy could be used so that firebug could step through javascript line by line, but display the coffeescript code instead.

Tough project, but could turn into something pretty killer!

jashkenas commented 14 years ago

I'd be glad to help someone out with this ticket, but I'm not going to be spearheading it myself ... so closing as frozen for the time being. If someone wants to step up and do a port of FireSass for CoffeeScript, do let us know.

jashkenas commented 13 years ago

Alright -- this ticket now has life again, in Firefox at least:

https://bugzilla.mozilla.org/show_bug.cgi?id=618650

Reopening...

subtleGradient commented 13 years ago

Quick helpful workaround:

Include each line of the original coffeescript source inside a block comment before the js code that it generated. You could include the original coffeescript source lune numbers in the comments.

That would at least help simplify the learning and debugging process.

StanAngeloff commented 13 years ago

Mascara has a source-line mapping utility that only works in Firefox. It's quite possible to implement something similar:

try
  console.log 'Hello'
  a for i in [0..10]
catch error
  lines = { 31: 3, 32: 3, 33: 3 }
  error.message = error.message + ' on line ' + lines[error.stack.split('\n').shift().split(':').pop()]
  throw error

Run on Try CoffeeScript. Firefox only, though.

andrewschaaf commented 13 years ago

Related code:

Closure Compiler (JS -> JS)

...can generate source-to-source mappings:

java -jar $CLOSURE_JAR --js example.js --create_source_map ./map --js_output_file compiled.js

SourceMapLegacy.java has comments describing the format.

_PYXC-PJ (PY -> JS)_

...can generate mappings using the same format, but with more file info: [{"sha1":"...",...}] instead of []

commit: "changed emit() return type from a string to list of strings and kids"

commit: "source-to-source mappings! (same format as Closure Compiler)"

\ exception-server **

An exception logging/viewing server has been written (in NodeJS) that uses these mappings to convert an exception's stack information into pre-compilation code snippets with TextMate URLs.

(your compilers/scripts send the server (code_sha1, mapping)s and (sha1, code)s at compile-time)

I'll post it [REDACTED TIMEFRAME] when it's more documented/polished.

\ common-exception **

...can extract file/line/[col] information from NodeJS, Firefox, Chrome, Safari

lucaswoj commented 13 years ago

Crazy idea: What if when the CoffeeScript was compiled to JavaScript, it was done such that the lines corresponded 1:1 by placing multi-line JavaScript conversions on a single line.

pmuellr commented 13 years ago

I suspect it is crazy, but it would be useful. I actually do that with https://github.com/pmuellr/scooj , but only because the source is largely unprocessed JavaScript anyway. The nice thing is that even though you see some grungy code in the debugger, you do know the exact line number to go to in your editor.

alexandertrefz commented 13 years ago

I suggest a --debug coffee parameter that does something like this:

# Add Core-Utils to Underscore Namespace
_.ducktype = (obj, methods...) ->
  some("code")

to become:

var __slice = Array.prototype.slice;
// Line 2
_.ducktype = function() {
  var methods, obj;
  obj = arguments[0], methods = 2 <= arguments.length ? __slice.call(arguments, 1) : [];
  // Line 3
  return some("code");
};

this would work Browser independent and i think(im not a coffescript compiler expert, though) it could be implement without big hassles.

andrewschaaf commented 13 years ago

Let's define some terms:

Def a "compiler" compiles a non-empty set of "source file"s to exactly one "compiled file"

Defs

a "(coord to coord)-mapping" is a function from (line, col) in the compiled file to (source_file_ref, line, col)
a "(line to line)-mapping" is a function from (line) in the compiled file to (source_file_ref, line)
a compiled file is "{coord,line}-{coord,line}-commented" if it contains comments in some format which provide a ({coord,line} to {coord,line})-mapping
a "(...mapping...) file" is a byte-string in some format which contains a representation of a (...mapping...)

Notes:

Don't worry, the representations of these mappings are usually much more space-efficient than the ultra-trivial {coord1:info1, ...}
source_file_ref should include sha1(source_file) if possible, because that's really useful when used with my not-yet-released exception server

Defs

"CCMF" is a (coord to coord)-mapping format. It's what Closure Compiler generates
"CCMFH" is a backward-compatible extension with required hashes and optional other info

See my writeup of CCMF[H]: https://gist.github.com/769845

andrewschaaf commented 13 years ago

.lineno

Recall: adding .lineno to each node instance is a solved problem:

https://github.com/jashkenas/coffee-script/pull/955

With no side effects, that grammar.coffee change could be merged at any time.

andrewschaaf commented 13 years ago

Let's get this done! (Please? Please? Please?)

CoffeeScript should support both

{coord,line}-line-commenting in whatever that format turns out to be (e.g. via --line-commenting)
exporting CCMFH files (e.g. via --ccmfh-out=)

(1) might need to wait for some format finalization, but (2) does not.

And the nodes.coffee work required for (2) will apply to (1).

I'm trying to do a FOSS triple-launch in [REDACTED TIMEFRAME]:

a NodeJS web framework
a coherent Freedom Box attempt
an exception server with awesome mapping support and TextMate links to your uncompiled source files
- Screenshot of a mapping-following representation of an exception that occured while NodeJS was running code compiled by PYXC-PJ from Python syntax

(all three are written in CoffeeScript)

...and it would be really, really awesome to have CoffeeScript CCMFH support before then so I can integrate it.

Imagine being able to hit Command-R and have jashkenas/coffee-script-tmbundle compile, invoke node, and then represent the resulting exception in a beautiful and convenient mapping-following way with CoffeeScript source links and code snippets...

andrewschaaf commented 13 years ago

Some modest proposals

The Good (but more complex and possibly slower)

Something like what PYXC-PJ does.

EDIT: e.g.

Throw
  compileNode: (o) ->
    ...
    something(o, ["throw ", @expression, ";"])

Instead of having the main compilation function return a string, have it return a BAR.

BAR ::= ELEMENT-list

ELEMENT ::= string | FOO | BAR

FOO ::= something which can
    1. provide the code fragment for that subtree
    2. provide the ((line, col) -> (source line))
       mapping information for that fragment

Some function can then use the resulting BAR to generate both

the usual JavaScript
a CCMFH file

PYXC-PJ commits:

I'm not a nodes.coffee expert (yet) so I'm not sure if subnode.compile(o) can be deferred. If so would let FOO be, e.g., {subnode:@expression, o:o}

Would o have been destructively modified by the time foo.subnode(foo.o) gets run?

I think The Good could be done without deferring .compile, but that might be harder.

The Bad (but simple (though verbose) to implement and possibly faster)

Something like what Closure Compiler does.

Instead of the beatifully concise current compileNode methods, do something like:

    something.startNode this
    something.add           "throw {"
    something.addSubnode    @expression, o

    # addSubnode: (node, o) ->
    #   ...
    #   node.compile o
    #   ...

    something.add           ";"
    something.endNode()

This would invoke the .compiles in the same order as the current implementation.

The Fugly

Have whatever methods invoke compileNode replace ([...].compileNode([...])) with

((() ->
    token = newUniqueToken()
    (
        "/*FUGLY:#{token}:@lineno*/" + 
        [...].compileNode([...]) +
        "/*/FUGLY:#{token}*/"
    )
)())

...and write a crazy function that uses the resulting text monstrosity to generate (comment-free JS, a CCMFH file)

This would be totally insane, but it would work and it would involve the least modification to nodes.coffee

andrewschaaf commented 13 years ago

I'll work on a

something(o, ["throw ", @expression, ";"])

approach

andrewschaaf commented 13 years ago

BAR-ification of nodes.coffee in progress:

https://github.com/andrewschaaf/coffee-script/commits/master

After each change, I run

git checkout -- lib && cake build && cake build && cake test && git status

and confirm that the tests pass and that no lib/*.js files other than nodes.coffee are changed

andrewschaaf commented 13 years ago

From #coffeescript yesterday:

my point about a codegen method's "//#{@child.compile o}" vs "#{@child.compile o}//" was that the codegen method currently discards the information distinguishing the two.

the relativeCharOffset of each kid matters.

one option would be for the codegen method to store that information in its kids

e.g. for Throw ("throw #{@expression...};"),

@expression.relativeCharOffset = "throw ".length

but this could make the codegen methods uglier than with the BAR-returning approach

with the BAR-returning approach, the Base method calling the codegen method can process the BAR immediately, computing the flattenedCodeString as usual and (processing the mapping implications if mapping:true)

BAR-returning codegen methods are a way to abstract out the mapping-handling code

from the outside, you're still calling the top node's .compile or .compileWithMapping

andrewschaaf commented 13 years ago

From #coffeescript yesterday:

my point about a codegen method's "//#{@child.compile o}" vs "#{@child.compile o}//" was that the codegen method currently discards the information distinguishing the two.

the relativeCharOffset of each kid matters.

one option would be for the codegen method to store that information in its kids

e.g. for Throw ("throw #{@expression...};"),

@expression.relativeCharOffset = "throw ".length

but this could make the codegen methods uglier than with the BAR-returning approach

with the BAR-returning approach, the Base method calling the codegen method can process the BAR immediately, computing the flattenedCodeString as usual and (processing the mapping implications if mapping:true)

BAR-returning codegen methods are a way to abstract out the mapping-handling code

from the outside, you're still calling the top node's .compile or .compileWithMapping

ozra commented 13 years ago

About ten years ago I wrote som crude source-to-source compilers for Perl and C++ which I called Pyrl and Cython. They simply gave you pythonish indent-syntax. They had the flag -p for pretty output and some other for line-to-line, which esentially means that it simply made sure that lines in the output ended up in the same line number as in the input, no matter if certain lines became 'ugly' because of code stacking.

Simple solution. Works in all environments (like Jake that compiles the coffee-makefile from inside it's script), any editor, etc., shouldn't be to hard to implement?

ghost commented 13 years ago

How hard would it be to add an option to have the js output include a call to a custom trace function on every function call? The param could be the callee function name and optionally the original line number and file. This would really help debug those cases where the code doesn't finish because I forgot to call something else at the end of a callback. These cases drive me crazy and cause me to put log calls all over the place.

The convention could be that if a specific function exists at the top level then the feature is enabled and that function is called.

debugFnCallLog = (funcName, fileName, lineNum) ->
    myCustomLogRoutine ...

mathieuravaux commented 13 years ago

For reference, here is a related webkit ticket: https://bugs.webkit.org/show_bug.cgi?id=30933

ghost commented 13 years ago

In my fantasy world V8 and other engines would be broken into a separate parser and byte-code interpreter while retaining the same JS semantics.

aseemk commented 13 years ago

Just want to toss out a major +1 to preserving line numbers, i.e. mapping 1:1 from source Coffee line # to generated JS line #. The excellent Streamline.js library has this option for its transformations, and it has proved invaluable.

Btw, if it helps, our platform is Node. We too compile dynamically w/out generating JS files, so things like line-numbers-in-comments aren't helpful. Our use case also isn't e.g. Firebug debugging generally but more "my program threw an error on line 512, where is that?"

Thanks in advance! Excitedly looking forward to seeing what comes of this.

thelinuxlich commented 13 years ago

+1 for this

andrewschaaf commented 13 years ago

Anyone in the NYC area interested in meeting up for a line-mapping mini-hackathon on Sunday the 15th?

lucaswoj commented 13 years ago

No, but I'd personally donate beer money. This needs to be done.

On May 4, 2011, at 10:27 AM, andrewschaafreply@reply.github.com wrote:

Anyone in the NYC area interested in meeting up for a line-mapping mini-hackathon on Sunday the 15th?

Reply to this email directly or view it on GitHub: https://github.com/jashkenas/coffee-script/issues/558#comment_1100079

tanepiper commented 13 years ago

+1 I'd send some coffee + beer money too for this, it would be the biggest evolution for CoffeeScript

ghost commented 13 years ago

I'm surprised that no one has ever implemented one of the several quick kludges that have been suggested.

On Wed, May 4, 2011 at 6:14 PM, tanepiper < reply@reply.github.com>wrote:

+1 I'd send some coffee + beer money too for this, it would be the biggest evolution for CoffeeScript

Reply to this email directly or view it on GitHub: https://github.com/jashkenas/coffee-script/issues/558#comment_1103210

Mark Hahn Website Manager mark@boutiquing.com 949-229-1012

michaelficarra commented 13 years ago

I'll actually be in the NYC area on the 15th with nothing to do. I may be up for this.

lephyrius commented 13 years ago

+1 for some way of debugging coffeescript. It would be really helpful.

andrewschaaf commented 13 years ago

The 15th is a go. [EDIT: multiple parties cancelled.]

Possible attack plan:

Add .line_number to nodes (satyr's implementation involves more changes but seems cleaner and safer than this hack)
Create mapping files via fugly hack (see above)
NodeJS exception logging/viewing server (to run on localhost) with {mapping file support, code snippets, TextMate links}
Create mapping files properly

ozra commented 13 years ago

I definitely vote for line to line compiling without mapping files. Way to go! :-) If I wasn't in 'way after deadline'-mode on my current project, I'd help out right away.

pmuellr commented 13 years ago

This WebKit / Web Inspector bug may be of interest to folks:

Bug 60550: Web Inspector: add protocol method for loading script source mapping

Presumably for Traceur (it's from Google folks), and just describes the "protocol" by which source mappings are obtained, not the format of the source mappings of themselves.

geraldalewis commented 13 years ago

@michaelficarra and @andrewschaaf -- somehow missed you two were working on a solution to this together (today?). I got somewhere with it if it helps you! https://gist.github.com/973659 Edit: Hah! @andrewschaaf, missed everything after early May before working on this -- came to many of the same solutions you did :) Wish I'd seen it sooner!

andrewschaaf commented 13 years ago

@geraldalewis: Sweet!

@michaelficarra and I were too busy that day to meet up and hack. I made some progress on a line-mapping-friendly exception-server. I'll get it done and posted [redacted].

andrewschaaf commented 13 years ago

With this many comments on a GitHub page, we need to start posting some 9f09aeb-style lulz...

pmuellr commented 13 years ago

Web Inspector: draft implementation of compiler source mappings support in debugger:

https://bugs.webkit.org/show_bug.cgi?id=63940

Draft description of source maps for Web Inspector (linked to in the bug):

http://bit.ly/n4YaTz

The actual format of the sourcemap is not yet specified, though it's presumably JSON as the bug talks about retrieving the source maps via JSONP.

The interface to be provided based on the data is not completely specified, but there's bits of it here, in the section for new source file "CompilerSourceMappings.js":

https://bugs.webkit.org/attachment.cgi?id=99713&action=prettypatch

The idea is, we have the raw source map data, and then provide a JS object which matches that shape that use the data to return the expected results, and voilà! CoffeeScript debugging in Web Inspector.

treeform commented 13 years ago

pmuellr: When will that be live?

I would not mind the one-to-one idea in the mean time: "Crazy idea: What if when the CoffeeScript was compiled to JavaScript, it was done such that the lines corresponded 1:1 by placing multi-line JavaScript conversions on a single line." - lucaswoj

pmuellr commented 13 years ago

treeform: no telling when it will be "live". Work in progress. That could be committed tomorrow, say, or never. Good news is that it's being done by the GOOGlers who are the primary Web Inspector devs, so it will likely ship, sometime. Would first be available in WebKit nightlies, so might want to install that now so you can be ready the minute it makes it into the build! cc yourself on the bug, of course.

I'd rather see any work done, be in generating "source maps", whatever form that may be. It's the answer long-term, compared to the 1-1 line # story. Easy for me to say tho, since I'm not doing ANY of this work ... :-)

pmuellr commented 13 years ago

It appears that the Moz and Google Closure folks are now also on the case! Here's another Source Map proposal linked to in WebKit bug 63940.

andrewschaaf commented 13 years ago

Speaking of formats, here's what Closure Compiler uses now:

https://gist.github.com/1082926#file_foo.simple.js.mapping

(it's a newer format than what they were using earlier in this thread...)

fitzgen commented 13 years ago

Hello! I am a summer intern for Mozilla acting as the lead on the source map project. The source map project will allow mapping any X to JS compilation's line/col/source location info including among others CoffeeScript to JS and JS to Minified JS as long as there exists a source map file. Here's a quick overview of the project status:

I have implemented the core library for parsing and generating source map files (the spec which describes what Mozilla, Closure Compiler, and (if I'm not mistaken) Webkit will support is here https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?pli=1) - https://bugzilla.mozilla.org/show_bug.cgi?id=669999 and https://github.com/mozilla/source-map
I have implemented the bulk of the integration between the webconsole and source maps WRT logged messages and uncaught errors - https://bugzilla.mozilla.org/show_bug.cgi?id=670002
The feature page which tracks the source map project is located here: https://wiki.mozilla.org/DevTools/Features/SourceMap

Now I'd like to begin working source map generation in to CoffeeScript. I'll take a look at previous patches and see if I can't reuse a bunch of that code.

So yeah, I just wanted to say hello and publicly announce my intents.

MichaelBlume commented 13 years ago

@fitzgen: awesome. I'm sure I'm not the only one here who's hugely excited about this ^_^

michaelficarra commented 13 years ago

@fitzgen: from the source map proposal:

Additional fields may be added to the top level source map provided the fields begin with the “x_” naming convention.

That sounds like a really bad design decision. Why not nest objects under some extensions property?

edit: Sorry, I meant to bring this up a few weeks ago, but I couldn't add comments to the google document, so I just kind of forgot about it.

Aupajo commented 13 years ago

@fitzgen Awesome!

I agree with @michaelficarra – an extensions property seems a better choice here.

alexandertrefz commented 13 years ago

+1 for @michealficarra's idea.

showell commented 12 years ago

What is the status on adding lines/columns to the Node objects?

fitzgen commented 12 years ago

@showell, The commit above should maintain the line/col info in the AST after parsing is finished. Unfortunately, I haven't finished the second half where I actually use that information to generate source maps. I have a bunch of dirty stuff that needs to be revisited and probably rethought, but at the moment I am preoccupied with other things.

pmuellr commented 12 years ago

We'll also need a VLQ library, to read/write integers.

There was also some discussion in webkit bug 60969 concerning how Web Inspector will retrieve the sourcemap files. There are options:

JSONP via script injection
JSON via CORS (claims that it won't work though)
IFRAME html wrapper that access the files

We may need to be generate JSONP wrappers for the sourcemap/source, or an extra .html file, or something.

fitzgen commented 12 years ago

@pmuellr, the spec already defines how a source map should be fetched, see "Linking generated code to source maps" in https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/edit?pli=1

Also, my core source map lib, which is meant to be used by Firefox internals, compilers and minifiers, developer tools, etc already handles the VLQ stuff as well as actually generating and consuming source maps as long as you can pass it the information required.

https://github.com/mozilla/source-map

Basically, all that needs to happen to make CoffeeScript generate source maps is to replace all the string interpolation and concatenation with building a tree of SourceNodes which are part of that core library above. Each mapping in a source map needs the original line/col, original source, and generated line/col. Using the SourceNodes just allows us to maintain the original line/col/source info while building up snippets of generated JS. Then when the tree is complete, we can do an in order traversal of the tree as we concatenate the snippets, and by doing this we will always know the generated line/col for a given location while we concatenate and we will have all the info we need.

Little bit of a brain dump, if it doesn't make sense I can try and clarify.

jashkenas / coffeescript