Strange bug merging sourcemaps

[qraynaud]

Hi,

I'm trying to write a gulp plugin allowing to apply regexp with a sourcemap support. As far as I know I'm producing valid sourcemaps in my module but the resulting sourcemaps after merge in applySourceMap are wrong.

I used my plugin twice in a row to create the bug. Here is my test case:

gulp.src('test.js')
  .pipe(gulpSourceMaps.init())
  .pipe(module.exports(/^/, "(function() {"))
  .pipe(module.exports(/$/, "})();"))
  .pipe(gulpSourceMaps.write('.', {addComment: false}))
  .pipe(gulp.dest('build/'))
;

test.js looks like:

'use strict';

console.log('this is a test');

After the first call to the plugin here is the resulting source map structure:

{ version: 3,
  sources: [ 'replace-43237791-afa7-49f9-a334-b8f3d92ccf5c', 'test.js' ],
  names: [],
  mappings: 'AAAA,aCAA;AAAA;AAAA',
  file: 'test.js',
  sourcesContent: 
   [ '(function() {',
     '\'use strict\';\n\nconsole.log(\'this is a test\');\n' ] }

This looks good to me.

The second call to the plugin creates the following result:

{ version: 3,
  sources: [ 'test.js', 'replace-b9687322-214c-4721-8ca8-1fe8b13552f4' ],
  names: [],
  mappings: 'AAAA;AAAA;AAAA;ACAA',
  file: 'test.js',
  sourcesContent: 
   [ '(function() {\'use strict\';\n\nconsole.log(\'this is a test\');\n',
     '})();' ] }

This stills seems right to me. But here is the result after applySourceMap on the two maps:

{ version: 3,
  sources: 
   [ 'replace-43237791-afa7-49f9-a334-b8f3d92ccf5c',
     'replace-b9687322-214c-4721-8ca8-1fe8b13552f4' ],
  names: [],
  mappings: 'AAAA;AAAA;AAAA;ACAA',
  file: 'test.js',
  sourcesContent: [ '(function() {', '})();' ] }

There is no test.js in the resulting sourcemap and the mapping is a copy/paste from the second call without anything looking like the mapping generated by the first.

Any ideas on what I might have done wrong?

[sokra]

The SoureMaps are wrong.

Here is a visualization of the first one.

The correct SourceMap would be:

{ version: 3,
  sources: [ 'replace-43237791-afa7-49f9-a334-b8f3d92ccf5c', 'test.js' ],
  names: [],
  mappings: 'AAAA,aCAA;AACA;AACA',
  file: 'test.js',
  sourcesContent: 
   [ '(function() {',
     '\'use strict\';\n\nconsole.log(\'this is a test\');\n' ] }

AAAA,aCAA;AAAA;AAAA => AAAA,aCAA;AACA;AACA

Visualization of the correct one.

[qraynaud]

Thanks @sokra, I was not sure if all lines should be mapped or not. This helps tremendously. I had some piece of code I tried before doing just that but I had commented it because it was not working either at the time. The result after this change is so much better but still not perfect:

The first mapping I get out looks like yours now:

{ version: 3,
  sources: [ 'replace-97324fa6-3913-405b-958c-1df04e16bb11', 'test.js' ],
  names: [],
  mappings: 'AAAA,aCAA;AACA;AACA',
  file: 'test.js',
  sourcesContent: 
   [ '(function() {',
     '\'use strict\';\n\nconsole.log(\'this is a test\');\n' ] }

Here is the second one:

{ version: 3,
  sources: [ 'test.js', 'replace-442add3b-6429-4229-b45d-4c13b7d0970f' ],
  names: [],
  mappings: 'AAAA;AACA;AACA;ACFA',
  file: 'test.js',
  sourcesContent: 
   [ '(function() {\'use strict\';\n\nconsole.log(\'this is a test\');\n',
     '})();' ] }

And here is the result of the merge:

{ version: 3,
  sources: 
   [ 'replace-97324fa6-3913-405b-958c-1df04e16bb11',
     'test.js',
     'replace-442add3b-6429-4229-b45d-4c13b7d0970f' ],
  names: [],
  mappings: 'AAAA;ACCA;AACA;ACFA',
  file: 'test.js',
  sourcesContent: 
   [ '(function() {',
     '\'use strict\';\n\nconsole.log(\'this is a test\');\n',
     '})();' ] }

As you can see on the visualization of the last mapping there is a small glitch on the first line.

Any idea ? I suspect I'm not handling the final \n in the file properly or something along those lines…

[sokra]

That's working as intended. The resulting SourceMap resolution is the minimum of both provided SourceMaps.

[qraynaud]

Hum ok… Thanks! That will prevent me from trying to track in my code why the mapping of the first line of code in test.js is lost ^^.

But won't that prevent me from adding breakpoints on the 1st line of the test.js file if it contains a meaningful line?

If so, is there another package/module/function I could use somewhere to get a more usable output?

[sokra]

If so, is there another package/module/function I could use somewhere to get a more usable output?

I don't think so.

Try to add new code as separate lines if you want it to be mapped.

[qraynaud]

Ok, thanks a lot for your help at the least !

[qraynaud]

@sokra : I have a last minute question : do you have any reference on a bug somewhere explaining the rationale behind how applySourceMap is implemented ? I mean why is it making a minimum of both sourcemaps ? In some instances (like this one) I believe it is not the most useful / logical output but I'm sure I'm missing something.

[sokra]

It's for technical reasons:

SourceMaps doesn't map to ranges in the original code. They map to points in the original code.

Here is the raw data of the SourceMaps:

1: [replace-43237791-afa7-49f9-a334-b8f3d92ccf5c] 0->1:0  [test.js] 13->1:0
2: 0->2:0
3: 0->3:0

+

1: [test.js] 0->1:0
2: 0->2:0
3: 0->3:0
4: [/source/replace-a504d523-c5a9-4304-b61d-faace91e511d] 0->1:0

Now you replace the [test.js] in the second one with the first SourceMap, but keep in mind that you don't now anything else about how the chars are mapped in the file. So you only look up the original poisitions (1:0, 2:0, 3:0) in the first SourceMap. They map to ([replace-43237791-afa7-49f9-a334-b8f3d92ccf5c] 1:0, [test.js] 2:0, 3:0). This results in the SourceMap:

1: [replace-43237791-afa7-49f9-a334-b8f3d92ccf5c] 0->1:0
2: [test.js] 0->2:0
3: 0->3:0
4: [/source/replace-a504d523-c5a9-4304-b61d-faace91e511d] 0->1:0

[qraynaud]

Oh yeah, this looks sensitive said like that…

I will probably say something stupid but wouldn't it be possible to keep aside the last mapping inserted from the first sourcemap and each time we want to add a mapping from the second, if the mapped position in the first source map is on a line further on the one in the mapping set aside, we find and insert again all those in between before going on?

[sokra]

You can do the replacement on SourceNode level. This would give you better results.

Here is how I've done it: https://github.com/webpack/core/blob/master/lib/ReplaceSource.js

[qraynaud]

That looks very cool to me. Very interesting. I might give it a try!

[qraynaud]

Thanks a real lot. I really appreciated your help on this! I hope you will stay around in the community ^^.

By the way since I just discovered you maintain webpack: I'm using it for a package and I love it. It really saved my day after I lost a lot of time trying to configure browserify to do my bidding.