Create a pipeline to analyze a JavaScript/TypeScript-based app, including webpacked JS

[pombredanne]

When I use JavaScript in an application, I have a few typical configurations and contexts:

At development time:

• I will use npm with a package.json (or more rarely bower) to describe my top level dependencies, and I will save a package-lock.json (or yarn.lock if I use yarn) that contains a frozen set of resolved package deps.
• I may have a directory with nested node_modules that contain my actual dependencies
• I may use webpack to combine one or more libraries together (typically for use in the web browser for frontend code) or some other "minifying" or bundling tool
• I may use TypeScript as source code and transpile it to JS, or I may use other transpilers such as Babel.

At run time:

• I may deploy some node_modules as-is (typical for a node-based server-side application) or as resources in some other deployed artifacts (seen sometimes in Java)
• I will deploy the transpiled/webpacked code together with a possible .jsmap map file that may contain the original sources and the mapping of transpiled symbols back to their source code equivalent

My problem is to find which subset of the devel codebase (the From side) has been deployed (the To side) knowing that the deployed JS may be:

• remixed, minified, transpiled or webpacked
• copied as-is
• The same can be said for JS and CSS, and CSS could be transpiled from SCSS and/or Compass

[pombredanne]

In terms of solution we can start with:

1. Inside the To codebase, relating .jsmap and .cssmap files to their .js or .css files
2. Inside the To codebase, relating .scss to their .js or .css files
3. Between the From and the To codebase, mapping .js and .css files on the To side to non-transpiled code on the From side, and design how good this mapping can be
4. Later, if we do not get correct from path mapping, we could extend to introspecting the To .jsmap files to extract the source content and checksum-map it to the corresponding sources of the From side, and/or map using symbols.

[pombredanne]

We have two cases:

1. Using heuristics, we can match the path of a deployed JS or CSS or SCSS file from the To-side to the From-side. If the path match conclusively, we can further validate some paths using actual content
2. For css and JS map files, we further have a list of paths to the files used and combined/transpiled/compiled in a given CSS of JS file. For exmample:

   "sources":[
     "../../../../../../src/main/resources/META-INF/resources/style.scss",
     "../../../../../../../../../node_modules/@clayui/css/src/scss/cadmin/variables/_globals.scss"
   ],

   These path should be traced back and mapped to the To-side OR to third-party external code, like the @clayui/css npm package in this example.

To support this tracing we likely need to keep a list of these stored in the DB and have a way to relate each such path to either another From-side resource or a file inside a package, which means at least adding some DiscoveredPackage.

Adding these DiscoveredPackages (that may not be always deployed yet and not in the To-side at all) could be done this way:

• we can assume that we have all the corresponding npm already indexed in the purldb (We are collecting all the From-side purls and indexing them otherwise with #70)
• then we can perform various matching:
  • path matching to the purlDB
  • content/checksum matching between the sourceContent and actual upstream sources in the purlDB. This may be made difficult as the content may not be bit-for-bit the same
  • symbol matching using the symbols names and some purlDB index that would allow symbol matching, either directly or with some fingerprint

To process map files, we can consider https://github.com/mattrobenolt/python-sourcemap or parse by hand since this is a simple JSON format. Note that webpack may sometimes create weird paths in the js.map files.

[pombredanne]

I suggest that we start with path matching this way processing each file one at a time:

• use path matching heuristics for the various map and css and js files back to the From-side
• for .map files, process each of the "sources" path references and also match them back to the From-side. If we are not able to match these correctly, we will to do a path-matching looking in the purlDB. If we cannot match we need to tag the map file with a special status telling us that we could not find an origin.

If we match the path to an unambiguous package, we would:

1. create a DiscoveredPackage
2. relate this somehow to the map file
3. assign some status or tag that tells us that this package was matched via path because of a map file "sources" reference

[mjherzog]

Regarding PurlDB matching we need to consider that some teams package their own code as private npm packages kept in a private repo so these will never appear in PurlDB.

[pombredanne]

This is mostly done and operational as steps in the develop_to_deploy pipeline ... so we are just keeping this open to create a JS-only pipeline