Automate OSS registration

joaomoreno commented 5 years ago

[x] Use default component detection task #62453
[ ] ~~Get component detection task to detect OSSREADME.json~~
[x] Migrate to cgmanifest.json format
- [x] Get commit versions of OSS that we get directly from Github
- [x] Check whether cgmanifest.json handles extra fields in the JSON files, since it would be convenient to keep having licenseDetails in there.

joaomoreno commented 5 years ago

Talked to @alexandrudima and updated the first post with checkboxes with what needs to be done.

isidorn commented 5 years ago

Created branch isidorn/cgmanifest Where I have manually created two simple cgmanifest.json files with extra licenseDetails in there.

[x] It is fine to add additional fields to the cgmanifest.json, so we can move all our licenseDetail there.
[ ] create a list of dependencies we depend from ossreadme.json
[ ] contact @aeschli to get commit hash for each dependency

isidorn commented 5 years ago

Here's the full list of oss we depend on from the OSSREADME.json Note that for some components we do not have a repository. I have sent a message to @aeschli to check if he can share the commit of each component.

https://github.com/Jxck/assert
http://www.chromium.org/Home
https://github.com/nodejs/node
https://github.com/electron/libchromiumcontent
https://github.com/electron/electron
https://github.com/jrsoftware/issrc
https://github.com/gabime/spdlog
https://github.com/paulmillr/async-each
https://github.com/aadsm/jschardet/blob/master/LICENSE
https://github.com/inikulin/parse5
https://github.com/Microsoft/TypeScript
https://www.github.com/DefinitelyTyped/DefinitelyTyped
markdown-it-named-headers
uc.micro
mdn-data
devtools-protocol
buffer-alloc
expand-template
tunnel-agent
noop-logger
buffer-alloc-unsafe
buffer-fill
https://github.com/mmims/language-batchfile
https://github.com/atom/language-clojure
https://github.com/atom/language-coffee-script
https://github.com/atom/language-c
https://github.com/textmate/c.tmbundle
https://github.com/dotnet/csharp-tmLanguage
https://github.com/atom/language-css
https://github.com/moby/moby
https://github.com/emmetio/expand-abbreviation
https://github.com/ionide/ionide-fsgrammar
https://github.com/textmate/git.tmbundle
https://github.com/textmate/diff.tmbundle
https://github.com/atom/language-go
https://github.com/textmate/groovy.tmbundle
https://github.com/daaain/Handlebars
https://github.com/tgjones/shaders-tmLanguage
https://github.com/textmate/html.tmbundle
https://github.com/beautify-web/js-beautify
http://www.w3.org/TR/2015/WD-html51-20151008/
https://github.com/ionic-team/ionic-site
https://github.com/textmate/ini.tmbundle
https://github.com/atom/language-java
https://github.com/Microsoft/TypeScript-TmLanguage
https://github.com/textmate/javascript.tmbundle
https://github.com/Microsoft/vscode-JSON.tmLanguage
https://github.com/atom/language-less
https://github.com/emilast/vscode-logfile-highlighter
https://github.com/textmate/lua.tmbundle
https://github.com/fadeevab/make.tmbundle
https://github.com/textmate/markdown.tmbundle
https://github.com/chriskempson/tomorrow-theme
https://github.com/textmate/markdown.tmbundle
https://github.com/atom/language-objective-c
https://github.com/textmate/perl.tmbundle
https://github.com/atom/language-php
https://github.com/powershell/editorsyntax
https://github.com/davidrios/pug-tmbundle
https://github.com/MagicStack/MagicPython
https://github.com/Ikuyadeu/vscode-R
https://github.com/demyte/language-cshtml
https://github.com/textmate/ruby.tmbundle
https://github.com/zargony/atom-language-rust
https://github.com/atom/language-sass
https://github.com/tgjones/shaders-tmLanguage
https://github.com/tgjones/shaders-tmLanguage
https://github.com/atom/language-shellscript
https://github.com/Microsoft/vscode-mssql
https://github.com/owensd/vscode-swift
https://github.com/freebroccolo/atom-language-swift
https://github.com/jesseweed/seti-ui
https://github.com/Microsoft/TypeScript-TmLanguage
https://github.com/Colorsublime/Colorsublime-Themes
Unicode
Document Object Model
https://github.com/WICG/BackgroundSync
https://github.com/textmate/asp.vb.net.tmbundle
https://github.com/atom/language-xml

isidorn commented 5 years ago

@alexandrudima do you have an idea what should we do for components that do not have a repository specified. Example

aeschli commented 5 years ago

@isidorn For the text mate grammars we have npm scripts (npm run update-grammar) that fetch the latest version of the grammar from the original repo, massages it and store it as JSON in our language extenions. Part of the massaging is to store the commit number in the grammar, see e.g. https://github.com/Microsoft/vscode/blob/d4bfba4653915fd199d2deb05c4fc4f8e3c11240/extensions/go/syntaxes/go.tmLanguage.json#L7

I suggest to store that commit also in the the new OSSREADME file. The grammar update script can be found here @alexr00 now took over the grammar udating so maybe you can do the changes together.

Similar for the Seti Theme, which has a npm run update. That one currently takes it from what I have checked out on my disk (repo https://github.com/jesseweed/seti-ui, cloned on same level as vscode) , but can be changed back to use master from head. (You will have to look at the script)

The html dependencies are more complicated as the actual reference is in vscode-html-languageservice. That one has an OSSREADME too. The jsbeautify is a dev dependency, has also a script that converts the formatter code to the form we need.

alexdima commented 5 years ago

@alexandrudima do you have an idea what should we do for components that do not have a repository specified.

Sorry, no idea. Perhaps reach out to the dev that added that and ask them for a URL to something, even if that something is not a git repo ?

isidorn commented 5 years ago

Work is being tracked in this branch https://github.com/Microsoft/vscode/tree/isidorn/cgmanifest

Here's the current todo list:

[x] Generate cgmanifest.json for all ossreadme.json @isidorn
[x] Insert git commit hashes or npm versions for all our dependencies @isidorn @alexr00
[x] Update the distro tools to work with cgmanifest.json @alexandrudima
[x] Commit rust cargo files for the inno-update package @joaomoreno
[x] Make sure distro tools for thirdPartyNotices.txt understand rust cargo files @joaomoreno
[x] move licenses overrides to a new file called cglicenses.json @alexandrudima
[x] make the OSS tool produce 100% the same result as before @alexandrudima
[x] Once the tools are nicely working delete all ossreadme.json and merge @isidorn

For reference here are examples of supported cgmanifest.json formats

isidorn commented 5 years ago

I have added all the commit hashes and the versions for our componenents and now all the components are nicely detected https://monacotools.visualstudio.com/Monaco/_build/results?buildId=42680&view=logs

We just need to add versions for the Unicode and Document Object Model that typescript depends on and I am checking this with @mattbierner

We are currently not specifiying if the component is a DevelopmentDependency however that is optional by the component governance team.

All in all it seems we are good here. Will close once we update the typescript component versions

isidorn commented 5 years ago

Closing this as we are done here. Thanks everybody for great help. There are two follow up items which we will tackle #63564 and #63577

microsoft / vscode

Automate OSS registration #62060