NETBEANS-2263 Integrate the localisation bundles from Oracle's 3rd donation into the current GitHub repository

[borisheithecker]

To check the integrity of the file files from this commit with the files from the donated zip file:

1. First, merge only the first commit (328b07c)
2. Go the project root, do "find netbeans-l10n/ -type f -exec md5sum {} + | sort -k 2 > md5sum.txt"
3. Unzip ApacheNetBeansDonation3.zip and subsequently netbeans-l10n.zip
4. Go to the unzipped directory ApacheNetBeansDonation3/
5. Repeat step 2.: ..... md5sum-orig.txt
6. diff -u md5sum-orig.txt path_to_incubator-netbeans/md5sum.txt. An empty result is expected. Works for me.

[matthiasblaesing]

Thank you - I'll only comment on the license side of the import. I want an opinion from others more knowledgeable of the l10n part of the netbeans infrastructure, if the source code location is sensible and/or if we should open a separate repository of the the l10n data.

• the procedure you described to verify, that only donated files were imported works (I had to rename my intermediate directory from the netbeans-l10n.zip, but that works ok). The first commit is ok from my POV.
• the license change (second commit) is good to be separated, as it turns out, it is not complete:

  # Get the files, that were imported (first commit)
  git diff --name-only c6eb7aa76370f9a059736b686ba2b1f5f622e87c..328b07c6203fa2268c49bb0bd6b7c9d51636ddc4 | sort > imported.txt
  # Get the files, that had changes (this assumes, that the license header was updated) (second commit)
  git diff --name-only 328b07c6203fa2268c49bb0bd6b7c9d51636ddc4..3fc7acd08836cc736d96c81fe53f3b6efaad2445 | sort > modfied.txt
  # both lists should match, as it is assumed, that all files carry the Oracle/Sun CDDL/GPLv2CP header and so output should be empty
  diff imported.txt modfied.txt 

Running the output of diff through diffstat reveals, that 6287 were not caught. My gut feeling tells me, that you missed the files, that Oracle inherited from Sun and never updated.

• the change in the ignore file looks sane
• the build change (last commit) looks ok, as we target JDK8 as minimum runtime JDK

[borisheithecker]

I ran the convert tool from https://github.com/apache/incubator-netbeans-tools to bulk replace the license headers. A large number of files without any license header can be found in the donated zip. It seems that these are the files revealed by your script. This can be either because they can't have one (.gif files for localized image resources, .url files pointing the translated online resources), or because they're not supposed to have one like the many bundle_xx.properties files that contain translated strings for the names of modules. The convert tool only replaces, it doesn't insert a license if none was present.

I suggest that a separate repository for the l10n bundles shouldn't be used. The build scripts assume that the bundles are placed in a subfolder of the root directory, and if they're pulled from a different repository, I think that make working with the source only more difficult.

[junichi11]

cc: @masakikatakai

[matthiasblaesing]

Indeed the files are partitially broken. Here is the list of all files, that are not updated in the second commit:

unmodified.txt

One of the files (netbeans-l10n/src/zh_CN/webcommon/html-angular/html-angular/org/netbeans/modules/html/angular/Bundle_zh_CN.properties):

Bundle_zh_CN.properties.txt

Some one added property definitions above the license header. This makes it now much harder to fix the files. In all fairness - the sample is a mess:

• license in the middle of the file
• dead definitions, that are masked as comments

These files all need to be cleaned up. What IMHO could be done:

• check if the license header is mostly the same
• if so, remove it from the middle of the file
• add the ALv2 header
• repeat for variations

[borisheithecker]

As said, I've used the convert tool from incubator-netbeans-tools. The sources of it show that for .properties files it expects the original license to be at the top. That could be adapted to also detect a valid but misplaced license somewhere in the middle of the file and consequently remove it and insert the new Apache license at the top of the file.

My question is what to do with the (text) files which were donated without any license at all.

[matthiasblaesing]

The files, that don't have a license, but could hold one (property files, java files, ...) need one added. There are options to relex that requirement if the text is used in the GUI and adding the license header would interfere with the display.

[borisheithecker]

And no action is required for binary files (image resources)?

[matthiasblaesing]

Ah sorry - good question. For the regular modules this is solved by introducing an "licenseinfo.xml". That files contains the necessary data. See here for an example: https://github.com/apache/incubator-netbeans/blob/master/ide/image/licenseinfo.xml

This might be interesting: https://cwiki.apache.org/confluence/display/NETBEANS/Legal+FAQ Specificly: https://cwiki.apache.org/confluence/display/NETBEANS/Legal+FAQ#LegalFAQ-licenseinfoPerfileinfo-licenseinfo.xml

[borisheithecker]

I was somehow relying on the conversion tool, but now, reviewing the converted files, I understand that it far from converts everything. It's a great number of files and fixing them all is incremental work and it's definitely going to take much more time. I'm making small commits to make changes trackable - for now fixing the original license headers so they can be auto converted later. I'm adapting the convert tool from incubator-netbeans-tools to catch more of the uncomplying licensing cases. It's on github (https://github.com/borisheithecker/incubator-netbeans-tools/tree/netbeans-l10n), my working branch is netbeans-l10n.

[borisheithecker]

Yes, the first PR should deal only with the licenses. But this is not yet done completely. For the next steps, we need to think about how the contribute should be used in the future. Do we want a multilingual IDE or only localisation support for platform applications?

[borisheithecker]

Another question: There are several files in this PR which contain licensing statements not in the file header but as a localisable string to be displayed to the user (see, for example, netbeans-l10n/src/af/other/installer/infra/build/web/js/common_strings_af.js). These pieces of text surely will have to be updated, too, but does is have to be done for the first PR already?

[matthiasblaesing]

No I don't think these changes are strictly necessary. However, at some point we either need to remove these translations or update them to the correct information. If you encounter such properties, creating an issue for updating these would be great.

[geertjanw]

What is the status of this?

[borisheithecker]

This is much more work than expected. More than 1300 license headers pending to be reviewed and fixed individually, one by one. Even after a couple of improvements on the conversion tool. We need it, esp. the german language pack, and I still feel committed to it. I can't resume work on it before July 20.

[matthiasblaesing]

@borisheithecker thank you for working on this! I'll mark this PR as work-in-progress, so that people know the state.

[junichi11]

I suggest we create a branch for l10n then merging this PR into it once. Some people can work on this per language. If OK, I'll create a new branch (e.g. 3rd-donation-l10n).

[borisheithecker]

What is the sense of integrating it into the main repository if we're building individual extensions per language? Another option could be to create a new repository that holds the extension plus the donated sources. It could be added as a git "submodule" to the main repository (if needed). Boris

On Tue, 30 Jul 2019 at 16:23, Junichi Yamamoto notifications@github.com wrote:

I suggest we create a branch for l10n then merging this PR into it once. Some people can work on this per language. If OK, I'll create a new branch (e.g. 3rd-donation-l10n).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/netbeans/pull/1167?email_source=notifications&email_token=AECA34PB4MXVYR5FQN2IEITQCBFGZA5CNFSM4G7BZN72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3EENOI#issuecomment-516441785, or mute the thread https://github.com/notifications/unsubscribe-auth/AECA34PONP6KVLRYPCJ4I33QCBFGZANCNFSM4G7BZN7Q .

-- Boris Heithecker

Dr. Boris Heithecker Lüneburger Str. 30 28870 Ottersberg Festnetz: +49 4205 315834 Mobil: +49 170 6137015

[borisheithecker]

What I'm wondering is, why does this pull request pass the CI test if it's disapproved of by the rat job because of missing licenses?

[junichi11]

What is the sense of integrating it into the main repository if we're building individual extensions per language?

I meant that we should create PRs to change the license headers per language because this integration is so huge. It's hard work to do it only one person and it takes much time :) Also see @JaroslavTulach 's comment.

Thanks.

[matthiasblaesing]

What I'm wondering is, why does this pull request pass the CI test if it's disapproved of by the rat job because of missing licenses?

The rat check in the CI run does not check the git repository, but the released source zip. As the language files are not part of the build, they don't cause a build failure.

[kishida]

What is the status?

[junichi11]

Here: https://github.com/apache/netbeans-l10n

Maybe, we can close this, I suppose.

[borisheithecker]

Close this, as the repository has been created. Work is going to be continued on the translations, eyeing a first release. I need to know how to proceed with the license header adaptions in a way that meets all legal requirements.

[junichi11]

Closing.