search

odpi / egeria

Egeria core
https://egeria-project.org
Apache License 2.0
795 stars 259 forks source link

Windows build failure (merge) #4917

Closed planetf1 closed 3 years ago

planetf1 commented 3 years ago

The Windows Egeria Merge build is failing

https://github.com/odpi/egeria/actions/workflows/windows-daily.yml

For example for one specific log: See https://github.com/odpi/egeria/runs/2126504680?check_suite_focus=true

This looks like an issue with launching the chassis for FVT tests. I made changes in this area recently under #4896

Linux/macOS are fine

planetf1 commented 3 years ago

Trying to debug this I also hit build problems in the conformance suite, which seemed to be caused by symbolic links . Will add notes to readme for 2.9 & leave to future release

planetf1 commented 3 years ago

One option is to consider whether we advise windows users to work within WSL2 (the linux environment supported in windows which can run a full ubuntu etc environment) to avoid these platform specific issues and improve performance 3x

The one gotcha will be around IDE integration.

Are there any windows based developers that would like to take this issue on to work out what works best? Most of the core team are using MacOS and/or Linux

planetf1 commented 3 years ago

Building on a clean machine, the first issue hit is in the conformance suite:

[INFO] 
[INFO] --- apache-rat-plugin:0.13:check (rat-check) @ open-metadata-conformance-suite ---
[INFO] Added 3 additional default licenses.
[INFO] Enabled default license matchers.
[INFO] Added 2 custom approved licenses.
[INFO] Will parse SCM ignores for exclusions...
[INFO] Finished adding exclusions from SCM ignore files.
[INFO] 66 implicit excludes (use -debug for more details).
[INFO] 46 explicit excludes (use -debug for more details).
[INFO] 176 resources included (use -debug for more details)
[INFO] Rat check: Summary over all files. Unapproved: 29, unknown: 29, generated: 0, approved: 147 licenses.

The rat report shows :

29 Unknown Licenses

*****************************************************

Files with unapproved licenses:

  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-classification-search-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-classification-update-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-classification-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-creation-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-declassification-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-delete-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-history-retrieval-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-purge-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-re-home-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-restore-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-retrieval-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-search-history-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-search-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-undo-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-update-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-graph-history-query-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-graph-query-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-performance-environment-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-creation-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-delete-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-history-retrieval-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-purge-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-re-home-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-restore-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-retrieval-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-search-history-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-search-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-undo-performance-test-case.md
  C:/Users/nigel/IdeaProjects/egeria/open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-update-performance-test-case.md
planetf1 commented 3 years ago

If we take an example of one of these files, they contain a simple stub ie:

../profiles/classification-search/README.md

which seems to indicate the target of the link, but the rat check doesn't understand that ...

Running a query on a linux environment shows the same list of files are symbolic links checked into git

➜  egeria git:(master) git ls-files -s | awk '/120000/{print $4}'
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-classification-search-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-classification-update-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-classification-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-creation-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-declassification-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-delete-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-history-retrieval-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-purge-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-re-home-performance-test-case.mdopen-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-restore-performance-test-case.mdopen-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-retrieval-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-search-history-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-search-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-undo-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-entity-update-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-graph-history-query-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-graph-query-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-performance-environment-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-creation-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-delete-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-history-retrieval-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-purge-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-re-home-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-restore-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-retrieval-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-search-history-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-search-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-undo-performance-test-case.md
open-metadata-conformance-suite/docs/performance-workbench/test-cases/repository-relationship-update-performance-test-case.md

Given that the windows git client doesn't properly handle symlinks I think we need to avoid using them if possible. Or we could skip certain areas on windows, or agreeing not to support. It's unfortunate since windows/NTFS does support a variety of links - hardlinks, symbolic links, and junction

cc: @cmgrote

planetf1 commented 3 years ago

Will continue checking for other issues...

planetf1 commented 3 years ago

It looks like the config setting 'core.symlinks=true' may help assuming the user is using WIndows 10, NTFS or another fs with symlink support, AND is in developer mode ???

Will retry (after next tests of FVT -- windows build is very slow). If that works then I'm inclined to add docs to the project and not revert the symlink change.

Additionally it's worth noting that intellij now supports IDE usage on windows where the actual build/file management is run under WLS2 (linux kernel) which should avoid all of these kind of issues. this may also be worth recommending.

planetf1 commented 3 years ago

The following does not work

It only seems to work if, as documented, WIndows 'developer mode' is enabled.

Will repeat a build...

planetf1 commented 3 years ago

With that change, the conformance test build completes. I will later raise a PR to add notes to the windows setup documentation.

The build now fails with:

[INFO] --- process-exec-maven-plugin:0.9:start (chassis-start) @ asset-consumer-fvt ---
[INFO] arg: java
[INFO] arg: -Dserver.port=10443
[INFO] arg: -Dloader.path=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/server/lib
[INFO] arg: -Dspringdoc.api-docs.enabled=false
[INFO] arg: -Djavax.net.ssl.trustStore=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/truststore.p12
[INFO] arg: -Dserver.ssl.trust-store=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/truststore.p12 [INFO] arg: -jar
[INFO] arg: ../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/server/server-chassis-spring-2.9-SNAPSHOT.jar
[INFO] Full command line: java -Dserver.port=10443 -Dloader.path=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/server/lib -Dspringdoc.api-docs.enabled=false -Djavax.net.ssl.trustStore=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/truststore.p12 -Dserver.ssl.trust-store=../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/truststore.p12 -jar ../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/server/server-chassis-spring-2.9-SNAPSHOT.jar
[INFO] Starting process: chassis-start
[INFO] Using working directory for this process: C:\Users\nigel\IdeaProjects\egeria\open-metadata-test\open-metadata-fvt\access-services-fvt\asset-consumer-fvt\target\classes
[ERROR] Error: Unable to access jarfile ../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/server/server-chassis-spring-2.9-SNAPSHOT.jar
[INFO] Started process: chassis-start
[INFO]
[INFO] --- groovy-maven-plugin:2.1.1:execute (server-start) @ asset-consumer-fvt ---
=== Checking platform at https://localhost:10443 is available (12 attempts remaining) ===
=== Checking platform at https://localhost:10443 is available (11 attempts remaining) ===
=== Checking platform at https://localhost:10443 is available (10 attempts remaining) ===
=== Checking platform at https://localhost:10443 is available (9 attempts remaining) ===
[INFO] Stopping all processes ...
[INFO] Stopping process: chassis-start
[INFO] Stopped process: chassis-start

This looks as if it may be some kind of path length/parsing issue ie

nigel@MEGA C:\Users\nigel\IdeaProjects\egeria\open-metadata-test\open-metadata-fvt\access-services-fvt\asset-consumer-fvt\target\classes
$ dir ..\..\..\..\..\..\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution\egeria-omag-2.9-SNAPSHOT
 Volume in drive C is Win10
 Volume Serial Number is 765D-6B1A

 Directory of C:\Users\nigel\IdeaProjects\egeria\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution

2021-04-15  18:48    <DIR>          egeria-omag-2.9-SNAPSHOT
               0 File(s)              0 bytes
               1 Dir(s)  23,115,759,616 bytes free

nigel@MEGA C:\Users\nigel\IdeaProjects\egeria\open-metadata-test\open-metadata-fvt\access-services-fvt\asset-consumer-fvt\target\classes
$ dir ..\..\..\..\..\..\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution\egeria-omag-2.9-SNAPSHOT\*
The system cannot find the path specified.

nigel@MEGA C:\Users\nigel\IdeaProjects\egeria\open-metadata-test\open-metadata-fvt\access-services-fvt\asset-consumer-fvt\target\classes
$ dir ..\..\..\..\..\..\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution\egeria-omag-2.9-SNAPSHOT\
The system cannot find the path specified.

nigel@MEGA C:\Users\nigel\IdeaProjects\egeria\open-metadata-test\open-metadata-fvt\access-services-fvt\asset-consumer-fvt\target\classes
$ dir ..\..\..\..\..\..\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution\egeria-omag-2.9-SNAPSHOT
 Volume in drive C is Win10
 Volume Serial Number is 765D-6B1A

 Directory of C:\Users\nigel\IdeaProjects\egeria\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution

2021-04-15  18:48    <DIR>          egeria-omag-2.9-SNAPSHOT
               0 File(s)              0 bytes
               1 Dir(s)  23,115,759,616 bytes free

nigel@MEGA C:\Users\nigel\IdeaProjects\egeria\open-metadata-test\open-metadata-fvt\access-services-fvt\asset-consumer-fvt\target\classes
$ cd  ..\..\..\..\..\..\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution

nigel@MEGA C:\Users\nigel\IdeaProjects\egeria\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution
$ dir egeria-omag-2.9-SNAPSHOT
 Volume in drive C is Win10
 Volume Serial Number is 765D-6B1A

 Directory of C:\Users\nigel\IdeaProjects\egeria\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution\egeria-omag-2.9-SNAPSHOT

2021-04-15  18:48    <DIR>          .
2021-04-15  18:48    <DIR>          ..
2021-04-15  18:48    <DIR>          conformance-suite
2021-04-15  18:48    <DIR>          content-packs
2021-04-15  18:14            11,689 LICENSE
2021-04-15  18:14               276 NOTICE
2021-04-15  18:48    <DIR>          sample-data
2021-04-15  18:48    <DIR>          samples
2021-04-15  18:48    <DIR>          server
2021-04-15  18:14             1,866 truststore.p12
2021-04-15  18:48    <DIR>          user-interface
2021-04-15  18:48    <DIR>          utilities
               3 File(s)         13,831 bytes
               9 Dir(s)  23,115,759,616 bytes free

nigel@MEGA C:\Users\nigel\IdeaProjects\egeria\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution
$

Note that the attempt to access the contents of the directory by relative path failed -- not found, yet a 'cd' then trying to list the remainder of the path worked ok....

planetf1 commented 3 years ago

This could be a limt of 260 chars in path - see https://www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over-260-characters/

To fix this -> https://www.howtogeek.com/266621/how-to-make-windows-10-accept-file-paths-over-260-characters/

ie edit HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled to 1

planetf1 commented 3 years ago

After that change the file can be accessed at the CLI, but the windows java process still fails ie

$ dir "../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/server/server-chassis-spring-2.9-SNAPSHOT.jar"
 Volume in drive C is Win10
 Volume Serial Number is 765D-6B1A

 Directory of C:\Users\nigel\IdeaProjects\egeria\open-metadata-distribution\open-metadata-assemblies\target\egeria-2.9-SNAPSHOT-distribution\egeria-omag-2.9-SNAPSHOT\server

2021-04-15  18:35       144,336,410 server-chassis-spring-2.9-SNAPSHOT.jar
               1 File(s)    144,336,410 bytes
               0 Dir(s)  23,115,759,616 bytes free

nigel@MEGA C:\Users\nigel\IdeaProjects\egeria\open-metadata-test\open-metadata-fvt\access-services-fvt\asset-consumer-fvt\target\classes
$ java -jar "../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/server/server-chassis-spring-2.9-SNAPSHOT.jar"
Error: Unable to access jarfile ../../../../../../open-metadata-distribution/open-metadata-assemblies/target/egeria-2.9-SNAPSHOT-distribution/egeria-omag-2.9-SNAPSHOT/server/server-chassis-spring-2.9-SNAPSHOT.jar
planetf1 commented 3 years ago

I'm inclined to suggest windows users build under WSL2 which will behave similarly to linux and macOS. If a native windows build is desired then pull requests to fix issues can be submitted.

Any readers care to comment on this idea?

planetf1 commented 3 years ago

^ @mandy-chessell

planetf1 commented 3 years ago

See https://github.com/odpi/egeria/pull/5084 which proposes use of WSL2 only to address this issue