search

odpi / egeria

Egeria core
https://egeria-project.org
Apache License 2.0
812 stars 261 forks source link

Git Clone on Windows advisory #4259

Closed exonianp closed 3 years ago

exonianp commented 4 years ago

When downloading the project on Windows one is confronted with the issue of the long file names.

`? Choose default git protocol HTTPS

C:\Projects\Egeria>gh repo clone odpi/egeria Cloning into 'egeria'... remote: Enumerating objects: 22, done. remote: Counting objects: 100% (22/22), done. remote: Compressing objects: 100% (22/22), done. remote: Total 297927 (delta 13), reused 1 (delta 0), pack-reused 297905 Receiving objects: 100% (297927/297927), 168.42 MiB | 31.26 MiB/s, done. Resolving deltas: 100% (154982/154982), done. warning: unable to access 'open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-encrypted-file-store-connector/src/main/java/org/odpi/openmetadata/adapters/adminservices/configurationstore/encryptedfile/.gitattributes': Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-encrypted-file-store-connector/src/main/java/org/odpi/openmetadata/adapters/adminservices/configurationstore/encryptedfile/DocStoreErrorCode.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-encrypted-file-store-connector/src/main/java/org/odpi/openmetadata/adapters/adminservices/configurationstore/encryptedfile/EncryptedFileBasedServerConfigStoreConnector.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-encrypted-file-store-connector/src/main/java/org/odpi/openmetadata/adapters/adminservices/configurationstore/encryptedfile/EncryptedFileBasedServerConfigStoreProvider.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-encrypted-file-store-connector/src/main/java/org/odpi/openmetadata/adapters/adminservices/configurationstore/encryptedfile/package-info.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-encrypted-file-store-connector/src/test/org/odpi/openmetadata/adapters/adminservices/configurationstore/encryptedfile/TestEncryptedFileBasedServerConfigStoreConnector.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-file-store-connector/src/main/java/org/odpi/openmetadata/adapters/adminservices/configurationstore/file/FileBasedServerConfigStoreConnector.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/configuration-store-connectors/configuration-file-store-connector/src/main/java/org/odpi/openmetadata/adapters/adminservices/configurationstore/file/FileBasedServerConfigStoreProvider.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/inmemory-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/inmemory/InMemoryOpenMetadataTopicConnector.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/inmemory-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/inmemory/InMemoryOpenMetadataTopicProvider.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaConfigurationWrapper.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaOpenMetadataEventConsumer.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaOpenMetadataEventConsumerConfiguration.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaOpenMetadataEventConsumerProperty.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaOpenMetadataEventProducer.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaOpenMetadataTopicConnector.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaOpenMetadataTopicConnectorAuditCode.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaOpenMetadataTopicConnectorErrorCode.java: Filename too long error: unable to create file open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/src/main/java/org/odpi/openmetadata/adapters/eventbus/topic/kafka/KafkaOpenMetadataTopicProvider.java: Filename too long fatal: cannot create directory at 'open-metadata-implementation/adapters/open-connectors/governance-daemon-connectors/data-platform-connectors/cassandra-metadata-extractor-connector/src/main/java/org/odpi/openmetadata/adapters/connectors/metadataextractor/cassandra': Filename too long warning: Clone succeeded, but checkout failed. You can inspect what was checked out with 'git status' and retry with 'git restore --source=HEAD :/'

exit status 128`

The solution to this is to issue this command: git config --system core.longpaths true

Provided that you use Windows, version 1607 and higher where MAX_PATH limitations have been removed from common Win32 file and directory functions. However, you must opt-in to the new behavior. A registry key allows you to enable or disable the new long path behavior. To enable long path behavior (set to 1) the registry key at HKLM\SYSTEM\CurrentControlSet\Control\FileSystem LongPathsEnabled (Type: REG_DWORD)

This fix works on terminal mode, but it does not yet work while using the github desktop client.

grahamwallis commented 4 years ago

@exonianp Thank you for capturing this. Sounds like we should add this into the documentation.

planetf1 commented 3 years ago

This was originally raised in https://github.com/odpi/egeria/issues/36 and implemented in https://github.com/odpi/egeria/commit/66341030c5261aaaca31e9a20428961d18227ab7 in the Community Guide - at least in terms of the git config. I have mentioned the reg change to some when intiially investigating but I do not believe it's needed purely for git.

However more recently it seems this was moved to the dojo docs at https://github.com/odpi/egeria/blob/316c63767ef36ada1ecf1e266ca812059af39c9a/open-metadata-resources/open-metadata-tutorials/git-and-git-hub-tutorial/task-set-long-path-names.md which is less obvious except for someone following the dojo in detail.

The developer resources at https://github.com/odpi/egeria/tree/master/developer-resources are where we have much of this information, but some of the guides are more focussed on a developer changing egeria -- I think we need a more simple 'Building.md' and similar that are probably in the top level, or close thereto - the example here is so obviously needed before a developer or user even gets as far as more detail on egeria

This brings up two points

a) The docs are not clear enough - the docs are in the detailed dojo only which is way to far hidden b) The registry update may be needed for tools other than git that may run against that deep source tree, and so is a good thing to advise in any case

planetf1 commented 3 years ago

To add - many thanks again for raising. Glad you are up and running now. It's good to get the direct feedback so we can try and make it as easy as possible to get started. If you have any thoughts about what you needed / where you would expect to find it please do add below :-) Thanks!

planetf1 commented 3 years ago

Going to reopen - the docs need to be better

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

planetf1 commented 3 years ago

I just setup a new test windows environment, and can confirm the git config command was sufficient to allow a git clone/build to work

exonianp commented 3 years ago

Thank you Nigel! I will do over the weekend because I plan to setup it up in Virtualbox Windows VM (instead of its current installation in AWS); so I will do the whole process from scratch and let you know.

planetf1 commented 3 years ago

Unfortunately there appear to be other issues in the windows build too - see https://github.com/odpi/egeria/issues/4917

I believe the FVTs are not running correctly - it's probably a simple error with path separators or directory names, or perhaps process launching (the FVTs launch an egeria platform, configure it, then run tests against it). When I went to debug that yesterday I hit another build issue in our CTS code with apparent symbolic links. However due to priorities I put it on the back burner as most developers working on the project are using MacOS or Linux, and runtimes are basically all linux. I don't exactly have a properly working windows machine but setup a new cloud instance this week...

So I expect you'll hit an issue. Is windows important to you? Are you using an IDE or just building via the command line? Are you familiar with WSL/WSL2 (since that's a 'proper' linux environment so the build should work as normal).

I'm just trying to dig into the priority - if it's a blocker for you I will try and revisit the issue for you. If you do get a chance to run through it would be helpful - let me know how far it gets - since it's also possible it's just our test environment in github thats bad ....

exonianp commented 3 years ago

Do not worry about it Nigel. I can just as easily set it up on Linux.

It just so happened that I felt it would have been better to transfer data from the MS SQL server that was running on Windows I was using at the time.

Now I have set up MS SQL Server on kubuntu and it works fine, so I will be happy to set up everything on Kubuntu...

Here is the proof, that I am already set up in that end, I will just do the installation on this machine ;)

image

So do not worry about it. ;)

planetf1 commented 3 years ago

See https://github.com/odpi/egeria/pull/5084 which proposes use of WSL2 only to address this issue

planetf1 commented 3 years ago

Closing as per docs update above