search

aws / aws-iot-device-sdk-cpp-v2

Next generation AWS IoT Client SDK for C++ using the AWS Common Runtime
Apache License 2.0
183 stars 108 forks source link

Windows git checkout fails if the base directory is longer than 60 characters #157

Closed EeroKurimo closed 4 years ago

EeroKurimo commented 4 years ago

Confirm by changing [ ] to [x] below to ensure that it's a bug:

Describe the bug

Git on Windows have characters limits for paths, and the latest dependencies are causing some build artifacts to be ~200 characters long, failing the build if the root directory of the build is close to or over 60 characters. The normally recommended git setting core.longpaths=true doesn't resolve the issue.

Platform/OS/Device

Windows 10

SDK version number

v1.8.0

To Reproduce (observed behavior)

With an admin account:

git config --system core.longpaths true
mkdir C:\data\awstemp_1234567890123456789012345678901234567890\sdk-cpp-workspace
pushd C:\data\awstemp_1234567890123456789012345678901234567890\sdk-cpp-workspace
git clone -c core.longpaths=true --recursive https://github.com/aws/aws-iot-device-sdk-cpp-v2.git

Logs/output

< ✂ >
Cloning into 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs'...
fatal: '$GIT_DIR' too big
fatal: clone of 'https://github.com/awslabs/aws-templates-for-cbmc-proofs.git' into submodule path 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs' failed
Failed to clone 'verification/cbmc/aws-templates-for-cbmc-proofs'. Retry scheduled
Cloning into 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs'...
fatal: '$GIT_DIR' too big
fatal: clone of 'https://github.com/awslabs/aws-templates-for-cbmc-proofs.git' into submodule path 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs' failed
Failed to clone 'verification/cbmc/aws-templates-for-cbmc-proofs' a second time, aborting
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-compression': checked out 'f2be13afe410611fcac07b6519b96ce1ad4e4831'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-event-stream': checked out '873f1c035a5b6b4698280ee3798d1db5cc9ce86c'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-http': checked out '98b3e4979fc41f54c13a68475919fa56b5377aef'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-io': checked out '919c27a6c0369c71770cb37f855cea67e81a8d4a'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-mqtt': checked out '199352c90c8a7b38b68e9d285538f32fec774f0d'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-checksums': checked out 'fb96b3e964fe9bde2625c3ac9547e54d6c802211'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n': checked out '1616f313f69c168c141d3e78f34cd1f9da920f3a'
Submodule 'tests/cbmc/aws-templates-for-cbmc-proofs' (https://github.com/awslabs/aws-templates-for-cbmc-proofs) registered for path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs'
Cloning into 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs'...
fatal: '$GIT_DIR' too big
fatal: clone of 'https://github.com/awslabs/aws-templates-for-cbmc-proofs' into submodule path 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs' failed
Failed to clone 'tests/cbmc/aws-templates-for-cbmc-proofs'. Retry scheduled
Cloning into 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs'...
fatal: '$GIT_DIR' too big
fatal: clone of 'https://github.com/awslabs/aws-templates-for-cbmc-proofs' into submodule path 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs' failed
Failed to clone 'tests/cbmc/aws-templates-for-cbmc-proofs' a second time, aborting
Failed to recurse into submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common'
Failed to recurse into submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n'
Failed to recurse into submodule path 'aws-common-runtime/aws-crt-cpp'
EeroKurimo commented 4 years ago

The same fatal: '$GIT_DIR' too big error also happens if your configure core.longpaths=true in your own .gitconfig file instead of the system global settings.

jmklix commented 4 years ago

On top of setting the core.longpaths=true you should also enable long windows file paths. Here is a guide on how to do that. Please let me know if it still doesn't work for you.

EeroKurimo commented 4 years ago

Oh, that closed fast. Hopefully this can be reopened, since I now had a chance to try it out, and it still does not work.

To avoid that the problem is just my machine, I tried this on a Windows 10 VM, and after enabling long path support by setting the valueLongPathsEnabled = 1 at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem, and then installing the latest 64-bit git 2.28.0 for windows from https://git-scm.com/, running the checkout will still fails. Rebooting before the checkout (which should not be needed, but who knows) also does not help.

Here's the commands I ran:

reg add HKLM\SYSTEM\CurrentControlSet\Control\FileSystem /f /v LongPathsEnabled /t REG_DWORD /d 1
git config --system core.longpaths true
mkdir C:\data\awstemp_1234567890123456789012345678901234567890\sdk-cpp-workspace
pushd C:\data\awstemp_1234567890123456789012345678901234567890\sdk-cpp-workspace
git clone -c core.longpaths=true --recursive https://github.com/aws/aws-iot-device-sdk-cpp-v2.git

Here's the relevant part of the output:

<Everything goes fine until this point>
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-auth': checked out 'e2c57c40c8910eec7d69467692062f4bafcab4a5'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-cal': checked out '2b9c589b296bad0c411e5014222739c0edef5f03'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common': checked out '4a21a1c0757083a16497fea27886f5f20ccdf334'
Submodule 'tests/cbmc/aws-templates-for-cbmc-proofs' (https://github.com/awslabs/aws-templates-for-cbmc-proofs.git) registered for path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs'
Cloning into 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs'...
fatal: '$GIT_DIR' too big
fatal: clone of 'https://github.com/awslabs/aws-templates-for-cbmc-proofs.git' into submodule path 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs' failed
Failed to clone 'verification/cbmc/aws-templates-for-cbmc-proofs'. Retry scheduled
Cloning into 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs'...
fatal: '$GIT_DIR' too big
fatal: clone of 'https://github.com/awslabs/aws-templates-for-cbmc-proofs.git' into submodule path 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common/verification/cbmc/aws-templates-for-cbmc-proofs' failed
Failed to clone 'verification/cbmc/aws-templates-for-cbmc-proofs' a second time, aborting
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-compression': checked out 'f2be13afe410611fcac07b6519b96ce1ad4e4831'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-event-stream': checked out '873f1c035a5b6b4698280ee3798d1db5cc9ce86c'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-http': checked out '8f706b21f36c5cd786dd34ae3eb6b5be5d6e5be2'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-io': checked out '7fef123ea803d20a833c17c243b06506b78911f4'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-mqtt': checked out 'f4634671e6ef7674dde2fe3ae4f3dffacc1e87dc'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-checksums': checked out 'fb96b3e964fe9bde2625c3ac9547e54d6c802211'
Submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n': checked out '1616f313f69c168c141d3e78f34cd1f9da920f3a'
Submodule 'tests/cbmc/aws-templates-for-cbmc-proofs' (https://github.com/awslabs/aws-templates-for-cbmc-proofs) registered for path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs'
Cloning into 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs'...
fatal: '$GIT_DIR' too big
fatal: clone of 'https://github.com/awslabs/aws-templates-for-cbmc-proofs' into submodule path 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs' failed
Failed to clone 'tests/cbmc/aws-templates-for-cbmc-proofs'. Retry scheduled
Cloning into 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs'...
fatal: '$GIT_DIR' too big
fatal: clone of 'https://github.com/awslabs/aws-templates-for-cbmc-proofs' into submodule path 'C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs' failed
Failed to clone 'tests/cbmc/aws-templates-for-cbmc-proofs' a second time, aborting
Failed to recurse into submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/aws-c-common'
Failed to recurse into submodule path 'aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n'
Failed to recurse into submodule path 'aws-common-runtime/aws-crt-cpp'

C:\data\awstemp_1234567890123456789012345678901234567890\sdk-cpp-workspace>

Based on the fatal: '$GIT_DIR' too big error, my colleague Risto Kankkunen found that the error is likely coming from the directory C:/data/awstemp_1234567890123456789012345678901234567890/sdk-cpp-workspace/aws-iot-device-sdk-cpp-v2/aws-common-runtime/aws-crt-cpp/aws-common-runtime/s2n/tests/cbmc/aws-templates-for-cbmc-proofs exceeding PATH_MAX - 40 at https://github.com/git/git/blob/7814e8a05a59c0cf5fb186661d1551c75d1299b5/setup.c#L812:

static const char *setup_explicit_git_dir(const char *gitdirenv,
                      struct strbuf *cwd,
                      struct repository_format *repo_fmt,
                      int *nongit_ok)
{
    const char *work_tree_env = getenv(GIT_WORK_TREE_ENVIRONMENT);
    const char *worktree;
    char *gitfile;
    int offset;

    if (PATH_MAX - 40 < strlen(gitdirenv))
        die(_("'$%s' too big"), GIT_DIR_ENVIRONMENT);
jmklix commented 4 years ago

It is not recommended to use file paths longer than 260 characters. When trying to use long paths you run into inconsistencies because of how each app handles it. The only solution is to use a shorter starting file path.

bmsebastian2 commented 2 years ago

Amazing, thank yo...

ghost commented 1 year ago

I've got the same problem on the espressif/esp-matter repository. I have long paths enabled in my windows registry, long paths enabled on my git install, and git-lfs installed, and it still doesn't work. I think I'm going to dump the windows environment and use a linux vm.

Hazard-Hat commented 1 year ago

What Ghost said, none of this works at all.

Here's hoping a Linux VM downloading Git repositories and porting them to the Host machine will work.