Maintain shallow clone when git pull-ing

[adnan360]

This maintains shallow clone status when updating already cloned git repos by previous build attempts. This should keep size smaller (and have a single latest commit) as expected, similar to a new shallow clone.

As posted in this comment before.

NOTE: I haven't tested it yet!! Just posting so that it can be discussed or tested. Especially if the commands work for everyone. Also I'm not sure if --rebase would be a safe option here. It helped me update a very old repo once without any error messages so I included it here. Feel free to test with older cloned dependency repos!

[Alexey-T]

you have not tested it yet? i must wait until it happens.

[adnan360]

I tested it a couple of times. I think the 'git pull' and other commands are running even when the repo has just been cloned which is unnecessary.

So changed this:

    do  
        ...
        [ ! -d "$temp/.git" ] && git clone --depth 1 "$i"   
        cd "$temp"
        git pull --depth 1 --rebase origin master
        git tag -d $(git tag -l)
        git reflog expire --expire=all --all
        git gc --prune=all
        cd ../
    done

into this:

    do  
        ...
        if [ ! -d "$temp/.git" ]; then
            git clone --depth 1 "$i"    
        else
            cd "$temp"
            git pull --depth 1 --rebase origin master
            git tag -d $(git tag -l)
            git reflog expire --expire=all --all
            git gc --prune=all
            cd ../
        fi
    done

This should make the first clone at least faster.

Would like to skip "git gc" (because it takes a bit of time to run) and other commands if we can detect there are no new commits. Feel free to suggest anything.

[adnan360]

Got an idea. Changed to this:

        if [ ! -d "$temp/.git" ]; then
            git clone --depth 1 "$i"    
        else
            cd "$temp"
            last_commit="$(git log -n 1 --pretty=format:'%H')"
            git pull --depth 1 --rebase origin master
            if [ "$last_commit" != "$(git log -n 1 --pretty=format:'%H')" ]; then
                # There are new commits, so make size smaller like a new shallow clone
                git tag -d $(git tag -l)
                git reflog expire --expire=all --all
                git gc --prune=all
            fi
            cd ../
        fi

It checks for the last commit hash and compares it after git pull. If it is not same, assumes new commits came in and runs git gc and other commands. Otherwise skips them making it faster.

[Alexey-T]

@veksha Can you review this? I dont understand the patch. code of project is not mine.

[veksha]

1. found out that if [ ! -d "$temp/.git" ] failed on my system, I had Carriage Return char (\r) at the end of folder names. fixed it. (By default, Git on Windows converts line endings to CRLF when checking out files, and to LF when committing files.)
2. added git stash before pulling and git stash pop after. (because my cudatext.lpi file always has changes, changed cudatext.exe target path in Lazarus. so stashing and restoring stash is convenient). is it ok for you?
3. changed pull command to git pull --depth 1 --rebase. ok?
4. added echo commands.

@adnan360 @Alexey-T What do you say?

diff --git a/cudaup.sh b/cudaup.sh
index 8d1e43a..e67378e 100755
--- a/cudaup.sh
+++ b/cudaup.sh
@@ -114,12 +114,27 @@ then
    cd src
    for i in $Repos
    do  
+       i=${i%%[[:space:]]} # removes line breaks from the right end of the var!
        temp=${i/'https://github.com/Alexey-T/'/''}
        temp=${temp/'https://github.com/bgrabitmap/'/''}
-       [ ! -d "$temp/.git" ] && git clone --depth 1 "$i"   
-       cd "$temp"
-       git pull
-       cd ../
+       if [ ! -d "$temp/.git" ]; then
+           echo Clonning "$i"
+           git clone --depth 1 "$i"    
+       else
+           cd "$temp"
+           echo Pulling "$i"
+           last_commit="$(git log -n 1 --pretty=format:'%H')"
+           git stash > /dev/null
+           git pull --depth 1 --rebase
+           git stash pop > /dev/null 2>&1
+           if [ "$last_commit" != "$(git log -n 1 --pretty=format:'%H')" ]; then
+               # There are new commits, so make size smaller like a new shallow clone
+               git tag -d $(git tag -l)
+               git reflog expire --expire=all --all
+               git gc --prune=all
+           fi
+           cd ../
+       fi
    done
    cd ../
 fi

[Alexey-T]

@veksha Thanks, it seems to me your path is OK (but I don't know Bash). applied. PR cannot be merged now - conflicts.

[veksha]

please see https://github.com/Alexey-T/CudaText_up/pull/22 (do not pull tags. because they will be removed by git tag -d $(git tag -l) anyway.)

[veksha]

saw some errors/messages:

 + 174b98d...02c13b7 master     -> origin/master  (forced update)
fatal: refusing to merge unrelated histories

remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0

changed to git pull --depth 1 --rebase --no-tags --allow-unrelated-histories --quiet

will test it.

[veksha]

very strange behavior. calling first time: some fatal errors. calling second time: all ok.

first call:

Pulling https://github.com/bgrabitmap/bgrabitmap
Pulling https://github.com/Alexey-T/EncConv
Pulling https://github.com/Alexey-T/ATBinHex-Lazarus
Pulling https://github.com/Alexey-T/ATFlatControls
Pulling https://github.com/Alexey-T/ATSynEdit
Pulling https://github.com/Alexey-T/ATSynEdit_Cmp
fatal: Not possible to fast-forward, aborting.
Pulling https://github.com/Alexey-T/EControl
error: Could not read ccd79993d5e7e43a530c37ca15f2e0cd2ef572f2
error: Could not read ccd79993d5e7e43a530c37ca15f2e0cd2ef572f2
error: Could not read ccd79993d5e7e43a530c37ca15f2e0cd2ef572f2
Pulling https://github.com/Alexey-T/ATSynEdit_Ex
Pulling https://github.com/Alexey-T/Python-for-Lazarus
Pulling https://github.com/Alexey-T/Emmet-Pascal
Pulling https://github.com/Alexey-T/CudaText

second call:

Pulling https://github.com/bgrabitmap/bgrabitmap
Pulling https://github.com/Alexey-T/EncConv
Pulling https://github.com/Alexey-T/ATBinHex-Lazarus
Pulling https://github.com/Alexey-T/ATFlatControls
Pulling https://github.com/Alexey-T/ATSynEdit
Pulling https://github.com/Alexey-T/ATSynEdit_Cmp
Deleted tag '2019.07.19' (was 2e7ac8f)
Deleted tag '2019.09.29' (was 37ecb4a)
Deleted tag '2020.01.14' (was b539177)
Deleted tag '2020.02.03' (was 52fbc42)
Deleted tag '2020.03.25' (was bc1077a)
Deleted tag '2020.06.03' (was def704d)
Deleted tag '2020.09.05' (was 3bf3efe)
Deleted tag '2020.09.20' (was 3cb4a81)
Deleted tag '2020.09.29' (was 0f8f37d)
Deleted tag '2020.10.04' (was 9d91a02)
Deleted tag '2020.10.11' (was db8c639)
Deleted tag '2021.01.12' (was 206b41b)
Deleted tag '2021.01.17' (was 7a5a613)
Deleted tag '2021.02.15' (was f73d34e)
Deleted tag '2021.02.28' (was f1751f6)
Deleted tag '2021.03.07' (was a53df7f)
Deleted tag '2021.03.08' (was 53f879b)
Deleted tag '2021.04.01' (was f5dbe88)
Deleted tag '2021.05.03' (was ea6888a)
Deleted tag '2021.07.09' (was 49dfbe6)
Deleted tag '2021.07.16' (was daddbb9)
Deleted tag '2021.07.20' (was 75f0621)
Deleted tag '2021.08.12' (was 364f15f)
Deleted tag '2021.08.20' (was 58c613e)
Deleted tag '2021.09.03' (was 7d78d0e)
Deleted tag '2021.09.14' (was b268e6e)
Deleted tag '2021.12.28' (was d6dea11)
Deleted tag '2022.01.21' (was 0914acd)
Deleted tag '2022.05.04' (was aa58946)
Deleted tag '2022.08.02' (was 2a5eb03)
Deleted tag '2022.08.28' (was 7ae41f8)
Deleted tag '2022.09.01' (was 2ad9f18)
Deleted tag '2022.09.18' (was 6c29139)
Deleted tag '2022.10.03' (was edf8f5d)
Deleted tag '2022.10.15' (was 2b451ec)
Deleted tag '2022.10.18' (was 5e879d9)
Deleted tag '2023.03.10' (was 127a7e7)
Deleted tag '2023.03.14' (was ae56852)
Deleted tag '2023.03.28' (was fb80caf)
Deleted tag '2023.04.04' (was c094616)
Deleted tag '2023.04.08' (was c7f8c9a)
Pulling https://github.com/Alexey-T/EControl
Pulling https://github.com/Alexey-T/ATSynEdit_Ex
Pulling https://github.com/Alexey-T/Python-for-Lazarus
Pulling https://github.com/Alexey-T/Emmet-Pascal
Pulling https://github.com/Alexey-T/CudaText

i will test this: git pull --depth 1 --no-tags --quiet