"repo init" attempts to delete the entire repository when cloning repo itself fails

[GoogleCodeExporter]

Often we use 'repo init' to update the branch on an existing repository. If one 
of the 'git clone' commands used by 'repo init' fails, it will delete the 
entire .repo directory and all subdirectories.

Deleting the entire repository just because of a issue with cloning is 
counterproductive. This repository might be many gigabytes in size.

Here is an example such output:

$ repo init --repo-url https://chromium.googlesource.com/external/repo 
--manifest-url 
https://chrome-internal-review.googlesource.com/chromeos/manifest-internal 
--manifest-name official.xml --manifest-branch master
Get https://chromium.googlesource.com/external/repo
fatal: remote error: 

Invalid user name or password.

Please generate a new password at:
  https://chromium.googlesource.com/new-password

Traceback (most recent call last):
  File "/b/depot_tools/repo", line 809, in <module>
    main(sys.argv[1:])
  File "/b/depot_tools/repo", line 779, in main
    os.rmdir(os.path.join(root, name))
OSError: [Errno 20] Not a directory: 
'.repo/projects/chromium/src/third_party/libmtp.git/svn'

You can see here that the code for deleting the .repo directory deleted much of 
the .repo directory, but not the entire directory. In subsequent runs, repo 
sync was broken because necessary files were deleted from 
.repo/projects/chromium/src/third_party/libmtp.git, so the repository needed to 
be manually repaired.

You can see the problematic code here:

try:
  _Init(args)
except CloneFailure:
  for root, dirs, files in os.walk(repodir, topdown=False):
    for name in files:
      os.remove(os.path.join(root, name))
      for name in dirs:
        os.rmdir(os.path.join(root, name))
  os.rmdir(repodir)
  sys.exit(1)

A few issues with the above code:
  - It does not check for symlinks. In the case where some of the "dirs" are actually symlinks we will need to use os.remove instead of os.rmdir. shutil.rmtree would be a good replacement for this block.
  - It deletes the whole .repo, even if there are 'projects' and 'project-objects' inside there. Probably it should just delete .repo/repo?
  - Quite possibly, it shouldn't delete anything at all just because a git clone failed.

Original issue reported on code.google.com by davidjames@google.com on 7 Mar 2014 at 1:32

[GoogleCodeExporter]

The implementation of the cleanup was changed in [1].  It will no longer throw 
the "Not a directory" error, and instead delete the entire .repo folder.

[1] https://gerrit-review.googlesource.com/#/c/55095/3

Original comment by david.pu...@sonymobile.com on 11 Jun 2014 at 12:30

[GoogleCodeExporter]

Hi David,

Is the problem you reported still existing? Or has the aforesaid solution for 
cleanup fixed the issue?

If the problem is still existing, kindly please provide the command yet again 
with the fresh repo and manifest url where its reproducible.

Best Regards, Sushan.

Original comment by ray.sus...@gmail.com on 6 Aug 2014 at 10:00

[GoogleCodeExporter]

The latest version of repo fixes some of the above problems, but still has 
others:

 try:
   _Init(args)
 except CloneFailure:
   shutil.rmtree(repodir, ignore_errors=True)
   sys.exit(1)

1. It deletes the whole .repo dir when it should just be deleting .repo/repo. 
This means it's deleting the whole repository.
2. It deletes .repo in the current directory when it should be deleting .repo 
from the root of the checkout.

Example reproduction recipe with any repo checkout:
  1. Delete .repo/repo
  2. Run repo init with an invalid --repo-url
  3. Notice that all of .repo is deleted now.

Original comment by davidjames@chromium.org on 6 Aug 2014 at 7:46

[GoogleCodeExporter]

Hi David,

I did the following:

1. Create the repo in my local workspace using init
testuser@01HW595640:~/workspace161$ repo init -u 
https://android.googlesource.com/platform/manifest
...
...
repo has been initialized in /home/testuser/workspace161

2. Checking the contents of .repo
testuser@01HW595640:~/workspace161$ cd .repo/
testuser@01HW595640:~/workspace161/.repo$ ls
manifests  manifests.git  manifest.xml  repo

testuser@01HW595640:~/workspace161/.repo$ cd repo/
testuser@01HW595640:~/workspace161/.repo/repo$ la
color.py     docs        .git             git_config.pyc  hooks             
pager.pyc     project.pyc    repo                trace.py
color.pyc    editor.py   .gitattributes   .gitignore      main.py           
progress.py   .pydevproject  repoc               trace.pyc
command.py   editor.pyc  git_command.py   git_refs.py     manifest_xml.py   
progress.pyc  .pylintrc      subcmds             wrapper.py
command.pyc  error.py    git_command.pyc  git_refs.pyc    manifest_xml.pyc  
.project      pyversion.py   SUBMITTING_PATCHES  wrapper.pyc
COPYING      error.pyc   git_config.py    git_ssh         pager.py          
project.py    pyversion.pyc  tests

3. Deleting the .repo/repo
testuser@01HW595640:~/workspace161/.repo$ rm -rf repo
testuser@01HW595640:~/workspace161/.repo$ ls
manifests  manifests.git  manifest.xml

4. Executing repo init with invalid --repo-url

testuser@01HW595640:~/workspace161/.repo$ cd ..

testuser@01HW595640:~/workspace161$ repo init --repo-url 
https://android.googlesource.com/platform/repo --manifest-url  
https://android.googlesource.com/platform/manifest
Get https://android.googlesource.com/platform/repo
fatal: remote error: Git repository not found
Traceback (most recent call last):
  File "/home/testuser/bin/repo", line 775, in <module>
    main(sys.argv[1:])
  File "/home/testuser/bin/repo", line 748, in main
    os.rmdir(os.path.join(root, name))
OSError: [Errno 20] Not a directory: '.repo/manifests/.git/objects'
testuser@01HW595640:~/workspace161$ 

5. Checking if .repo gets deleted or not
testuser@01HW595640:~/workspace161$ la
.repo
testuser@01HW595640:~/workspace161$ 
testuser@01HW595640:~/workspace161$ cd .repo/
testuser@01HW595640:~/workspace161/.repo$ ls
manifests  manifests.git  manifest.xml  repo

testuser@01HW595640:~/workspace161/.repo$ cd repo
testuser@01HW595640:~/workspace161/.repo/repo$ ls
testuser@01HW595640:~/workspace161/.repo/repo$

So I somehow couldn't reproduce the deletion of .repo. Am I missing something 
here? 

And also could you explain what you meant by "2. It deletes .repo in the 
current directory when it should be deleting .repo from the root of the 
checkout."?

Since the .repo is created relative to root of the checkout workspace, so from 
where else would the repo init command be executed?

Kindly suggest.

Regards,
Sushan.

Original comment by ray.sus...@gmail.com on 7 Aug 2014 at 2:27

[GoogleCodeExporter]

Hi Sushan,

You are using a stable version of repo and not the latest version. You can see 
from the traceback there though that it was trying to delete .repo and failed. 
This was the first bug I reported above. See the following traceback:

Traceback (most recent call last):
  File "/home/testuser/bin/repo", line 775, in <module>
    main(sys.argv[1:])
  File "/home/testuser/bin/repo", line 748, in main
    os.rmdir(os.path.join(root, name))
OSError: [Errno 20] Not a directory: '.repo/manifests/.git/objects'

If you retrieve the latest version of repo then you'll get a different result. 
You can get the latest version of repo by running "git clone 
https://gerrit.googlesource.com/git-repo -b stable" and grabbing the repo 
launcher from there.

Test case 1:

$ mkdir -p ~/test-directory
$ cd ~/test-directory
$ repo init -u https://android.googlesource.com/platform/manifest
$ repo init --repo-url https://fake-url
fatal: Cannot get https://fake-url/clone.bundle
fatal: error [Errno -2] Name or service not known

Now, notice that .repo is missing entirely!

Test case 2:

$ mkdir -p ~/test-directory
$ cd ~/test-directory
$ repo init -u https://android.googlesource.com/platform/manifest
$ mkdir subdir
$ cd subdir
$ repo init --repo-url https://fake-url
fatal: Cannot get https://fake-url/clone.bundle
fatal: error [Errno -2] Name or service not known

Now notice that the ".repo" directory was not deleted. This is because repo was 
attempting to delete ~/test-directory/subdir/.repo instead of 
~/test-directory/.repo

Original comment by davidjames@chromium.org on 7 Aug 2014 at 5:39

[GoogleCodeExporter]

Thanks David.

I was able to reproduce the problem locally with the aforesaid steps. And have 
started working on fixing the problem.

Regards, Sushan.

Original comment by ray.sus...@gmail.com on 8 Aug 2014 at 8:19

[GoogleCodeExporter]

Hi David,

The scenario wherein we want deletion of .repo/repo instead of the entire 
.repo, can be easily fixed. And Ive fixed that in my local environment.

However regarding the second scenario, should we actually try to execute the 
repo init from a subdirectory instead of root of the repo installation, 
specially when we are trying to update an already existing repository?

Currently, the repo implementation is such that it always first tries to create 
a new .repo from the current working directory when we try to init from any 
subdirectory. And in case of CloneFailure, it actually deletes the newly 
created .repo searching from the current directory. And that is expected. But 
then we check if the .repo is deleted from the root of repo installation, which 
doesn't happen as expected.

Incase you want to delete the .repo/repo from the root of repo installation 
when executing the repo init from a subdirectory, then we need to do two things:

1. Delete the entire .repo from the current working directory, as we already 
have the repository content at the root of repo installation.

2. Then delete the .repo/repo from the root of repo installation.

Kindly comment.

Regards,
Sushan.

Original comment by ray.sus...@gmail.com on 14 Aug 2014 at 11:40

[GoogleCodeExporter]

https://gerrit-review.googlesource.com/#/c/59630/

Original comment by mani.cha...@tcs.com on 28 Aug 2014 at 7:13

[GoogleCodeExporter]

Sorry for the slow response. You are right, we just need to tweak the existing 
logic to delete .repo/repo in the current directory

Original comment by davidjames@google.com on 3 Sep 2014 at 12:40