search

chrisjbillington / hg-export-tool

A tool to convert mercurial repositories to git ones locally, working around some deficiencies in github's importer and in `hg-fast-export`
GNU General Public License v2.0
21 stars 9 forks source link

Branch mapping #6

Closed kakaiba-talaga closed 4 years ago

kakaiba-talaga commented 4 years ago

Ran this in Windows 10 Pro 1909, under Git Bash 2.25.1.windows.1, and Python 3.7.5 using the latest version (Commit: 136ee344c20fd2c77f4aa0c90c4ac1a6c11c6a6a [136ee34], Date: Wednesday, 1 April 2020 14:21:27).

$ python /d/Repositories/Others/hg-export-tool/exporter.py --bash="/c/Program Files/Development/Git/git-bash.exe" map-f ile.json -A authors.map --hg-hash

I ran this against a big mercurial repo with and without the --hg-hash flag. It seems doing its thing then these errors occurred:

Note: The directory names have been sanitized.

Initialized empty Git repository in C:/Users/user_here/AppData/Local/Temp/majic-web-hg-git-70209d4df8a87fa41f41d034da898e59/.git/
Traceback (most recent call last):
  File "D:/Repositories/Others/hg-export-tool/exporter.py", line 254, in <module>
    main()
  File "D:/Repositories/Others/hg-export-tool/exporter.py", line 250, in main
    BASH
  File "D:/Repositories/Others/hg-export-tool/exporter.py", line 172, in process_repo
    hg_repo_copy = copy_hg_repo(hg_repo)
  File "D:/Repositories/Others/hg-export-tool/exporter.py", line 48, in copy_hg_repo
    shutil.copytree(hg_repo, hg_repo_copy)
  File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\shutil.py", line 368, in copytree
    raise Error(errors)
shutil.Error:

[

(
'D:\\Repositories\\company_here\\Mercurial\\app_here\\Application\\client_here\\Websites\\App_PurchaseOrder\\packages\\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.5.1\\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.5.1.nupkg',
'C:\\Users\\user_here\\AppData\\Local\\Temp\\app_here-09f1b72f6c2dae34e53530749d30f608\\Application\\client_here\\Websites\\App_PurchaseOrder\\packages\\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.5.1\\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.5.1.nupkg',
"[Errno 2] No such file or directory: 'C:\\\\Users\\\\user_here\\\\AppData\\\\Local\\\\Temp\\\\app_here-09f1b72f6c2dae34e53530749d30f608\\\\Application\\\\client_here\\\\Websites\\\\App_PurchaseOrder\\\\packages\\\\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.5.1\\\\Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.2.5.1.nupkg'"
),

(
'D:\\Repositories\\company_here\\Mercurial\\app_here\\Web MAJIC Job Pricing Monitor\\Installer\\PricingMonitorInstaller\\Express\\DVD-5\\DiskImages\\DISK1\\ISSetupPrerequisites\\{72D7E3D1-C9DF-4FA6-9F9B-4E5117AB2919}\\dotnetfx45_full_x86_x64.exe',
'C:\\Users\\user_here\\AppData\\Local\\Temp\\app_here-09f1b72f6c2dae34e53530749d30f608\\app_here Job Pricing Monitor\\Installer\\PricingMonitorInstaller\\Express\\DVD-5\\DiskImages\\DISK1\\ISSetupPrerequisites\\{72D7E3D1-C9DF-4FA6-9F9B-4E5117AB2919}\\dotnetfx45_full_x86_x64.exe',
"[Errno 2] No such file or directory: 'C:\\\\Users\\\\user_here\\\\AppData\\\\Local\\\\Temp\\\\app_here-09f1b72f6c2dae34e53530749d30f608\\\\app_here Job Pricing Monitor\\\\Installer\\\\PricingMonitorInstaller\\\\Express\\\\DVD-5\\\\DiskImages\\\\DISK1\\\\ISSetupPrerequisites\\\\{72D7E3D1-C9DF-4FA6-9F9B-4E5117AB2919}\\\\dotnetfx45_full_x86_x64.exe'"
),

(
'D:\\Repositories\\company_here\\Mercurial\\app_here\\app_here Job Pricing Monitor\\Installer\\PricingMonitorInstaller\\Express\\SingleImage\\DiskImages\\DISK1\\ISSetupPrerequisites\\{72D7E3D1-C9DF-4FA6-9F9B-4E5117AB2919}\\dotnetfx45_full_x86_x64.exe',
'C:\\Users\\user_here\\AppData\\Local\\Temp\\app_here-09f1b72f6c2dae34e53530749d30f608\\app_here Job Pricing Monitor\\Installer\\PricingMonitorInstaller\\Express\\SingleImage\\DiskImages\\DISK1\\ISSetupPrerequisites\\{72D7E3D1-C9DF-4FA6-9F9B-4E5117AB2919}\\dotnetfx45_full_x86_x64.exe',
"[Errno 2] No such file or directory: 'C:\\\\Users\\\\user_here\\\\AppData\\\\Local\\\\Temp\\\\app_here-09f1b72f6c2dae34e53530749d30f608\\\\app_here Job Pricing Monitor\\\\Installer\\\\PricingMonitorInstaller\\\\Express\\\\SingleImage\\\\DiskImages\\\\DISK1\\\\ISSetupPrerequisites\\\\{72D7E3D1-C9DF-4FA6-9F9B-4E5117AB2919}\\\\dotnetfx45_full_x86_x64.exe'"
)

]

The files that it says that are missing is not missing.

kakaiba-talaga commented 4 years ago

After some thought, I decided that may be running it in Windows is the issue. So, I tried it in Ubuntu via Window Subsystem for Linux (WSL). Now, I can see statuses that the import is actually being done until it isn't...

$> python3 ./hg-export-tool/exporter.py map-file.json -A authors.map --hg-hash

Initialized empty Git repository in /tmp/majic-web-hg-git-434315ec61588cbfb59cab1f10886b13/.git/
2560 files updated, 0 files merged, 875 files removed, 0 files unresolved
marked working directory as branch AsyncSchedulingModule-1
saved backup bundle to /tmp/app-here-f09db68cbe0ffffe427263dbb4d97d2b/.hg/strip-backup/701bf349be4e-4fb7c600-amend.hg
Laoded 35 authors

...

Issued 16240 commands
error: cannot lock ref 'refs/heads/Deliveries/Scheduling_(Chris)': 'refs/heads/Deliveries' exists; cannot create 'refs/heads/Deliveries/Scheduling_(Chris)'
/usr/lib/git-core/git-fast-import statistics:
---------------------------------------------------------------------
Alloc'd objects:     140000
Total objects:       138473 (    208619 duplicates                  )
      blobs  :        47930 (    187077 duplicates      27258 deltas of      47846 attempts)
      trees  :        74361 (     21542 duplicates      64896 deltas of      73363 attempts)
      commits:        16182 (         0 duplicates          0 deltas of          0 attempts)
      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)
Total branches:         225 (       510 loads     )
      marks:        1048576 (      8091 unique    )
      atoms:          16504
Memory total:          9735 KiB
       pools:          3173 KiB
     objects:          6562 KiB
---------------------------------------------------------------------
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 35184372088832
pack_report: pack_used_ctr            =      25131
pack_report: pack_mmap_calls          =       6736
pack_report: pack_open_windows        =          2 /          2
pack_report: pack_mapped              =  646542674 /  646542674
---------------------------------------------------------------------

Traceback (most recent call last):
  File "./hg-export-tool/exporter.py", line 255, in <module>
    main()
  File "./hg-export-tool/exporter.py", line 251, in main
    BASH
  File "./hg-export-tool/exporter.py", line 176, in process_repo
    convert(hg_repo_copy, temp_git_repo, fast_export_args, bash)
  File "./hg-export-tool/exporter.py", line 148, in convert
    cwd=git_repo,
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/bash', 'hg-fast-export.sh', '-r', '/tmp/app-here-f09db68cbe0ffffe427263dbb4d97d2b', '-A', '/home/kakaiba/scratch/authors.map', '--hg-hash']' returned non-zero exit status 1.

The error:

error: cannot lock ref 'refs/heads/Deliveries/Scheduling(Chris)': 'refs/heads/Deliveries' exists; cannot create 'refs/heads/Deliveries/Scheduling(Chris)'

is about 2 closed branches.

  1. Deliveries
  2. Deliveries/Scheduling_(Chris)

@chrisjbillington Is there an option that I can map the Deliveries/Scheduling_(Chris) branch to some other name to prevent this error?

I tried using the -B branches.map but it didn't work.

branches.map

"Deliveries/Scheduling_(Chris)"="xDeliveries/Scheduling_(Chris)"
chrisjbillington commented 4 years ago

I suspect the error you're getting on Windows is due to the filepaths being too long - longer than 256 characters. I'm not sure what do do instead of shutil.copytree() that can deal with that, but I'll look into it. My tool does create temporary directories with 32 characters of extra random hex in their names which exacerbates the problem, though.

Branches with slashes in them with the bit before the slash being an existing branch name are invald in git, and yes, you can use a branch mapping file to rename the branches. See the documentation of hg-fast-export for this. This tool will just pass extra command line arguments on to hg-fast-export, so it should work fine!

kakaiba-talaga commented 4 years ago

Manually specifying the mappings for all the incompatible branches and tags is the way to go. Just don't forget to include the -n flag. Otherwise, the "auto sanitize" feature will kick in and will override/ignore any mappings for branches and tags.

python3 ./exporter.py map-file.json -A authors.map -B branches.map -T tags.map -n

Thank you, @chrisjbillington, for this wrapper for hg-fast-export.