search

freebsd / git_conv

Scripts used for the svn to git conversion of the FreeBSD repositories
12 stars 14 forks source link

Vendor tag branches are being recorded falsely #9

Closed uqs closed 4 years ago

uqs commented 4 years ago

No idea what is going on here, but note when 20170608 was imported vs. where the tag is coming from (same for 1.14.2). The contents of the tag are all fine, but the branchpoint is messed up. This affects numerous vendor tags

* 75b6c55cb02a - Import mandoc 1.14.2 
|       svn path=/vendor/mdocml/dist/; revision=321807
|  
| * ddffdf6064c6 - (vendor/mandoc/1.14.2) Tag import of mandoc 1.14.2 
|/        svn path=/vendor/mdocml/1.14.2/; revision=321808
|    
* 7ad21139cd51 - Import mandoc 2017-06-08 
|       svn path=/vendor/mdocml/dist/; revision=319715
|  
* 6cae2c9f93ec - Import mandoc 1.14.1 
|       svn path=/vendor/mdocml/dist/; revision=314095
|  
* 696a89ee8dc1 - Import mandoc 1.4.1rc2 
|       svn path=/vendor/mdocml/dist/; revision=313956
|  
| * b2877366839f - (vendor/mandoc/20170608) tag import of mandoc snapshot 20170608 
|/        svn path=/vendor/mdocml/20170608/; revision=319716
|    
| * 4483a76c09db - (vendor/mandoc/1.4.1) Tag import of mandoc 1.4.1 
|/        svn path=/vendor/mdocml/1.4.1/; revision=314096
|    
| * 1d298b9caca1 - (vendor/mandoc/1.4.1rc2) Tag import of mandoc 1.4.1rc2 
|/        svn path=/vendor/mdocml/1.4.1rc2/; revision=313957
|    
* ef7476572c9a - Import mandoc cvs snapshot 20170121 (pre 1.14) 
|       svn path=/vendor/mdocml/dist/; revision=312591
|

The converter logs these down like so:

progress SVN r321807 branch vendor/mandoc/dist = :321407                                                                                                               
progress SVN r321808 branch vendor/mandoc/1.14.2 = :319394 # from branch vendor/mandoc/dist at r321806 => r319715
progress SVN r321808 branch vendor/mandoc/1.14.2 = :321408

where it messes up the "from" of the branch. It's also in the regular output:

Exporting revision 321807 ................................................................................... 83 modifications from SVN /vendor/mdocml/dist/ to freebsd-base.git/vendor/mandoc/dist done
Exporting revision 321808     /vendor/mdocml/1.14.2 was copied from /vendor/mdocml/dist rev 321806
.freebsd-base.git : branch vendor/mandoc/1.14.2 is branching from vendor/mandoc/dist
Creating branch: "vendor/mandoc/1.14.2" from "vendor/mandoc/dist" ( 321806 "from branch vendor/mandoc/dist at r321806 => r319715" )

This is coming from FastImportRepository::markFrom in repository.cpp

uqs commented 4 years ago

I fixed some of these (like vendor/dtc) but there are others still. It's when they create the SVN tag in a strange way.

Here's the output of commit hash, tree hash for zstd and it uses an older rev to branch from (this can be seen in the svn log -v output on the top-level dir, clearly).

* 40dbdde6971c 9b41925d7159  - (tag: vendor/zstd/1.4.5) Tag Zstd 1.4.5
|       svn path=/vendor/zstd/1.4.5/; revision=361424
| svn path=/vendor/zstd/1.4.5/; revision=361424; tag=vendor/zstd/1.4.5
|
| * bc64b5ce191d 9b41925d7159  - (vendor/zstd/dist) Import Zstd 1.4.5
|/        svn path=/vendor/zstd/dist/; revision=361423
|
* ea68403922c3 9870b0c69585  - Import Zstd 1.4.4
|       svn path=/vendor/zstd/dist/; revision=354371
|
| * 16712072cdcf 9870b0c69585  - (tag: vendor/zstd/1.4.4) Tag Zstd 1.4.4
|/        svn path=/vendor/zstd/1.4.4/; revision=354372
|   svn path=/vendor/zstd/1.4.4/; revision=354372; tag=vendor/zstd/1.4.4
|
* 90f4bdbe917e c579df59365b  - (tag: vendor/zstd/1.4.2) import zstd 1.4.2
|       svn path=/vendor/zstd/dist/; revision=350754
| svn path=/vendor/zstd/1.4.2/; revision=350755; tag=vendor/zstd/1.4.2
|
* fa94c7381af4 1f6152d1ad65  - (tag: vendor/zstd/1.4.1) import zstd 1.4.1
|       svn path=/vendor/zstd/dist/; revision=350752
| svn path=/vendor/zstd/1.4.1/; revision=350753; tag=vendor/zstd/1.4.1
|
* 3f774a5e8643 2ba6d7db9473  - import zstd 1.4.0
|       svn path=/vendor/zstd/dist/; revision=346361
|
| * 58055709c1fb 2ba6d7db9473  - (tag: vendor/zstd/1.4.0) tag zstd 1.4.0
|/        svn path=/vendor/zstd/1.4.0/; revision=346362
|   svn path=/vendor/zstd/1.4.0/; revision=346362; tag=vendor/zstd/1.4.0
|
uqs commented 4 years ago

This could be fixed post-conversion:

for all tags foo/tag not on mainline foo/dist: if there is one and only one commit on mainline with the same tree-hash -> rewrite tag to point to that commit instead

uqs commented 4 years ago

I think I understand what is happening here now.

A good svn cp will result in just the prefix being copied. As there are no further modifications, we end up with an annotated tag for that copy.

But with the messed up svn copies, the exporter has an actual diff to work with, creating a new commit and then the tag points to that commit (with the identical tree hash, urgh)

uqs commented 4 years ago

At least the case of tags predating the vendor imports is now fixed.

uqs commented 4 years ago

Good:

reset refs/tags/vendor/mandoc/1.13.2
from :42276466

progress SVN r276217 branch refs/tags/vendor/mandoc/1.13.2 = :42276466 # from branch vendor/mandoc/dist

Bad:

reset refs/tags/vendor/mandoc/1.13.3
from :42280269

progress SVN r280024 branch refs/tags/vendor/mandoc/1.13.3 = :42280269 # from branch vendor/mandoc/dist

commit refs/tags/vendor/mandoc/1.13.3
mark :42280270
committer Baptiste Daroussin <bapt@FreeBSD.org> 1426418973 +0000
data 28
Tag import of mdocml 1.13.3

D Makefile.local
M 100644 :18446744073709551614 INSTALL
M 100644 :18446744073709551613 Makefile
M 100644 :18446744073709551612 NEWS
...

I wonder if we could trick svn2git into just never re-adjusting the tag to the 2nd commit, because that is specially handled (but we'd miss all the tags that have been advanced after their initial creation, good thing I have a diff script for that).

Exporting revision 280024     /vendor/mdocml/1.13.3 was copied from /vendor/mdocml/dist rev 280021
rev 280024 /vendor/mdocml/1.13.3/ matched rule: "/data/git_conv/freebsd-base.rules:1342 /(vendor)/(mandoc|mdocml)/((?!dist)[^/]+)/"    exporting.
.freebsd-base.git : branch refs/tags/vendor/mandoc/1.13.3 is branching from vendor/mandoc/dist
Creating branch: "refs/tags/vendor/mandoc/1.13.3" from "vendor/mandoc/dist" ( 280021 "from branch vendor/mandoc/dist at r280021 => r279525" )
rev 280024 /vendor/mdocml/1.13.3/INSTALL matched rule: "/data/git_conv/freebsd-base.rules:1342 /(vendor)/(mandoc|mdocml)/((?!dist)[^/]+)/"    exporting.

Creating annotated tag vendor/mandoc/1.13.3 (refs/tags/vendor/mandoc/1.13.3) for freebsd-base.git
.copy from branch "vendor/mandoc/dist" to branch "refs/tags/vendor/mandoc/1.13.3" @rev 280023
"WARN: repository freebsd-base.git branch refs/tags/vendor/mandoc/1.13.3 has some files copied from vendor/mandoc/dist@280023"
WARN: found branchpoint from lower mark, about to recreate branch from different revision
Creating branch: "refs/tags/vendor/mandoc/1.13.3" from "vendor/mandoc/dist" ( 280023 "from branch vendor/mandoc/dist" )
add/change file ( /vendor/mdocml/1.13.3/INSTALL -> "refs/tags/vendor/mandoc/1.13.3" "INSTALL" )
rev 280024 /vendor/mdocml/1.13.3/Makefile matched rule: "/data/git_conv/freebsd-base.rules:1342 /(vendor)/(mandoc|mdocml)/((?!dist)[^/]+)/"    exporting.

Re-creating annotated tag vendor/mandoc/1.13.3 for freebsd-base.git

Note that last line which -- in this specific case -- messes up the tags position.

uqs commented 4 years ago 
% grep "Re-creating annotated tag" log-base|wc -l
     996

Sigh ....

uqs commented 4 years ago

This is fixed by a post-processing script. Sadly, it means the converter can no longer run in incremental mode, but then again, with git version 2.27 this is already no longer possible.