search

martinvonz / jj

A Git-compatible VCS that is both simple and powerful
https://martinvonz.github.io/jj/
Apache License 2.0
8.39k stars 289 forks source link

jj git fetch downloads branches with names uppercased #4329

Closed zenshixd closed 1 month ago

zenshixd commented 1 month ago

Description

jj git fetch (or jj git clone) downloads branch names uppercased, example: I have branch called feature/TASKFLOW-24635-optimize-deletemodules-network-call on remote Bitbucket Server, but when downloaded through jj git fetch I am getting:

ownelek@MacBook-Pro wkapp-taskflow % jj git fetch --debug           
2024-08-23T17:24:36.933716Z  INFO jj_cli::cli_util: debug logging enabled
2024-08-23T17:24:37.199970Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_branch: false, abandon_unreachable_commits: true }}: jj_lib::git: remote.download
2024-08-23T17:24:37.523766Z  INFO run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_branch: false, abandon_unreachable_commits: true }}: jj_lib::git: trying ssh_key_from_agent username="git"
2024-08-23T17:24:38.479797Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_branch: false, abandon_unreachable_commits: true }}: jj_lib::git: remote.prune
2024-08-23T17:24:40.446461Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_branch: false, abandon_unreachable_commits: true }}: jj_lib::git: remote.update_tips
2024-08-23T17:24:40.942664Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_branch: false, abandon_unreachable_commits: true }}: jj_lib::git: default_branch="master"
2024-08-23T17:24:40.942679Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_branch: false, abandon_unreachable_commits: true }}: jj_lib::git: remote.disconnect
2024-08-23T17:24:40.990414Z DEBUG run_command:cmd_git_fetch{args=GitFetchArgs { branch: [Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })], remotes: [], all_remotes: false }}:fetch{remote_name="origin" branch_names=[Glob(Pattern { original: "*", tokens: [AnySequence], is_recursive: false })] git_settings=GitSettings { auto_local_branch: false, abandon_unreachable_commits: true }}: jj_lib::git: import_refs
branch: Feature/TASKFLOW-24635-optimize-deletemodules-network-call@origin [new] untracked

Notice the prefix: Feature/ instead of feature/. This is problematic, because trying to push to this branch fails:

ownelek@MacBook-Pro wkapp-taskflow % jj git push --debug
2024-08-23T17:58:04.223705Z  INFO jj_cli::cli_util: debug logging enabled
Branch changes to push to origin:
  Move forward branch Feature/TASKFLOW-24635-optimize-deletemodules-network-call from 2efbe63a593f to 6848904af55f
2024-08-23T17:58:04.801444Z  INFO run_command: jj_lib::git: trying ssh_key_from_agent username="git"
2024-08-23T17:58:05.496380Z  INFO run_command: jj_lib::git: Cannot push refs/heads/Feature/TASKFLOW-24635-optimize-deletemodules-network-call to Some(CommitId("6848904af55fc312b2033e2fdbd96f0ef39fa37a")); it is at unexpectedly at None on the server as opposed to the expected Some(CommitId("2efbe63a593f2dc25009b83af5d87214db08b63a"))
Error: Refusing to push a branch that unexpectedly moved on the remote. Affected refs: refs/heads/Feature/TASKFLOW-24635-optimize-deletemodules-network-call
Hint: Try fetching from the remote, then make the branch point to where you want it to be, and push again.

Following the hint doesnt work, subsequent jj git fetch returns "Nothing changed.".

Using Git binary doesnt have this issue When using pure Git repo, branch name is correct:

ownelek@MacBook-Pro Projects % git clone [...] testrepo
Klonowanie do „testrepo”...
remote: Enumerating objects: 1629313, done.
remote: Counting objects: 100% (6104/6104), done.
remote: Compressing objects: 100% (4063/4063), done.
remote: Total 1629313 (delta 4072), reused 2697 (delta 1978), pack-reused 1623209
Pobieranie obiektów: 100% (1629313/1629313), 300.09 MiB | 12.86 MiB/s, gotowe.
Rozwiązywanie delt: 100% (1232534/1232534), gotowe.
Aktualizowanie plików: 100% (17978/17978), gotowe.
ownelek@MacBook-Pro Projects % cd testrepo
ownelek@MacBook-Pro testrepo % git branch -a | grep 24635
  remotes/origin/feature/TASKFLOW-24635-optimize-deletemodules-network-call

Steps to Reproduce the Problem

Unfortunately the repo is on internal Bitbucket Server, so I cannot share it. I tried reproducing this issue using Bitbucket Cloud but no success so far.

  1. Clone repo using jj git clone
  2. Search for branch using jj branch list -a 'glob:...'

Expected Behavior

Branch name is exactly the same locally as on remote.

Actual Behavior

Branch prefix is uppercased which prevents pushing changes to remote.

Specifications

PhilipMetzger commented 1 month ago

On what jj version are you on? You missed this while filling out the bug template. Thanks.

zenshixd commented 1 month ago

Sorry, I am using jj@0.20.0.

After some investigation though I found the core of the issue - its the core.ignoreCase=false on Git. If I disable this setting (core.ignoreCase=true) then the same issue happens on Git.

For some reason prefixes BugFix/ and Feature/ are uppercased in our remote repo but Bitbucket reports them as lowercased, for some reason.

The workaround I found in jujutsu is to manually edit refs .jj/repo/store/git/refs/origin/ from BugFix/ to bugfix (and from Feature/ to feature/). After this change I was able to push again and all new branches also worked correctly.

I still dont know how to replicate it tho :(

martinvonz commented 1 month ago

After some investigation though I found the core of the issue - its the core.ignoreCase=false on Git. If I disable this setting (core.ignoreCase=true) then the same issue happens on Git.

Good find! I wonder where the Feature (with uppercase F) comes from. Are you sure it's really feature on the remote? Oh, maybe you have some other branches called Feature/..., would have caused the directory to be called that. If that's the case, then I would expect pushing your branch to fail with Git too, but maybe Git has some hack to convert it back to the lowercase feature.

I suspect you can fix the problem by calling git pack-refs, which should make sure the branches are stored in a proper data format instead of as files in the file system. Once you've done that, you should be able to have a feature/foo branch and a Feature/bar branch (and even a Feature/foo branch). You may need to re-fetch to get the canonical names into your repo.

zenshixd commented 1 month ago

Are you sure it's really feature on the remote?

I mean ... If I dont lowercase the on my system then push fails so yea :D It is also how Bitbucket reports it on web UI: obraz

If that's the case, then I would expect pushing your branch to fail with Git too

Yea thats also what happens with Git - but with Git its not too bad since I can push to a separate Feature/ on remote - with jj I am completely blocked.

Also, fiddled with it a bit more, since the issue on Git doesnt happen as often as on jj, but most reliable way I find was:

mkdir testrepo
cd testrepo
git init
git remote add origin [...]
git fetch

With this method even with core.ignorecase=true - I was also getting Feature/ instead of feature/.

martinvonz commented 1 month ago

I mean ... If I dont lowercase the on my system then push fails so yea :D It is also how Bitbucket reports it on web UI:

Yes, sorry, I was thinking it through as I typing the message. I could have deleted that sentence after typing the next one since it started to make sense to me then :)

Yea thats also what happens with Git - but with Git its not too bad since I can push to a separate Feature/ on remote - with jj I am completely blocked.

I don't think we should copy Git's hacks, so I think we can consider the remaining part of this issue to be #3235. Feel free to reopen if you disagree.

zenshixd commented 1 month ago

Some more thoughts + I managed to reproduce the issue here on Github!!! https://github.com/zenshixd/uppercase-bug

There was still 1 thing that puzzled me so I couldnt stop thinking about it but: the reason web UI reported feature/ but I was downloading Feature/ was because ... Git/JJ are downloading branches in alphabetical order :) Meaning: on macOS/Windows, if you download Feature/ branch first (which you will, because F is higher than f in ASCII!) then every subsequent branch with prefix feature or feaTure or any other casing where first letter is f, will land in the same directory Feature!!

You can see it well in repo above, if you do

git init
git remote add origin git@github.com:zenshixd/uppercase-bug.git
git fetch
git branch -a

You will see that all branches have prefix AAAA/ even though on remote they are all different! And if you remove AAAA/ on remote, clone it again, then ...

ownelek@MacBook-Pro testrepo % git branch -a
  remotes/origin/AAAa/test2
  remotes/origin/AAAa/test3
  remotes/origin/AAAa/test4
  remotes/origin/AAAa/test5
  remotes/origin/main

:)))))))))

This is just some food for thought, I have no idea what correct solution is here. On the other hand, I feel allowing specifying which branch to push to is just workaround, but I guess it would at least unblock people who encounter such issue.