--shallow-since on git clone is causing "the remote end hung up unexpectedly"

[beamery-tomht]

I'm running radius-tracker timelines ./path/to/config.js with a config like such

module.exports = [
  {
    displayName: 'app-something',
    repoUrl: 'git@github.com:MyOrg/app-something.git',
    maxWeeks: 1,
    subprojectPath: '/',
    isTargetModuleOrPath: /^@my-org\/components/
  },
  {
    displayName: 'app-other',
    repoUrl: 'git@github.com:MyOrg/app-other.git',
    maxWeeks: 1,
    subprojectPath: '/',
    isTargetModuleOrPath: /^@my-org\/components/
  },
  // etc...
]

For the majority of repos this works as expected, however there are some cases where the command spits out

fatal: the remote end hung up unexpectedly

I've narrowed this down to the --shallow-since=date that's automated from the maxWeeks or since options

There's a SO thread here about various solutions that have worked for people

Such as:

Updating .gitconfig to increase the buffer size

[url "git@github.com:"]
    insteadOf = https://github.com/

[ssh]
    postBuffer = 2048M
    maxRequestBuffer = 1024M
[http]
    postBuffer = 2048M
    lowSpeedLimit = 0
    lowSpeedTime = 999999
[pack]
    windowMemory = 256M
    packSizeLimit = 256M
[core]
    compression = 0 # also try other numbers up to 9

Updating /etc/resolve.confg with:

options single-request

Making sure all of the repoUrl options are supplied as SSH URLs so that the key is used, etc...

Installing git-lfs and hoping it changes the git command's behaviour 🤷‍♂️

None of these solutions worked. There are still 1 or 2 repos for me out of a list of 20 that fail with this error

If I run the command locally (clearing the cache repo each time), it fails no matter the value for --shallow-since

git --git-dir=/path/to/cache/repo/.git \
  --work-tree=/path/to/cache/repo \
  clone \
  --no-tags \
  --single-branch \
  --no-checkout \
  --shallow-since=2023-05-01 \
  git@github.com:MyOrg/other-app.git \
  /path/to/cache/repo/.git

Running the same command but removing --shallow-since and I don't get any issues

Also, setting --depth=3200 seems to work fine as well - and can run git fetch --unshallow after

But obviously can't specify this argument from the radius-tracker command and it doesn't solve the issue of taking commits from a specified date consistently across multiple repos

EDIT: (okay not a real edit as I've not submitted this issue yet...)

I've solved the issue by INCREASING the --shallow-since option with maxWeeks: 52... Seems like the script aborts and breaks if there are no commits for a given repo within the time window specified and throws this error

[smoogly]

Generally, I'm trying to fetch as little history as possible while still fetching enough to process the required timeline. This does indeed sometime cause issues, for example "error processing shallow info": https://github.com/rangle/radius-tracker/blob/3feab709f90a90f0337f1fe1d8eb4c5afa34f9b1/src/lib/cli/timelines/git.ts#L42-L51

Could you send me the full output of the failing git fetch with small --shallow-since? Full output of the example you shared would work nicely:

git --git-dir=/path/to/cache/repo/.git \
  --work-tree=/path/to/cache/repo \
  clone \
  --no-tags \
  --single-branch \
  --no-checkout \
  --shallow-since=2023-05-01 \
  git@github.com:MyOrg/other-app.git \
  /path/to/cache/repo/.git

[beamery-tomht]

Apologies for slow reply - had to focus on other tasks

Here's the full output but not so much extra info

git --git-dir=/path/to/cache/repo/.git \
        --work-tree=/path/to/cache/repo/ \
        clone \
        --no-tags \
        --single-branch \
        --no-checkout \
        --shallow-since=2023-06-03 \
        git@github.com:MyOrg/other-app.git \
        /path/to/cache/repo/.git

Cloning into /path/to/cache/repo/.git'...
fatal: the remote end hung up unexpectedly

I've also tried with --verbose and --progress but no other information shared