Crash with Brokenpipe

[nomennescio]

Although I was able to convert a repo, after some git replaces, I get errors trying to redo it. First did an analysis to check if I could learn more:

$ git filter-repo --analyze --debug --force
Processed 134318 blob sizesly: ".git/filter-repo/analysis"
Processed 38635 commitswarning: exhaustive rename detection was skipped due to too many files.
warning: you may want to set your diff.renameLimit variable to at least 2928 and retry the command.
Processed 38834 commits
Writing reports to .git/filter-repo/analysis...done.

$ git config diff.renameLimit 10000

$ git filter-repo --analyze --debug --force
Processed 134318 blob sizesly: ".git/filter-repo/analysis"
Processed 38834 commits
Writing reports to .git/filter-repo/analysis...done.

Then tried running again

$ git filter-repo --replace-refs delete-no-add --force --debug
[DEBUG] Passed arguments:
Namespace(analyze=False, report_dir=None, inclusive=False, path_changes=[], use_base_name=False, subdirectory_filter=None, to_subdirectory_filter=None, replace_text=None, max_blob_size=0, strip_blobs_with_ids=set(), tag_rename=None, replace_message=None, preserve_commit_hashes=False, preserve_commit_encoding=False, mailmap=None, replace_refs='delete-no-add', prune_empty='auto', prune_degenerate='auto', no_ff=False, filename_callback=None, message_callback=None, name_callback=None, email_callback=None, refname_callback=None, blob_callback=None, commit_callback=None, tag_callback=None, reset_callback=None, source=None, target=None, help=False, version=False, force=True, partial=False, refs=['--all'], dry_run=False, debug=True, state_branch=None, stdin=False, quiet=False, repack=True)
[DEBUG] Running: git fast-export --show-original-ids --signed-tags=strip --tag-of-filtered-object=rewrite --fake-missing-tagger --reference-excluded-parents --no-data --use-done-feature --mark-tags --reencode=yes --all
  (saving a copy of the output at .git/filter-repo/fast-export.original)
[DEBUG] Running: git -c core.ignorecase=false fast-import --force --quiet --date-format=raw-permissive
  (using the following file as input: .git/filter-repo/fast-export.filtered)
Parsed 12374 commitsTraceback (most recent call last):
  File "/usr/bin/git-filter-repo", line 4005, in <module>
    main()
  File "/usr/bin/git-filter-repo", line 4002, in main
    filter.run()
  File "/usr/bin/git-filter-repo", line 3937, in run
    self._parser.run(self._input, self._output)
  File "/usr/bin/git-filter-repo", line 1409, in run
    self._parse_commit()
  File "/usr/bin/git-filter-repo", line 1260, in _parse_commit
    self._commit_callback(commit, aux_info)
  File "/usr/bin/git-filter-repo", line 3486, in _tweak_commit
    self._record_remapping(commit, orig_parents)
  File "/usr/bin/git-filter-repo", line 3261, in _record_remapping
    self._output.flush()
  File "/usr/bin/git-filter-repo", line 2721, in flush
    self.file1.flush()
BrokenPipeError: [Errno 32] Broken pipe

can't get anything more out of it, repeating leads to the same error. How to solve this?

[newren]

Any chance there's a .git/fast_import_crash_<number> file after the bad run (where <number> is some random positive integer, like 8434)? If yes, can you share its contents?

[nomennescio]

I already threw away the directory, but I will try to reproduce, however, that will take a while. I noticed however that switching to a different platform, doing a fast export|fast import, then switching back and run filter-repo did work, so that's my current work-around. (I could reproduce the issue)

Might it be there is an issue with atypical replacement objects?

[newren]

I think it'd be less likely to be an atypical replacement object, and more likely to be some non-canonical object (a tree with multiple of the same-named path? extended headers of some kind? who knows?) that doesn't parse nicely in filter-repo (though which filter-repo parses just well enough to not die itself but pass along something to fast-import which fast-import doesn't understand), and yet which when simply trying to canonicalize via the fast-export | fast-import round-trip gets stripped out or cleaned up.

But, without more information from the repo, particularly the output from the .git/fast_import_crash_<number> file, it's hard to dig any further. For now, I'm going to assume you've moved on and just close the ticket out. But, if you still have access to this repo and can reproduce, please do feel free to retrigger, provide the fast import crash file and reopen this ticket.