search

glandium / git-cinnabar

git remote helper to interact with mercurial repositories
GNU General Public License v2.0
295 stars 61 forks source link

`git cinnabar fsck` reports bad files #295

Open saschanaz opened 2 years ago

saschanaz commented 2 years ago 
> git cinnabar --version
0.5.9a
module-hash: 5e3702f55a47d0bbe5d819cb93d960888138b9be
helper-hash: 35b90b2bf83cfb025e256ca1d65dc8763437b517

> git cinnabar fsck
Checking 8 changeset heads
Loading 17316 manifests
Checking 23 manifest heads
Sha1 mismatch for file media/mozva/mozva.c
  revision 8e7f694721797cb82f6573f32c6d3ffb6ef15fac
  with parent 49aac12cc4dc6cc360de7c71fa93903341522857
Sha1 mismatch for file media/mozva/va/README
  revision 0250c7ce5720e7c10510d41ce30829da3a09fe1b
Sha1 mismatch for file media/mozva/va/va.h
  revision d2566169de6a0f824b56dedec361e05fe8f3c916
Sha1 mismatch for file media/mozva/va/va.patch
  revision 05a5c75ab6c1cab7c28571c2b2b2b8b5b6831463
Sha1 mismatch for file media/mozva/va/va_dec_av1.h
  revision b8bf4cc914f475fabfd505d223f7f8dca2863255
Sha1 mismatch for file media/mozva/va/va_dec_vp8.h
  revision bb3beb2542ff23bb0b1525d691fb3a87c5e76831
Sha1 mismatch for file media/mozva/va/va_dec_vp9.h
  revision 4feb8c50a635bb97369008c4a1f1ba44bfd7388c
Sha1 mismatch for file media/mozva/va/va_version.h
  revision 8017bcff5b3b54798c5408f2c983abb8e86f755b
Sha1 mismatch for file third_party/rust/clap/src/build/app/settings.rs
  revision 7d202d211c65ebc03593294d47c9c8589178592b
Sha1 mismatch for file third_party/rust/clap/src/build/arg/mod.rs
  revision 682fe035050186d92e7495025d49153174333946
Sha1 mismatch for file third_party/rust/clap/src/build/arg/settings.rs
  revision 51adf7ed600b68b0294f1df76af483227bf67675
Sha1 mismatch for file third_party/rust/clap/src/build/arg_group.rs
  revision 1eec6e8289428b9eb98d5a4dc9b2612325fd7437
Sha1 mismatch for file third_party/rust/clap/src/build/macros.rs
  revision a88060e7a1e2e4b949befa0178dcbe9426bdc9a5
Sha1 mismatch for file third_party/rust/terminal_size/README.md
  revision cf8539acb96fde89b45aa27555b29c3d0f1afd75
Checking 47265 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues

Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.

Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.

Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.

But those revision don't seem to actually exist?

glandium commented 2 years ago

Is it the same repo you fixed in #280? Can you create a git bundle the same way you did back then?

glandium commented 2 years ago

Can you do another one with git bundle create $file refs/cinnabar/metadata refs/cinnabar/checked refs/cinnabar/broken? (without --not ...)

saschanaz commented 2 years ago

Sent it 👍

glandium commented 2 years ago

This time, your repo is more busted than last time, but it's still the same root cause (#207). I'm bummed that the situation was not detected.

You seem to have a setup that amplified the problem, creating a diverging history from gecko-dev. I suspect you have a pref like cinnabar.data or remote.something.cinnabar-data set to always. I wouldn't recommend you use that.

You can fix your repo with git cinnabar rollback b1fe9b8493dbb38038c230f2fb346bfa803a60fe and re-pulling, but any local branch you have will need to be rebased.

I think you're the first person to whom this happened twice, so if you don't mind, I'd like to ask you to set something up after I work on some debugging code.

saschanaz commented 2 years ago

You seem to have a setup that amplified the problem, creating a diverging history from gecko-dev. I suspect you have a pref like cinnabar.data or remote.something.cinnabar-data set to always. I wouldn't recommend you use that.

No, this is all I have:

> git config --get-regexp cinnabar
cinnabar.version-check 1643634802
cinnabar.helper C:\Users\sasch\Documents\GitHub\git-cinnabar\git-cinnabar-helper.exe
cinnabar.fsck 1637682708

I think you're the first person to whom this happened twice, so if you don't mind, I'd like to ask you to set something up after I work on some debugging code.

👍 Feel free to ping me whenever it's ready.

saschanaz commented 2 years ago

During rebase...

> git pull
WARNING Mercurial libraries not found. Falling back to experimental native access.
Reading 1836 changesets
Reading and importing 1830 manifests
Reading and importing 3114 revisions of 1130 files
Importing 1836 changesets
Unpacking objects: 100% (40/40), 58.72 KiB | 7.34 MiB/s, done.
ERROR Error in file b'f917b11e765e626d9459c18cc3771e4decd75880'
ERROR Error in file b'8fc2e6a72551b0e749377adf2af35cdbd14c94ff'
Checking 1417 imported file root and head revisions
ERROR Object of type bytes is not JSON serializable
Run the command again with `git -c cinnabar.check=traceback <command>` to see the full traceback.
fatal: Unsupported command: error
fast-import: dumping crash report to .git/fast_import_crash_21684
fatal: error while running fast-import

> git cinnabar fsck
Checking 4 changeset heads
Loading 17606 manifests
Checking 21 manifest heads
Sha1 mismatch for file testing/web-platform/meta/cookies/attributes/secure.https.html.ini
  revision 8fc2e6a72551b0e749377adf2af35cdbd14c94ff
Sha1 mismatch for file gfx/angle/checkout/third_party/vulkan-deps/vulkan-headers/src/include/vulkan/vulkan_core.h
  revision f917b11e765e626d9459c18cc3771e4decd75880
Checking 48228 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues

Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.

Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.

Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.

👀 Should I send one again?

The crash report has nothing interesting:

fast-import crash report:
    fast-import process: 21684
    parent process     : 1
    at 2022-02-01 12:06:15 +0000

fatal: Unsupported command: error

Most Recent Commands Before Crash
---------------------------------
* error

Active Branch LRU
-----------------
    active_branches = 0 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Inactive Branches
-----------------

Marks
-----

-------------------
END OF CRASH REPORT
glandium commented 2 years ago 
ERROR Error in file b'f917b11e765e626d9459c18cc3771e4decd75880'
ERROR Error in file b'8fc2e6a72551b0e749377adf2af35cdbd14c94ff'
Checking 1417 imported file root and head revisions
ERROR Object of type bytes is not JSON serializable

Is it the first time you see something like this?

> git config --get-regexp cinnabar
cinnabar.version-check 1643634802
cinnabar.helper C:\Users\sasch\Documents\GitHub\git-cinnabar\git-cinnabar-helper.exe
cinnabar.fsck 1637682708

How do you push to try?

saschanaz commented 2 years ago

Is it the first time you see something like this?

I believe I saw it at least once before this 🤔

How do you push to try?

I use ./mach try.

glandium commented 2 years ago

I believe I saw it at least once before this thinking

I wish you had reported it back then because it showed that the diagnostic code for #207 worked but failed to report the problem correctly with python3 :-/

I use ./mach try

Do you maybe git cinnabar fetch from try?

+1 Feel free to ping me whenever it's ready.

Can you update git-cinnabar to latest master, run git config cinnabar.check unbundler, then start again from https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1026529368 ? If it doesn't break again on that update, like it did in https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1026774885, can you keep that cinnabar.check value set until the next time it happens, at which time it will hopefully give you some instructions.

saschanaz commented 2 years ago

I wish you had reported it back then because it showed that the diagnostic code for #207 worked but failed to report the problem correctly with python3 :-/

Oops, sorry 😬

Do you maybe git cinnabar fetch from try?

TIL it exists, but I think I never fetch from try.

Can you update git-cinnabar to latest master, run git config cinnabar.check unbundler, then start again from #295 (comment) ?

Will do that 👍 (Edit: It didn't break)

glandium commented 2 years ago

TIL it exists, but I think I never fetch from try.

Something doesn't line up. When you push to try, can you check whether git rev-parse refs/cinnabar/metadata returns the same thing before and after you do the push?

saschanaz commented 2 years ago 
> git pull --rebase origin bookmarks/central:main
WARNING Mercurial libraries not found. Falling back to experimental native access.
Reading 299 changesets
Reading and importing 299 manifests
Reading and importing 16 revisions of 12 filesfatal: Malformed file chunk for a59a87e470eca653bd2a0c76662372c9e5e1fe7b
fast-import: dumping crash report to .git/fast_import_crash_9400
Reading and importing 264 revisions of 249 files
Traceback (most recent call last):
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1003, in run
fatal: Unsupported command: error
f    retcode = func(args)
ast-import: dumping crash report to .git/fast_import_crash_17980
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\git-remote-hg", line 53, in main
    helper.run()
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\remote_helper.py", line 87, in run
    self.import_(*args)
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\remote_helper.py", line 422, in import_
    getbundle(self._repo, self._store, heads,
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 1071, in getbundle
    apply_bundle(store)
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 893, in __call__
    for rev_chunk in progress_enum(
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 224, in progress_enum
    for count, item in enum_iter:
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 869, in enumerate_files
    for count_chunks, (name, chunk) in enumerate(iterator, start=1):
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 837, in iter_files
    fh.write(chunk)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\git-remote-hg", line 59, in <module>
    run(main, sys.argv[1:])
fatal: error while running fast-import
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1012, in run
    message = message or ', '.join(
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1013, in <genexpr>
    fsdecode(a) for a in getattr(e, 'args', []))
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python310\lib\os.py", line 822, in fsdecode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not int
Exception ignored in atexit callback: <bound method GitHgHelper.close of <class 'cinnabar.helper.GitHgHelper'>>
Traceback (most recent call last):
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\helper.py", line 513, in close
    super(GitHgHelper, self).close()
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\helper.py", line 108, in close
    self._helper.wait()
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 835, in wait
    fh.close()
OSError: [Errno 22] Invalid argument
> git cinnabar fsck
Checking 3 changeset heads
Loading 331 manifests
Checking 3 manifest heads
Sha1 mismatch for file toolkit/components/glean/tests/browser/browser_fog_gmp.js
  revision 61b8f076564ff8c1f2b16b5438b021fe152827e8
  with parent 56b7e3dc829cecaac0f860c2479d90babd9a5f71
Sha1 mismatch for file toolkit/components/glean/tests/browser/browser_fog_socket.js
  revision d956b2c06e402732a8548180afde2e1b9176fa53
  with parent de7589f6616d03e269aeeb5e0509354e136f41a1
Sha1 mismatch for file devtools/client/netmonitor/docs/architecture.md
  revision 2e0a5c33feaa57639a305f1c10eaa2789d17592f
  with parent 5d7e2850f40bc245751a4388f7435269f675c888
Checking 1232 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues

Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.

Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.

Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.

👀

saschanaz commented 2 years ago

Hmm, I ran git cinnabar fsck --force and got tons of mismatches 🤔, I wonder how it happened...

glandium commented 2 years ago

did you git cinnabar rollback b1fe9b8493dbb38038c230f2fb346bfa803a60fe as per https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1026529368 ?

saschanaz commented 2 years ago

I did git cinnabar rollback --fsck instead and thought it should be equivalent 👀 (as it did require rebasing.)

glandium commented 2 years ago

Kind of. I'm baffled you didn't see the other error message before that...

saschanaz commented 2 years ago

Which error message?

glandium commented 2 years ago

https://github.com/glandium/git-cinnabar/blob/6ee02238b841b67f63b5d4d5dbc5cf13f0de7dce/cinnabar/githg.py#L1324-L1336

saschanaz commented 2 years ago

No, haven't seen that one so far, sorry 🥲

glandium commented 2 years ago

Do you still have the repo from https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1031561207 ?

saschanaz commented 2 years ago

I don't but I made a gitbundle from it, so maybe I can say I do?

glandium commented 2 years ago

Is it a different clone from the previous one?

saschanaz commented 2 years ago

Same clone but after git cinnabar rollback (and right after the errors)

glandium commented 2 years ago

That doesn't sound true. The bundle you sent me has a bustage that started in august (which explains why you didn't get the bustage detection error), and doesn't contain b1fe9b8493dbb38038c230f2fb346bfa803a60fe

saschanaz commented 2 years ago

I mean right after https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1031561207, I made the bundle after this as I expected your request to share it, 🤔

Weird that it still had the bustage, I guess git cinnabar rollback --fsck didn't really work then? (I wonder that's why I saw https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1031598598)

glandium commented 2 years ago

But it's a different clone than the one you opened this issue for, right?

saschanaz commented 2 years ago

It was still the same clone. I didn't do git cinnabar reclone, just the rollback.

glandium commented 2 years ago

So, this exposes a flaw in git cinnabar fsck, which is that it doesn't look at all the files that ever existed, but only those referenced by the current heads... So some files could still be broken, and, in your case, there were broken files in metadata older than b1fe9b8493dbb38038c230f2fb346bfa803a60fe. You'd probably be better off with a reclone at this point.

glandium commented 2 years ago

You do still seem to have hit #207 several times, so please keep cinnabar.check set to unbundler.

saschanaz commented 2 years ago

Yeah, I had to reclone to fix the broken pipe error. I'll report again if it happens again.

saschanaz commented 2 years ago

Just sent you the bundle files again. This time there was some interesting things in cinnabar.log:

ERROR authorization failed

ERROR no suitable response from remote hg

ERROR no suitable response from remote hg

ERROR Error in file db948f474be11dfe25e63d795061343452c4162c

ERROR Error in file 332c47c2565d9cb696af07345e6ccfaa5d850ac9

ERROR Error in file 1dc94db8b234de69246beb6f3e2b2b85a94d7c16

ERROR Error in file 5b8d62bc6ac27f86208ecb6a3d026d3470f2e0fa

ERROR It seems you have hit a known, rare, and difficult to reproduce issue.
Your help would be appreciated.
Please keep a copy of the .git/cinnabar-last-bundle file before doing the following.
Please try either `git cinnabar rollback` followed by the same command that just
failed, or `git cinnabar reclone`.
Please open a new issue (https://github.com/glandium/git-cinnabar/issues/new)
mentioning issue #207 and reporting whether the second attempt succeeded.

Please read all the above and keep a copy of this repository.

But not sure it's actually related as there was no timestamps.