search

animetosho / ParPar

High performance PAR2 create client for NodeJS
190 stars 19 forks source link

Repair completed but the data files still appear to be damaged #44

Closed Fossil01 closed 1 year ago

Fossil01 commented 1 year ago

Hi,

Odd question maybe. Seen many downloads fail recently, the PAR2 creation software reported by NZBGet is almost always ParPar (sometimes par2cmdline).

I tried uploading some things with Usenet Farm and then downloading them using 7 USP's in NZBGet 10 mins later. NZBGet reports 99,9% availability even on the same USP used to upload.

After downloading it starts repairing but it always fails with repair completed but the data files still appear to be damaged. Recovery files created by: ParPar v0.4.0-dev [https://animetosho.org/app/parpar]

I checked the logs for one of them and it had 1 bad block in just 1 file. After the failed repair it now had several bad blocks in several files.

Another upload seemed to download fine but then came back with corrupted RARs and another failed repair. The log for that one is pretty long and posted here: https://kopy.io/py5x5#yPCMI7lVUZXKtV

Here's a check on that same file with par2cmdline:

Loading "9d38b600a3c96c4c7a3d401b760a6eee.par2".
Loaded 114 new packets
Loading "9d38b600a3c96c4c7a3d401b760a6eee.par2".
No new packets found

There are 56 recoverable files and 0 other files.
The block size used was 5242880 bytes.
There are a total of 2225 data blocks.
The total size of the data files is 11663825678 bytes.

Verifying source files:

Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part01.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part02.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part01.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part02.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part03.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part04.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part03.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part04.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part05.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part06.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part05.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part06.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part07.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part08.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part07.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part08.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part09.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part10.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part09.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part10.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part11.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part12.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part11.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part12.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part13.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part14.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part13.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part14.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part15.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part16.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part15.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part16.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part17.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part18.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part17.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part18.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part19.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part20.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part19.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part20.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part21.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part22.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part21.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part22.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part23.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part24.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part23.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part24.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part25.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part26.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part25.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part26.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part27.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part28.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part27.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part28.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part29.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part30.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part29.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part30.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part31.rar"
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part32.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part31.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part33.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part32.rar" - damaged. Found 37 of 40 data blocks.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part34.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part33.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part35.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part34.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part36.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part35.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part37.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part36.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part38.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part37.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part39.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part38.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part40.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part39.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part41.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part40.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part42.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part41.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part43.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part42.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part44.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part43.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part45.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part44.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part46.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part45.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part47.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part46.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part48.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part47.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part49.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part48.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part50.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part49.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part51.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part50.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part52.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part51.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part53.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part52.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part54.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part53.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part55.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part54.rar" - found.
Opening: "9d38b600a3c96c4c7a3d401b760a6eee.part56.rar"
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part55.rar" - found.
Target: "9d38b600a3c96c4c7a3d401b760a6eee.part56.rar" - found.

Scanning extra files:

Repair is required.
1 file(s) exist but are damaged.
55 file(s) are ok.
You have 2222 out of 2225 data blocks available.
Repair is not possible.
You need 3 more recovery blocks to be able to repair.

I used Nyuu to upload these 200mb RARs and tried these settings:

  "check-connections": 3,
  "check-tries": 3,
  "check-delay": "5s",
  "check-retry-delay": "5s",
  "check-post-tries": 3,

  "post-retries": 3,
  "request-retries": 3,
  "connect-retries": 3,
  "retry-on-bad-resp": true,
  "reconnect-delay": 3000,

  "skip-errors": false,
  "quiet": false

Meanwhile I used ParPar to create the PARs like this:

parpar.js -n -r5% -s5M --seq-read-size=5M -dpow2 -o outputname.par2 sourcefiles

No clue what's happening and why and hoping you could shed some light on this.

Fossil01 commented 1 year ago

Did some more testing. Uploaded a rarred file (about 11GB), before upload I checked the rarset with par2verify and all was good. Uploaded to Usenet Farm, downloaded on my machine 10 min later and all my 7 USPs had 100% availability according to NZB Get.

Ran par2verify and got multiple failed RARs.

I used 5% repair blocks for ParPar and par2verify fails with:

You have 1598 out of 1612 data blocks available.

Fossil01 commented 1 year ago

Also uploaded a file with 10% repair blocks to Cheapnews and then downloaded the file 5 mins later and NZBGet said it failed 99.9% on all USPs, then it tried Cheapnews (backup block account) for the missing articles (3) and they all failed (100%).

PAR2 still failed to repair with just a few missing blocks.

Where is it going wrong? Is Nyuu breaking something while uploading or are the USPs corrupting the files?

animetosho commented 1 year ago

Are you able to compare the raw downloaded files (i.e. before NZBGet tries to repair/extract them) against those you uploaded?

Can you try running a PAR2 repair manually against the raw downloaded files?

Also, on the good file set (before upload), try moving one of the RAR files to somewhere else, to simulate damage, and try running a repair on that - does it work?

If you suspect Nyuu/ParPar, you can repeat the experiment with different clients.

Fossil01 commented 1 year ago

So I tested a bunch and compared SHA1 sums to the files as rarred and parred on the server before upload. Everytime I download the NZB some different files have incorrect SHA1 sums. So one download it might be part 03, 04 and the next time I download that same NZB it might be a bunch of completely other parts that are broken and 03, 04 are fine.

I can only think the USP messes it up but I tried both Cheapnews and Farm.

Fossil01 commented 1 year ago

Okay so the problem appears to be on my end. I have now downloaded the same NZB on a different (remote) machine and those SHA1 sums all match.

A friend of mine also downloaded the NZB with 3 missing segments but it repaired just fine.

Hardware error on my end?

animetosho commented 1 year ago

Is NZBGet being used in all instances? Are you checking the files before any attempted repair/extraction?

Are you uploading from and downloading to the same drive?
Hardware could be a possibility, but it'd be weird for it to only affect downloads, not uploads as well.

buggsi commented 1 year ago

Okay so the problem appears to be on my end. I have now downloaded the same NZB on a different (remote) machine and those SHA1 sums all match.

A friend of mine also downloaded the NZB with 3 missing segments but it repaired just fine.

Hardware error on my end?

This sounds like a memory problem. I suggest your run a memtest on the RAM.

Fossil01 commented 1 year ago

Sorry I forgot about this issue. I managed to "fix" it by changing some Proxmox settings.

I think something about shared GPU memory.