search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.82k stars 2.03k forks source link

[BUG] Invalid plot header magic #3143

Closed maximus1218 closed 3 years ago

maximus1218 commented 3 years ago

Bug: After every plot file is done plotting, the file becomes invalid.

Steps to reproduce the behavior:

  1. Go to Add a plot
  2. Once the plot is done, the file becomes invalid (error log at bottom)
  3. The error that I'm receiving in the log states that the plot files trying to be farmed have invalid plot header magic

After the plot file reaches 100%, it will stay there for around 4 - 6 hours. After, the plot will go to a list of Invalid Plots, containing all plots I have tried to make.

Windows 10 OS

CPU is an Intel Core i7-10700

Error Log

05:52:38.391 harvester chia.plotting.plot_tools: ERROR Failed to open file F:\plot-k32-2021-04-23-12-55-b8ddead30cee559d886c3b47c10dec32c66e9c6e6ff60e3ac08b13a0e4c95827.plot. Invalid plot header magic Traceback (most recent call last): File "chia\plotting\plot_tools.py", line 185, in load_plots ValueError: Invalid plot header magic

parko86 commented 3 years ago

Same here

zedzara commented 3 years ago

hey maximus! have you found any solutions yet? I have the same annoying issue..

maximus1218 commented 3 years ago

hey maximus! have you found any solutions yet? I have the same annoying issue..

No sorry, I haven't been able to find any solutions yet. Not sure what's causing it. What are you guys farming on?

zedzara commented 3 years ago

Windows 10 Core i7-8700 32GB RAM

so I went up to Keybase, which is somethling like discord. It's set up by Chia team. I asked there in the Support chat about this problem, and a random guy tried to help. didn't really help me, but it might help you. please let me know if it did:

"if you're having issues plotting (hung thread, corrupted files), it could be hardware related. make sure your drives has no bad sectors, run some stress test on memory and CPU. I've had issues plotting due to bad sectors on hard drive, and overly aggressive memory and CPU overclocks. once i've lowered my clocks and ran stress test tools to ensure everything fine, my plotting hasn't failed since"

"most os allows you to scan your hard drive, or you can use tools such as crystal disk info to check for bad sectors. on ubuntu, the os disk management util shows bad sectors if they're present. i think all of them just read SMART data off the disks if you see bad sectors on the drive, you may wanna switch em out"

maximus1218 commented 3 years ago

Windows 10 Core i7-8700 32GB RAM

so I went up to Keybase, which is somethling like discord. It's set up by Chia team. I asked there in the Support chat about this problem, and a random guy tried to help. didn't really help me, but it might help you. please let me know if it did:

"if you're having issues plotting (hung thread, corrupted files), it could be hardware related. make sure your drives has no bad sectors, run some stress test on memory and CPU. I've had issues plotting due to bad sectors on hard drive, and overly aggressive memory and CPU overclocks. once i've lowered my clocks and ran stress test tools to ensure everything fine, my plotting hasn't failed since"

"most os allows you to scan your hard drive, or you can use tools such as crystal disk info to check for bad sectors. on ubuntu, the os disk management util shows bad sectors if they're present. i think all of them just read SMART data off the disks if you see bad sectors on the drive, you may wanna switch em out"

None of my drives have bad sectors, but I'll try to plot without stressing my CPU as much and tell you what happens.

zedzara commented 3 years ago

Windows 10 Core i7-8700 32GB RAM so I went up to Keybase, which is somethling like discord. It's set up by Chia team. I asked there in the Support chat about this problem, and a random guy tried to help. didn't really help me, but it might help you. please let me know if it did: "if you're having issues plotting (hung thread, corrupted files), it could be hardware related. make sure your drives has no bad sectors, run some stress test on memory and CPU. I've had issues plotting due to bad sectors on hard drive, and overly aggressive memory and CPU overclocks. once i've lowered my clocks and ran stress test tools to ensure everything fine, my plotting hasn't failed since" "most os allows you to scan your hard drive, or you can use tools such as crystal disk info to check for bad sectors. on ubuntu, the os disk management util shows bad sectors if they're present. i think all of them just read SMART data off the disks if you see bad sectors on the drive, you may wanna switch em out"

None of my drives have bad sectors, but I'll try to plot without stressing my CPU as much and tell you what happens.

what is the number of threads you set for plotting? I heared that it's unecessary to set it higher than 2-4.

maximus1218 commented 3 years ago

Windows 10 Core i7-8700 32GB RAM so I went up to Keybase, which is somethling like discord. It's set up by Chia team. I asked there in the Support chat about this problem, and a random guy tried to help. didn't really help me, but it might help you. please let me know if it did: "if you're having issues plotting (hung thread, corrupted files), it could be hardware related. make sure your drives has no bad sectors, run some stress test on memory and CPU. I've had issues plotting due to bad sectors on hard drive, and overly aggressive memory and CPU overclocks. once i've lowered my clocks and ran stress test tools to ensure everything fine, my plotting hasn't failed since" "most os allows you to scan your hard drive, or you can use tools such as crystal disk info to check for bad sectors. on ubuntu, the os disk management util shows bad sectors if they're present. i think all of them just read SMART data off the disks if you see bad sectors on the drive, you may wanna switch em out"

None of my drives have bad sectors, but I'll try to plot without stressing my CPU as much and tell you what happens.

what is the number of threads you set for plotting? I heared that it's unecessary to set it higher than 2-4.

I've been setting 8 but I'm gonna try a 101 Gb plot with 4 threads right now

zedzara commented 3 years ago

Windows 10 Core i7-8700 32GB RAM so I went up to Keybase, which is somethling like discord. It's set up by Chia team. I asked there in the Support chat about this problem, and a random guy tried to help. didn't really help me, but it might help you. please let me know if it did: "if you're having issues plotting (hung thread, corrupted files), it could be hardware related. make sure your drives has no bad sectors, run some stress test on memory and CPU. I've had issues plotting due to bad sectors on hard drive, and overly aggressive memory and CPU overclocks. once i've lowered my clocks and ran stress test tools to ensure everything fine, my plotting hasn't failed since" "most os allows you to scan your hard drive, or you can use tools such as crystal disk info to check for bad sectors. on ubuntu, the os disk management util shows bad sectors if they're present. i think all of them just read SMART data off the disks if you see bad sectors on the drive, you may wanna switch em out"

None of my drives have bad sectors, but I'll try to plot without stressing my CPU as much and tell you what happens.

what is the number of threads you set for plotting? I heared that it's unecessary to set it higher than 2-4.

I've been setting 8 but I'm gonna try a 101 Gb plot with 4 threads right now

yes, that's what I wanted to suggest. I was doing also like 6-8 threads. now I went back to 4 threads. Currently I am at 58% in the plotting process. I'll let you know if it's working fine.

maximus1218 commented 3 years ago

Windows 10 Core i7-8700 32GB RAM so I went up to Keybase, which is somethling like discord. It's set up by Chia team. I asked there in the Support chat about this problem, and a random guy tried to help. didn't really help me, but it might help you. please let me know if it did: "if you're having issues plotting (hung thread, corrupted files), it could be hardware related. make sure your drives has no bad sectors, run some stress test on memory and CPU. I've had issues plotting due to bad sectors on hard drive, and overly aggressive memory and CPU overclocks. once i've lowered my clocks and ran stress test tools to ensure everything fine, my plotting hasn't failed since" "most os allows you to scan your hard drive, or you can use tools such as crystal disk info to check for bad sectors. on ubuntu, the os disk management util shows bad sectors if they're present. i think all of them just read SMART data off the disks if you see bad sectors on the drive, you may wanna switch em out"

None of my drives have bad sectors, but I'll try to plot without stressing my CPU as much and tell you what happens.

what is the number of threads you set for plotting? I heared that it's unecessary to set it higher than 2-4.

I've been setting 8 but I'm gonna try a 101 Gb plot with 4 threads right now

yes, that's what I wanted to suggest. I was doing also like 6-8 threads. now I went back to 4 threads. Currently I am at 58% in the plotting process. I'll let you know if it's working fine.

What are you sending plot files to once they're done plotting?

zedzara commented 3 years ago

from SSD nvme drive to regular HDD 4TB. gonna buy bigger SSD nvme drive later

zedzara commented 3 years ago

again.. at 100% for almost 1.5 hours now.. this is crazy.

zedzara commented 3 years ago

i'm starting to get suspicious of the hard drive.. what do you choose for destination?

zedzara commented 3 years ago

Okay.. So while my plotting is stuck at 100%, my plot file is already ready in my Plot Temp folder (my SSD drive) with a "plot.temp" at the end.. You know, it should copy it to the HDD drive where the rest of the 101GB plot files are. So I'm guessing I have a shitty hard drive, because when I try to copy simple files on hard drive, it only lets 10MB/s transfer speed, and even that is jumping between 0 and 10MB/s. So that fact just upsets me enough.

Anyway.. A probable solution: -I rewrite the ending to be simply ".plot" -and then copy that to a hard drive that works properly -and then "Add plot directory" (in Chia software, at "Plots" tab, next to the "+ Add a plot" button, click on the three dots, you will see it)
it will recognize it.

maximus1218 commented 3 years ago

Okay.. So while my plotting is stuck at 100%, my plot file is already ready in my Plot Temp folder (my SSD drive) with a "plot.temp" at the end.. You know, it should copy it to the HDD drive where the rest of the 101GB plot files are. So I'm guessing I have a shitty hard drive, because when I try to copy simple files on hard drive, it only lets 10MB/s transfer speed, and even that is jumping between 0 and 10MB/s. So that fact just upsets me enough.

Anyway.. A probable solution: -I rewrite the ending to be simply ".plot" -and then copy that to a hard drive that works properly -and then "Add plot directory" (in Chia software, at "Plots" tab, next to the "+ Add a plot" button, click on the three dots, you will see it) it will recognize it.

Alright well I just got a new HDD in the mail so ill try with that and test the solution you've provided. I also noticed this in the log while testing a 101 Gb plot this morning...

Could not copy "D:\plot-k32-2021-05-03-11-50-1ffe8e855b1ac64e25413e3aea39b4d2bccda57a6fc52f62ae873063034529a7.plot.2.tmp" to "F:\plot-k32-2021-05-03-11-50-1ffe8e855b1ac64e25413e3aea39b4d2bccda57a6fc52f62ae873063034529a7.plot.2.tmp". Error The volume for a file has been externally altered so that the opened file is no longer valid

It looks to me like theres an issue moving the plot file from the temp directory to the final directory due to a problem with the HDD. Im guessing chia trys to make the plot files movable onto the HDD, which leads to the file being rendered useless. Ill let you know what happens when I use this new HDD as the final directory.

RollingRoland commented 3 years ago

I just encountered this problems for 3 files. 1) I don't think the drive is the problem. All the plot files that are created on that machine and written on that drive are OK. 2) Only plot files that have been created on another machine and moved manually on that drive are showing up as "Invalid". 3) Files that were created more that 3 days ago and moved in such a way are OK. This makes me suspect that the move from 1.1.2 to 1.1.3 I made around that time has something to do with this issue appearing.

maximus1218 commented 3 years ago

I just encountered this problems for 3 files.

  1. I don't think the drive is the problem. All the plot files that are created on that machine and written on that drive are OK.
  2. Only plot files that have been created on another machine and moved manually on that drive are showing up as "Invalid".
  3. Files that were created more that 3 days ago and moved in such a way are OK. This makes me suspect that the move from 1.1.2 to 1.1.3 I made around that time has something to do with this issue appearing.

All of the plot files that came back as invalid were created on this machine.

meyerjoe83 commented 3 years ago

Also interested in this

xiaoliwe commented 3 years ago

so,Does have any the solutions for this issue?

maximus1218 commented 3 years ago

Alright, so it is the hard drive causing the plot files to become corrupt. I checked my SSD and HDD with Hard Disk Sentinel and Crystal Disk. I also ran diagnostics on my drives, just to find that all sectors were fine and no problems were detected. I have no idea what in the HDD was causing the plot files to become corrupted but when I installed an 8TB Seagate Nas HDD, the plot files were able to transfer to the HDD set as the final directory. So solution? Get a new HDD for your machine. The program needs to be written so that the files don't try to convert when one of these HDDs is set as the final directory. But I am still not sure exactly what was causing this to occur in the HDD.

joselcl98 commented 3 years ago

I had the same problem, and my problem was that there was a program that was preventing the modification of the file, if it does not let you or move it to another unit it may be the problem

erickseibert commented 3 years ago

I had the same problem, and my problem was that there was a program that was preventing the modification of the file, if it does not let you or move it to another unit it may be the problem

I created an account here just so I can reply to this thread and give credits to this guy. I'm also mining using Hpool and somehow it was preventing Chia GUI to recognize the plots. I did follow his instructions and now all plots are working again.

Kudos to you, sir.

hglakfs commented 3 years ago

Can you tell me how to deal with it in detail？

erickseibert commented 3 years ago

Can you tell me how to deal with it in detail？

First you need to check if there is another app using the plots by moving them. Just cut the file and try to paste it somewhere else. If Windows denies it because something else is using the file, well, something else is using the file.

Now you need to identify which app it is. In my case, I was using a pool miner called HPool, and it was not allowing Chia miner to use the plot.

I closed both Chia miner and HPool and opened Chia miner again = Great success! The plots were valid and farming once more. Oh and HPool could resume mining as well.

reza4247 commented 3 years ago

i have this problem too. I have no problem with my hard drive and no third program / service that wants to control the plotts. I cut out plots and copied them to another hard drive but no result. my plotts are produced by the main computer and they were OK but suddenly got errors! 20210604_183512

reza4247 commented 3 years ago

45 plotts are Invalid:( ![Uploading 20210604_183737.jpg…]()

github-actions[bot] commented 3 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

github-actions[bot] commented 3 years ago

This issue was automatically closed because it has been flagged as stale and subsequently passed 7 days with no further activity.