Open vstax opened 6 years ago
@vstax Thanks for reporting. seems like the root cause would be #845 (put and delete requests got inverted chronologically) and the storage node on which compaction is in-progress might be likely to face #845 for some reason. I will dig into further to get to the fact. anyway, just in case let me ask you this issue isn't big deal for your production system? if it is then let me know and I will do my best until the launch.
@mocchira No, it's not of priority, it's just something I noticed when looking at logs from the last experiment. Plus, since recover-file works, it's possible to fix these objects after noticing the errors, just a bit annoying that they have happened at all (I wasn't pushing the system or anything - just launched compaction with default parameters). I actually killed this cluster already so I can't execute "whereis" anymore; I have a copy of all its logs just in case, though. I managed to make somewhat better RING for the new (production) one after a few tries - only up to 8.5% difference in distribution between nodes.
I got one question, though. These clusters have W=2, D=1 - because applications don't really do any deletes on this data, I don't care what D is set to. I just thought, since multipart upload internally executes delete for temporary object - does that delete follow W or D? (I think it makes sense to follow W since it's all part of "write" operation from application perspective but not 100% sure). If it follows D now, could it be that setting D=2 would make a chance of having these errors less?
@vstax
No, it's not of priority
OK.
I got one question, though. These clusters have W=2, D=1 - because applications don't really do any deletes on this data, I don't care what D is set to. I just thought, since multipart upload internally executes delete for temporary object - does that delete follow W or D? (I think it makes sense to follow W since it's all part of "write" operation from application perspective but not 100% sure). If it follows D now, could it be that setting D=2 would make a chance of having these errors less?
Delete for temporary objects follow D so setting D=2 might make these errors less likely to happen.
I think it makes sense to follow W since it's all part of "write" operation from application perspective
This part seems to be arguable to me so let us consider for a while, we will file this problem if we judge Delete for temporary objects should follow W.
@mocchira Well, the thing is - my current configuration (W=2 and D=1) is incorrect regarding MU, I think. Since that delete uses D, I should've used D=W to make sure that all parts of "PUT" operations follow "2" quorum. So it makes sense to use D=2 even though I didn't plan to delete anything - but I had no way of knowing that beforehand. So I think either that delete of temporary object should use W as quorum or it's a good idea to mention it in documentation at least, that running with D smaller than W can create a pitfall regarding multipart upload and it's probably not a good idea.
Actually, now that I think, I don't understand why is D configurable at all. What are circumstances in which someone would want to run cluster where D != W? If it needs to be configurable for some very special reasons, maybe documentation should recommend everyone to always use D=W, even if they don't plan to do any delete, unless they are 100% sure they have good reasons to set it otherwise. This way, people (hopefully) won't repeat my mistake.
@vstax
Actually, now that I think, I don't understand why is D configurable at all. What are circumstances in which someone would want to run cluster where D != W? If it needs to be configurable for some very special reasons, maybe documentation should recommend everyone to always use D=W, even if they don't plan to do any delete, unless they are 100% sure they have good reasons to set it otherwise. This way, people (hopefully) won't repeat my mistake.
AFAIK, no users using D != W. however there are some cases D != W can be useful. the one is to realize read-after-delete consistency but no need read-after-write consistency. in this case, having D=N and W<N enable leofs to handle such scenarios and also keep W availability high while availability only for D get decreased (if there is only W then both availability have to be decreased). As for the documentation, totally agreed with you so we will add this information into somewhere in the doc.
@mocchira I will try, however I'd like to note that version with this problem already included fix from #845, and #926 should not affect this at all because I was uploading objects with script that only uploads each file once. I was uploading objects in parallel but each uploader instance worked with its own set of source files. So I don't think that anything should change because of #926 was fixed as well.
@vstax
however I'd like to note that version with this problem already included fix from #845, and #926 should not affect this at all because I was uploading objects with script that only uploads each file once. I was uploading objects in parallel but each uploader instance worked with its own set of source files. So I don't think that anything should change because of #926 was fixed as well.
I see, will keep vetting.
I'm uploading objects to cluster where one of the nodes (stor06) undergoes compaction. Nothing serious, default options - 4 files at once, the compaction itself goes pretty fast. No errors on gateway but I get errors about multipart objects on storage nodes. This seems a bit like #845 but different because only node that undergoes compaction gets inconsistencies. Logs on stor01
Logs on stor02:
On stor03:
similar situation for stor04 and stor05. All errors mention stor06.
There are also tons of "timeouts" in error logs on these nodes but it looks like all objects mentioned there are in correct state (deleted) so it's probably not a problem:
On stor06, there are various things in logs. First, similar not_found messages:
Timeout errors as well (again, some objects from these lines that I've checked all seem fine):
"case,unavailable" errors. Many lines repeated over and over here. Some objects that I've checked all seem to be fine (deleted):
Some other errors (lots of repeated lines here as well. Objects seem to be fine, deleted):
In info logs just some long operations are logged (with high processing_time) and compaction reports (all "result,success").
Problem: all objects mentioned with "case,not_found" in logs on every node are in inconsistent state on stor06:
(recover-file fixes this fine).