search

moosefs / moosefs

MooseFS Distributed Storage – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System / Software-Defined Storage
https://moosefs.com
GNU General Public License v2.0
1.69k stars 207 forks source link

Locked unused files, can't delete #35

Closed zcalusic closed 7 years ago

zcalusic commented 7 years ago

Hello, moosefs 3.0.86 user here.

I have 693 files in pending deletion state (goal 1), but they haven't been deleted in more than a week! The number hasn't changed the whole time. They're not in the trash, 'cause trash has been set to purge after only 1 day, and only 677 files are reported in trash, anyway (which is less than 693, so this is something different).

In chunk operations section, I see exactly 693 chunks in locked/unused state. All other numbers are zero. Can you explain what that state signifies, how I got there, and how I get rid of those chunks?

So far, I tried unmounting mfs on all clients (in case some files are locked that way, to free locks), and mounting again, but that hasn't helped at all.

This is a rather small moosefs installation with 3 cs, several TB on them, and several clients. Using goals 2 & 3. Nothing special. Beside that persistant locked chunks, everything's been working really great, I'm pretty much amazed with the software. Thanks for sharing it!

xandrus commented 7 years ago

Hi, Thank you for this information. I would like to add that in MooseFS we have implemented security mechanism to prevent unknow chunks against immediately deletion.

For example if you some how connect chunkserver from different MooseFS instance, MooseFS master will mark all chunks from such chunkserver as locked and after one week files will be deleted. So you have some time to react.

We had few cases connected with such a scenario. So to prevent unknown chunks deletion we add such a mechanism.

Do you still have this 693 chunks in locked state? Do you have any dropped mfsmount sessions? Maybe your chunkserver was disconnected some time?

Can you send us some system logs from MooseFS master?

zcalusic commented 7 years ago

Hello, and tnx for the answer.

The chunks eventually got deleted, I'd say it took even more than a week, but I'm not sure. This is one and only installation all the time, although I did some operations like relocating master to another host, and copying chunks from one server to another. Both of them succeeded without problems. I can't rememeber the exact timing. In any case, all seems perfect now.

In the master logs there are some (on two occasions) anomalies of the type:

chunkserver (1.2.3.4) has nonexistent chunk (000000000001AFAD_00000001), so create it for future deletion

And some of the type:

(2.3.4.5:9422 -> 6.7.8.9:9422) chunk: 000000000003C906 replication status: Wrong chunk version

I typically don't have dropped sessions, even if I catch one, most likely I'll terminate it manually.

I forgot to mention one thing, I'm breaking one rule from the recommended setup, the chunkservers don't have exclusive access to the disks, instead they are using free space on otherwise already used partitions. I understand this can produce higher rate of replications due to usage leveling, although I can live with that. I don't know if this higher rate of replication can provoke other problems.

zcalusic commented 7 years ago

Since this hasn't happen since, and resolved all by itself, I will close the issue. Probably the info could be added in the documentation and/or FAQ.