search

moosefs / moosefs

MooseFS Distributed Storage – Open Source, Petabyte, Fault-Tolerant, Highly Performing, Scalable Network Distributed File System / Software-Defined Storage
https://moosefs.com
GNU General Public License v2.0
1.69k stars 207 forks source link

[FEATURE] mfscli/mfscgi report used and available metadata dump space #480

Open inkdot7 opened 2 years ago

inkdot7 commented 2 years ago

Have you read through available documentation, open Github issues and Github Ideas Discussions?

Yes

System information

Your moosefs version and its origin (moosefs.com, packaged by distro, built from source, ...).

3.0.115 (Debian bullseye packages)

Operating system (distribution) and kernel version.

Debian, Linux 5.10.0-14-amd64

Hardware / network configuration, and underlying filesystems on master, chunkservers, and clients.

10 Gbps network, ext4 for master, xfs for chunkservers

How much data is tracked by moosefs master (order of magnitude)?

Describe your request:

What new feature would you like to see implemented in MooseFS?

Report the amount of space used by metadata (and its copies) in /var/lib/mfs and the amount of free space available for /var/lib/mfs in monitoring.

When the amount of free space is e.g. < 25 % of the used space, indicate as a possible issue with red color.

Why this feature? Is it a necessity or a nice to have? Is this feature related to any other features or problems in the open issues?

Why? Warn unsuspecting / newbie user before out-of-space... Prio? Nice-to-have. Related requests Did not find.

I just ran into the issue of a full /var/lib/mfs as it was residing on a shared and small /var filesystem, when testing to create a lot of small files. Strange coincidence that I had not many hours earlier read the '2. Enough space for metadata dumps' in moosefs -> support -> best practices.

While it may not have helped in this particular case, during normal operating, where filling of /var/lib/mfs is likely to be more gradual, such monitoring and warning has a chance to be noticed by an operator before becoming critical.

Probably a 50 % warning margin is better. It seems that the metadata changelogs can grow considerably when chunks are moved around as well.

Similar reporting for metadata followers would also be useful.

borkd commented 2 years ago

While on the nice-to-have topic, mfsmaster and mfsmetalogger know the number of change logs or metadata copies they are expected to retain (config variables). When (re)started on an existing configuration it would not hurt to periodically evaluate config constrains against actual metadata footprint and emit an early warning if projected use will exceed a given threshold. I.e. warn if projected usage suggests file system becoming 80% full. One could go as far as asking for an emergency threshold as well, and initiate clean shutdown if projected use exceeds whatever that might be.