search

oss-tsukuba / gfarm

distributed file system for large-scale cluster computing and wide-area data sharing. provides fine-grained replica location control.
Other
32 stars 12 forks source link

newest replica may be lost at file closing, if gfmd load average is too high. #419

Open gfarm-admin opened 12 years ago

gfarm-admin commented 12 years ago

this happened when the load average of gfmd is extremely high,
and a gfsd closed its network connection to gfmd
due to the network_receive_timeout condition
(i.e. the gfmd nearly hung due to the high load average),
and #420 (a site-wide network failure) happend at the same time.

If this problem happens, one of the following message numbers will be logged with "error occurred during close operation for writing" on the filesystem node:

In that case, the following operations are strongly recommended:

For all problems which may cause "lost all replicas", see a meta ticket #474

Reported by: n-soda

Original Ticket: gfarm/tickets/419

gfarm-admin commented 12 years ago

Original comment by: n-soda

gfarm-admin commented 12 years ago

a workaround is added in r6469 and r6470 on the main trunk,
and in r6471 on the 2.5 release branch as follows:

now gfsd and gfmd logs this condition (gfmd's log is not guaranteed though),
thus a system administrator can rescue the replica
(because the replica should remain with old generation number),
it's recommended that a monitoring system watches this error logs.

Original comment by: n-soda

gfarm-admin commented 12 years ago

Original comment by: n-soda

gfarm-admin commented 12 years ago

Original comment by: n-soda

gfarm-admin commented 12 years ago

Original comment by: n-soda

gfarm-admin commented 12 years ago

Original comment by: n-soda

gfarm-admin commented 12 years ago

Original comment by: n-soda

gfarm-admin commented 12 years ago

Original comment by: n-soda

gfarm-admin commented 12 years ago

Original comment by: n-soda

gfarm-admin commented 12 years ago

Replying to n-soda:

and in r6471 on the 2.5 release branch as follows:

This was released as gfarm-2.5.6

Original comment by: n-soda

gfarm-admin commented 10 years ago

Original comment by: otatebe

gfarm-admin commented 10 years ago

Original comment by: otatebe

gfarm-admin commented 10 years ago

Original comment by: otatebe

gfarm-admin commented 10 years ago

Diff:

Original comment by: otatebe

gfarm-admin commented 10 years ago

Original comment by: otatebe

gfarm-admin commented 10 years ago

Original comment by: otatebe

gfarm-admin commented 10 years ago

Original comment by: otatebe

gfarm-admin commented 10 years ago

Original comment by: otatebe

gfarm-admin commented 9 years ago

Original comment by: otatebe

gfarm-admin commented 9 years ago

Original comment by: otatebe