Closed GoogleCodeExporter closed 8 years ago
[deleted comment]
AFAICT, VolumeGrowth struct is locked way too late: multiple request are trying
to grow volumes but the first request already succeed in growing the volumes,
when the VolumeGrowth.accessLock is released there is no more free space in the
topology...
Instead it should be locked before. Here is a quick&dirty patch that fix the
issue:
From 6fef495467227fd00a8d87931c9621cf7a2cbe36 Mon Sep 17 00:00:00 2001
Date: Tue, 29 Oct 2013 15:36:50 +0100
Subject: [PATCH] Issue #51: lock early and double check before trying to grow
volumes.
---
go/weed/master.go | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/go/weed/master.go b/go/weed/master.go
index 7aa48e5..369e1a1 100644
--- a/go/weed/master.go
+++ b/go/weed/master.go
@@ -15,6 +15,7 @@ import (
"strconv"
"strings"
"time"
+ "sync"
)
func init() {
@@ -48,6 +49,7 @@ var (
var topo *topology.Topology
var vg *replication.VolumeGrowth
+var vgLock sync.Mutex
func dirLookupHandler(w http.ResponseWriter, r *http.Request) {
vid := r.FormValue("volumeId")
@@ -97,9 +99,14 @@ func dirAssignHandler(w http.ResponseWriter, r
*http.Request) {
writeJsonQuiet(w, r, map[string]string{"error": "No free volumes left!"})
return
} else {
- if _, err = vg.AutomaticGrowByType(rt, dataCenter, topo); err != nil {
- writeJsonQuiet(w, r, map[string]string{"error": "Cannot grow volume group!
" + err.Error()})
- return
+ vgLock.Lock()
+ defer vgLock.Unlock()
+ // redo the check for active volumes to avoid concurrency issues.
+ if topo.GetVolumeLayout(rt).GetActiveVolumeCount(dataCenter) <= 0 {
+ if _, err = vg.AutomaticGrowByType(rt, dataCenter, topo); err != nil {
+ writeJsonQuiet(w, r, map[string]string{"error": "Cannot grow volume
group! " + err.Error()})
+ return
+ }
}
}
}
--
1.8.3.4 (Apple Git-47)+GitX
Original comment by philo...@gmail.com
on 29 Oct 2013 at 3:00
I've commited the change to a github fork:
https://github.com/zenria/weed-fs/commit/779c22d74073b327959b48cd53684509ebcabe5
6
Original comment by philo...@gmail.com
on 29 Oct 2013 at 3:42
Thanks for the fix! Checked into HEAD now.
Original comment by chris...@gmail.com
on 29 Oct 2013 at 7:49
Original issue reported on code.google.com by
philo...@gmail.com
on 29 Oct 2013 at 1:24