collecting data with longer durations

[jdevoo]

Hello Adrian, Hello folks, I have compiled spigo on Windows and was looking at the CSV reports. The command I am trying specifically to generate flow files is e.g.

./spigo.exe -c -d 10 -a lamp

The above works and terminates properly on my laptop with -d 2 and -d 5 but not with -d 10. I noticed a hangup in asgard's ShutdownNodes() on the second round trying to read from listener. Has anyone else had this? I am learning Go still and am not yet comfortable with either the language or the way pirates are handled. I was for example not sure if I had to expect the victim in the noodle list...

TIA

[adrianco]

Hi, that's a bug. It fails for me as well on Mac. With -m option it lists the messages then hangs. It's some kind of race condition that leads to a deadlock. I tried removing the chaosmonkey entry from the arch file but it still hung. It fails whenever the -c option is used AFAIK, so the bug is probably in the flow.go package. I don't have time to look into it further right now, but will have a look soon.

[hubayirp]

Good to know between m and c options. On Thu, Jun 23, 2016 at 1:16 PM Adrian Cockcroft notifications@github.com wrote:

Hi, that's a bug. It fails for me as well on Mac. With -m option it lists the messages then hangs. It's some kind of race condition that leads to a deadlock. I tried removing the chaosmonkey entry from the arch file but it still hung. It fails whenever the -c option is used AFAIK, so the bug is probably in the flow.go package. I don't have time to look into it further right now, but will have a look soon.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/adrianco/spigo/issues/74#issuecomment-228170269, or mute the thread https://github.com/notifications/unsubscribe/ACWGWER4mGvLt-YiCAeUyUUltkL2pf0tks5qOumNgaJpZM4I8HmN .

[jdevoo]

Thank you @adrianco I noticed a lock in denominator at flow.Instrument(msg, name, nethist) which took me inside collect.Measure. Based on what I understood from Go's mutexes, I isolated a portion of Measure into an unexported function and got it to work. See if that makes sense and if outputs in csv_metrics for example make sense...

func Measure(h metrics.Histogram, d time.Duration) {
        if h != nil && archaius.Conf.Collect {
                if d > maxHistObservable {
                        h.Observe(int64(maxHistObservable))
                } else {
                        h.Observe(int64(d))
                }
                sampleLock.Lock()
                defer sampleLock.Unlock()
                myMeasure(h, d)
//              s := sampleMap[h]
//              if s != nil && len(s) < sampleCount {
//                      sampleMap[h] = append(s, int64(d))
//              sampleLock.Unlock()
//              }
        }
}

func myMeasure(h metrics.Histogram, d time.Duration) {
        s := sampleMap[h]
        if s != nil && len(s) < sampleCount {
                sampleMap[h] = append(s, int64(d))
        }
}

With this, I can run spigo -c -d 100 -a lamp for example without locking up. I am using it with Neo4j and loved the approach! More to explore for me still!!!