search

mesos / mesos-go

Go language bindings for Apache Mesos
Apache License 2.0
545 stars 146 forks source link

Subscribe Errors with DCOS/Mesos #349

Open kstedman9 opened 6 years ago

kstedman9 commented 6 years ago

I am trying to write a service that will subscribe to the Mesos Master to watch all the Tasks that are being run on the Cluster. So I started with the mwatch.go to see if things will work properly with DCOS. When I run the code I get an error : Error: bufio.Scanner: token too long

So I started debugging through the code to try to figure out what may be happening and I found a way to get it to work. I saw talk about it in another thread but that has been closed.

If I change the httpcli/http.go code and change the recordIOSourceFactory function to this it works all the time:

func recordIOSourceFactory(r io.Reader) encoding.Source {
    return func() framing.Reader { return recordio.NewReader(r, recordio.MaxMessageSize(1<<23)) }
}

Is there a way to change this without copying much of the code base or just changing the actual code like I did? I have not found any way to accomplish this.

I have tried many different numbers there but only 1<<23 seems to work. If I use less than that I get an error in other parts of the code. For example if I use 1<<22 I get this error: "Error: oversized frame, max size exceeded"

jdef commented 6 years ago

How big is your cluster? :) I think the max frame size was based on some maximum buffer size that was observed in the protobuf parsing code. We probably shouldn't have hard-coded the number here. Please feel free to submit a PR that refactors this magic number into something more configurable.