Open shownb opened 6 years ago
Your dispatcher is building up an unbound number of go routines which doesn't seem like a good idea, but obviously your servers are performing just fine according to your monitoring. Rather than the go routines being the problem, it is the s3upload that is causing havoc when it is unbound I believe. So the following would work I would think:
In the original "Try Again" just add a simple boolean channel. Spawn go routines for the s3 upload and pass the channel as a parameter having it drain itself after s3 upload is complete.
limiter := make(chan bool, MAX_FANOUT) (defined somewhere, globally, etc)
case job := <-Queue:
limiter <- true
go job.payload.UploadToS3(limiter)
Internal to UploadToS3:
<-limiter
Less is more. All you are trying to do is make a 2D processing "space" that you manage (MAX_FANOUT x MAX_QUEUE). I get that what you wrote is a lot sexier than what I am suggesting, but seeing more code than is necessary tends to bother me.
他的意思是case job := <-JobQueue:条件下,每次都搞一个go协程,这样会产生很多的协程。但服务器比较好,都抗住了,但是呢,相比这个问题,s3upload产生的更严重的问题当他无条件产生的时候。
新的demo
读码后感 《通过Go来处理每分钟达百万的数据请求》里面有什么bug 文中的第一次尝试简直是傻逼,没那么简单直接无限go,所以直接爆体了。 第二次尝试,加了chan的缓存,但是居然不用协程了,也是傻逼,所以没多久延时就会其高,因为都在等着Queue <- payload。
综合来看,要考虑的情况就是,究竟要多少协程去处理。 所以说到底就是找怎么控制协程的数量,找到 首先需要知道瓶颈在什么地方,因为瓶颈的存在,好像木桶理论,你其他木板再高,决定装水量的,还是在最短那块板。 知道瓶颈在哪里,再去找平衡,制定策略。 文章中,有几个短板需要注意 1, web方面,单机能处理多少请求 2,能运行处理多少个payload.UploadToS3 瓶颈在不能太多的运行 payload.UploadToS3 的协程,所以单机需要控制协程数量,然后web请求又是固定的(每秒百万),所以需要通过集群来处理。
以下是别人评测,很有道理。哈哈 https://gocn.vip/article/5
参考 http://www.cnblogs.com/artong0416/ https://gobyexample.com/worker-pools go作者