search

turbot / flowpipe

Flowpipe is a cloud scripting engine. Automation and workflow to connect your clouds to the people, systems and data that matters.
https://flowpipe.io
GNU Affero General Public License v3.0
383 stars 16 forks source link

Flowpipe server is crashing under high volume of calls #586

Closed luisffc closed 9 months ago

luisffc commented 10 months ago

While working on pipeling tests, writing pipelines to create reources on multiple AWS accounts in multiple regions, I've seen flowpipe server crashing while processing an amount of calls.

I have a shellscript running flowpipe run pipeline ... in parallel for each region in 3 AWS accounts. After a few minutes flowpipe server crashes.

I tried decreasing the amount of parallel calls, and it avoided flowpipe server to crash at first, but then, running the script again caused the server to crash

goroutine 63 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 64 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0d930, {0x103737b20?, 0x14000dca1e0}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 65 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca2d0, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 66 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 67 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0dad0, {0x103737b20?, 0x14000dca280}, {0x14001115b00, 0x2, 0x14000b8ed30?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 68 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca370, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 69 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 70 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0de10, {0x103737b20?, 0x14000dca320}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 71 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca410, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 72 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 73 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0d2b0, {0x103737b20?, 0x14000dca3c0}, {0x14001115b00, 0x2, 0x12?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 74 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca4b0, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 75 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 76 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0d860, {0x103737b20?, 0x14000dca460}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 77 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca550, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 78 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 79 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0da00, {0x103737b20?, 0x14000dca500}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 80 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca5f0, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 81 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 82 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0dba0, {0x103737b20?, 0x14000dca5a0}, {0x14001115b00, 0x2, 0x13?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 83 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca690, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 84 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 85 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0d110, {0x103737b20?, 0x14000dca640}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 86 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca730, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 87 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 88 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0c8f0, {0x103737b20?, 0x14000dca6e0}, {0x14001115b00, 0x2, 0x749?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 89 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca7d0, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 90 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 91 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0d450, {0x103737b20?, 0x14000dca780}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 92 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca870, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 93 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 94 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0d6c0, {0x103737b20?, 0x14000dca820}, {0x14001115b00, 0x2, 0x14000dc0c00?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 95 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca910, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 96 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 97 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0cea0, {0x103737b20?, 0x14000dca8c0}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 98 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dca9b0, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 99 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 100 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0d5f0, {0x103737b20?, 0x14000dca960}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 101 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcaa50, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 102 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 103 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e44340, {0x103737b20?, 0x14000dcaa00}, {0x14001115b00, 0x2, 0x1400117acc0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 104 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcaaf0, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 105 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 106 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0ca90, {0x103737b20?, 0x14000dcaaa0}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 107 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcab90, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 108 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 109 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0cd00, {0x103737b20?, 0x14000dcab40}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 110 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcac30, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 111 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 112 [chan receive, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0cdd0, {0x103737b20?, 0x14000dcabe0}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 113 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcacd0, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 114 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 115 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0dd40, {0x103737b20?, 0x14000dcac80}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 116 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcad70, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 117 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 118 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e441a0, {0x103737b20?, 0x14000dcad20}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 119 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcae10, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 120 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 121 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e44410, {0x103737b20?, 0x14000dcadc0}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 122 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcaeb0, 0x14000ce5810)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 123 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 124 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0cf70, {0x103737b20?, 0x14000dcae60}, {0x14001115b00, 0x2, 0x0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 125 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe.func1(0x14000dcaf50, 0x14000ce58c0)
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:198 +0x9c
created by github.com/ThreeDotsLabs/watermill/pubsub/gochannel.(*GoChannel).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/pubsub/gochannel/pubsub.go:197 +0x48c

goroutine 126 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:52 +0x70
created by github.com/ThreeDotsLabs/watermill/message.(*messageTransformSubscriberDecorator).Subscribe in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/decorator.go:51 +0xcc

goroutine 127 [chan receive]:
github.com/ThreeDotsLabs/watermill/message.(*handler).run(0x14000e0dc70, {0x103737b20?, 0x14000dcaf00}, {0x14001115b00, 0x2, 0x7?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:615 +0x278
github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers.func1()
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:438 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*Router).RunHandlers in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:435 +0x4d0

goroutine 128 [semacquire, 7 minutes]:
sync.runtime_Semacquire(0x0?)
    /usr/local/go/src/runtime/sema.go:62 +0x2c
sync.(*WaitGroup).Wait(0x14000c13610)
    /usr/local/go/src/sync/waitgroup.go:116 +0x74
github.com/ThreeDotsLabs/watermill/message.(*Router).closeWhenAllHandlersStopped(0x140007cd500, {0x103737b20, 0x14000f41ea0})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:473 +0x148
created by github.com/ThreeDotsLabs/watermill/message.(*Router).Run in goroutine 11
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:372 +0x238

goroutine 40 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0d520, {0x103737b20?, 0x14000dca0a0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 58
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 130 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e440d0, {0x103737b20?, 0x14000f41f40?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 52
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 146 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0dad0, {0x103737b20?, 0x14000dca280?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 67
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 12 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0cc30, {0x103737b20?, 0x14000dca000?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 55
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 27 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0d930, {0x103737b20?, 0x14000dca1e0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 64
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 178 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0dba0, {0x103737b20?, 0x14000dca5a0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 82
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 129 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0c8f0, {0x103737b20?, 0x14000dca6e0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 88
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 194 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0d110, {0x103737b20?, 0x14000dca640?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 85
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 210 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e44000, {0x103737b20?, 0x14000dca140?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 61
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 179 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0d6c0, {0x103737b20?, 0x14000dca820?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 94
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 211 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0cdd0, {0x103737b20?, 0x14000dcabe0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 112
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 147 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e44340, {0x103737b20?, 0x14000dcaa00?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 103
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 148 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0dd40, {0x103737b20?, 0x14000dcac80?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 115
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 131 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0d2b0, {0x103737b20?, 0x14000dca3c0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 73
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 41 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0d860, {0x103737b20?, 0x14000dca460?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 76
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 28 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0d450, {0x103737b20?, 0x14000dca780?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 91
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 163 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0da00, {0x103737b20?, 0x14000dca500?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 79
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 226 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0cd00, {0x103737b20?, 0x14000dcab40?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 109
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 195 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e441a0, {0x103737b20?, 0x14000dcad20?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 118
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 180 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0cea0, {0x103737b20?, 0x14000dca8c0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 97
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 132 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0cf70, {0x103737b20?, 0x14000dcae60?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 124
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 13 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0d5f0, {0x103737b20?, 0x14000dca960?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 100
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 212 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0ca90, {0x103737b20?, 0x14000dcaaa0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 106
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 149 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e44410, {0x103737b20?, 0x14000dcadc0?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 121
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 29 [select, 7 minutes]:
github.com/ThreeDotsLabs/watermill/message.(*handler).handleClose(0x14000e0dc70, {0x103737b20?, 0x14000dcaf00?})
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:744 +0x68
created by github.com/ThreeDotsLabs/watermill/message.(*handler).run in goroutine 127
    /root/go/pkg/mod/github.com/!three!dots!labs/watermill@v1.3.3/message/router.go:613 +0x248

goroutine 30 [IO wait]:
internal/poll.runtime_pollWait(0x12c57d260, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x14000e04880?, 0x0?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0x14000e04880)
    /usr/local/go/src/internal/poll/fd_unix.go:611 +0x250
net.(*netFD).accept(0x14000e04880)
    /usr/local/go/src/net/fd_unix.go:172 +0x28
net.(*TCPListener).accept(0x1400108abc0)
    /usr/local/go/src/net/tcpsock_posix.go:152 +0x28
net.(*TCPListener).Accept(0x1400108abc0)
    /usr/local/go/src/net/tcpsock.go:315 +0x2c
net/http.(*Server).Serve(0x14000eda000, {0x103732ff0, 0x1400108abc0})
    /usr/local/go/src/net/http/server.go:3056 +0x2b8
net/http.(*Server).ListenAndServe(0x14000eda000)
    /usr/local/go/src/net/http/server.go:2985 +0x84
github.com/turbot/flowpipe/internal/service/api.(*APIService).Start.func3()
    /go/src/flowpipe/internal/service/api/index.go:243 +0x2c
created by github.com/turbot/flowpipe/internal/service/api.(*APIService).Start in goroutine 1
    /go/src/flowpipe/internal/service/api/index.go:242 +0xb90

goroutine 181 [syscall, 7 minutes]:
os/signal.signal_recv()
    /usr/local/go/src/runtime/sigqueue.go:149 +0x2c
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:23 +0x1c
created by os/signal.Notify.func1.1 in goroutine 1
    /usr/local/go/src/os/signal/signal.go:151 +0x28

goroutine 182 [chan receive, 7 minutes]:
github.com/turbot/flowpipe/internal/service/manager.(*Manager).InterruptHandler.func1()
    /go/src/flowpipe/internal/service/manager/manager.go:465 +0x38
created by github.com/turbot/flowpipe/internal/service/manager.(*Manager).InterruptHandler in goroutine 1
    /go/src/flowpipe/internal/service/manager/manager.go:464 +0xf8

goroutine 46311 [select]:
net/http.(*persistConn).writeLoop(0x14002db9e60)
    /usr/local/go/src/net/http/transport.go:2421 +0x9c
created by net/http.(*Transport).dialConn in goroutine 46407
    /usr/local/go/src/net/http/transport.go:1777 +0x118c

goroutine 47274 [select]:
github.com/turbot/flowpipe/internal/container.(*Container).Run(0x140012b97a0)
    /go/src/flowpipe/internal/container/container.go:358 +0xcbc
github.com/turbot/flowpipe/internal/primitive.(*Container).Run(0x103737ae8?, {0x103737ae8, 0x140034d31d0}, 0x1?)
    /go/src/flowpipe/internal/primitive/container.go:309 +0xb20
github.com/turbot/flowpipe/internal/es/command.StepStartHandler.Handle.func1({0x103737ae8, 0x140034d31d0}, {0x103470840?, 0x14002a27030?}, {{0x103721dc0?, 0x1400073f200?}})
    /go/src/flowpipe/internal/es/command/step_start.go:147 +0x940
created by github.com/turbot/flowpipe/internal/es/command.StepStartHandler.Handle in goroutine 47273
    /go/src/flowpipe/internal/es/command/step_start.go:43 +0xec

goroutine 47196 [IO wait]:
internal/poll.runtime_pollWait(0x12c2ad420, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x14002f0a100?, 0x1400263d000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14002f0a100, {0x1400263d000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x14002f0a100, {0x1400263d000?, 0x14000bf4c00?, 0x0?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14000e94070, {0x1400263d000?, 0x140013643c0?, 0x36?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*persistConn).Read(0x14001918000, {0x1400263d000?, 0x140011e4680?, 0x14000bf4c00?})
    /usr/local/go/src/net/http/transport.go:1954 +0x50
bufio.(*Reader).fill(0x14000b662a0)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).ReadSlice(0x14000b662a0, 0xa?)
    /usr/local/go/src/bufio/bufio.go:379 +0x30
net/http/internal.readChunkLine(0x1400172b3f8?)
    /usr/local/go/src/net/http/internal/chunked.go:156 +0x20
net/http/internal.(*chunkedReader).beginChunk(0x14003247a40)
    /usr/local/go/src/net/http/internal/chunked.go:49 +0x28
net/http/internal.(*chunkedReader).Read(0x14003247a40, {0x14003342600?, 0x1400172b4a8?, 0x10057ea4c?})
    /usr/local/go/src/net/http/internal/chunked.go:125 +0x12c
net/http.(*body).readLocked(0x14001571380, {0x14003342600?, 0x140011e49c0?, 0x200000003?})
    /usr/local/go/src/net/http/transfer.go:839 +0x3c
net/http.(*body).Read(0x1400172b568?, {0x14003342600?, 0x1400172b568?, 0x100551010?})
    /usr/local/go/src/net/http/transfer.go:831 +0x128
net/http.(*bodyEOFSignal).Read(0x140015713c0, {0x14003342600, 0x200, 0x200})
    /usr/local/go/src/net/http/transport.go:2810 +0x16c
io.(*teeReader).Read(0x1400330b080, {0x14003342600, 0x1400172b698?, 0x200})
    /usr/local/go/src/io/io.go:620 +0x38
encoding/json.(*Decoder).refill(0x14002260140)
    /usr/local/go/src/encoding/json/stream.go:165 +0x164
encoding/json.(*Decoder).readValue(0x14002260140)
    /usr/local/go/src/encoding/json/stream.go:140 +0x88
encoding/json.(*Decoder).Decode(0x14002260140, {0x10333fe40, 0x14003c4b280})
    /usr/local/go/src/encoding/json/stream.go:63 +0x5c
github.com/docker/docker/client.(*Client).ContainerWait.func1()
    /root/go/pkg/mod/github.com/docker/docker@v24.0.7+incompatible/client/container_wait.go:60 +0x190
created by github.com/docker/docker/client.(*Client).ContainerWait in goroutine 47274
    /root/go/pkg/mod/github.com/docker/docker@v24.0.7+incompatible/client/container_wait.go:52 +0x284

goroutine 46918 [IO wait]:
internal/poll.runtime_pollWait(0x12c57d168, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x140023d8380?, 0x14000a6a000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140023d8380, {0x14000a6a000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x140023d8380, {0x14000a6a000?, 0x1400209da58?, 0x10062ce7c?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x1400012c050, {0x14000a6a000?, 0x14027910833?, 0x14000df80f8?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*connReader).Read(0x14000df80f0, {0x14000a6a000, 0x1000, 0x1000})
    /usr/local/go/src/net/http/server.go:791 +0x224
bufio.(*Reader).fill(0x14000b992c0)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).Peek(0x14000b992c0, 0x4)
    /usr/local/go/src/bufio/bufio.go:151 +0x60
net/http.(*conn).serve(0x140021f8090, {0x103737ae8, 0x14000ed3800})
    /usr/local/go/src/net/http/server.go:2044 +0x64c
created by net/http.(*Server).Serve in goroutine 30
    /usr/local/go/src/net/http/server.go:3086 +0x4cc

goroutine 45785 [IO wait]:
internal/poll.runtime_pollWait(0x12c57cb98, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x14002619200?, 0x14002e44000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14002619200, {0x14002e44000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x14002619200, {0x14002e44000?, 0x0?, 0x14000c40ea0?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14001598338, {0x14002e44000?, 0x0?, 0x0?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*persistConn).Read(0x140026dbe60, {0x14002e44000?, 0x1005418d0?, 0x140026dcf60?})
    /usr/local/go/src/net/http/transport.go:1954 +0x50
bufio.(*Reader).fill(0x14000f244e0)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).Peek(0x14000f244e0, 0x1)
    /usr/local/go/src/bufio/bufio.go:151 +0x60
net/http.(*persistConn).readLoop(0x140026dbe60)
    /usr/local/go/src/net/http/transport.go:2118 +0x14c
created by net/http.(*Transport).dialConn in goroutine 45784
    /usr/local/go/src/net/http/transport.go:1776 +0x1144

goroutine 47410 [IO wait]:
internal/poll.runtime_pollWait(0x12c2ad610, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x14002883400?, 0x14000ffc000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14002883400, {0x14000ffc000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x14002883400, {0x14000ffc000?, 0x14001deda58?, 0x10062ce7c?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14002340268, {0x14000ffc000?, 0x1402791c40e?, 0x14003325508?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*connReader).Read(0x14003325500, {0x14000ffc000, 0x1000, 0x1000})
    /usr/local/go/src/net/http/server.go:791 +0x224
bufio.(*Reader).fill(0x14000b47260)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).Peek(0x14000b47260, 0x4)
    /usr/local/go/src/bufio/bufio.go:151 +0x60
net/http.(*conn).serve(0x140008e5b00, {0x103737ae8, 0x14000ed3800})
    /usr/local/go/src/net/http/server.go:2044 +0x64c
created by net/http.(*Server).Serve in goroutine 30
    /usr/local/go/src/net/http/server.go:3086 +0x4cc

goroutine 42910 [select]:
net/http.(*persistConn).readLoop(0x14001918000)
    /usr/local/go/src/net/http/transport.go:2238 +0xb7c
created by net/http.(*Transport).dialConn in goroutine 42909
    /usr/local/go/src/net/http/transport.go:1776 +0x1144

goroutine 47563 [runnable]:
net/http.(*Client).makeHeadersCopier(0x14000c5c0c0, 0x14001865900)
    /usr/local/go/src/net/http/client.go:764 +0xec
net/http.(*Client).do(0x14000c5c0c0, 0x14001865900)
    /usr/local/go/src/net/http/client.go:611 +0x1c0
net/http.(*Client).Do(...)
    /usr/local/go/src/net/http/client.go:590
github.com/docker/docker/client.(*Client).doRequest(0x14000f30500, {0x103737ab0, 0x1046767c0}, 0x14001865800)
    /root/go/pkg/mod/github.com/docker/docker@v24.0.7+incompatible/client/request.go:135 +0x108
github.com/docker/docker/client.(*Client).sendRequest(0x0?, {0x103737ab0, 0x1046767c0}, {0x102a9f090, 0x4}, {0x14001f20c00?, 0x6?}, 0x2?, {0x0, 0x0}, ...)
    /root/go/pkg/mod/github.com/docker/docker@v24.0.7+incompatible/client/request.go:119 +0x98
github.com/docker/docker/client.(*Client).post(0x1016a3184?, {0x103737ab0, 0x1046767c0}, {0x14001f20c00, 0x52}, 0x102aa9855?, {0x0?, 0x0?}, 0x0?)
    /root/go/pkg/mod/github.com/docker/docker@v24.0.7+incompatible/client/request.go:45 +0xc8
github.com/docker/docker/client.(*Client).ContainerStart(0x10343ce20?, {0x103737ab0, 0x1046767c0}, {0x14003545840, 0x40}, {{0x0?, 0x0?}, {0x0?, 0x0?}})
    /root/go/pkg/mod/github.com/docker/docker@v24.0.7+incompatible/client/container_start.go:20 +0x208
github.com/turbot/flowpipe/internal/container.(*Container).Run(0x140012b97a0)
    /go/src/flowpipe/internal/container/container.go:344 +0xa44
github.com/turbot/flowpipe/internal/primitive.(*Container).Run(0x103737ae8?, {0x103737ae8, 0x14003438c90}, 0x1?)
    /go/src/flowpipe/internal/primitive/container.go:309 +0xb20
github.com/turbot/flowpipe/internal/es/command.StepStartHandler.Handle.func1({0x103737ae8, 0x14003438c90}, {0x103470840?, 0x1400332d180?}, {{0x103721dc0?, 0x1400073f200?}})
    /go/src/flowpipe/internal/es/command/step_start.go:147 +0x940
created by github.com/turbot/flowpipe/internal/es/command.StepStartHandler.Handle in goroutine 47562
    /go/src/flowpipe/internal/es/command/step_start.go:43 +0xec

goroutine 45786 [select]:
net/http.(*persistConn).writeLoop(0x140026dbe60)
    /usr/local/go/src/net/http/transport.go:2421 +0x9c
created by net/http.(*Transport).dialConn in goroutine 45784
    /usr/local/go/src/net/http/transport.go:1777 +0x118c

goroutine 42911 [select]:
net/http.(*persistConn).writeLoop(0x14001918000)
    /usr/local/go/src/net/http/transport.go:2421 +0x9c
created by net/http.(*Transport).dialConn in goroutine 42909
    /usr/local/go/src/net/http/transport.go:1777 +0x118c

goroutine 47194 [IO wait]:
internal/poll.runtime_pollWait(0x12c57c9a8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x14002421080?, 0x140013bc000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14002421080, {0x140013bc000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x14002421080, {0x140013bc000?, 0x140030fba58?, 0x10062ce7c?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x1400180e370, {0x140013bc000?, 0x14026f2ac2e?, 0x14003247028?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*connReader).Read(0x14003247020, {0x140013bc000, 0x1000, 0x1000})
    /usr/local/go/src/net/http/server.go:791 +0x224
bufio.(*Reader).fill(0x14000b2ab40)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).Peek(0x14000b2ab40, 0x4)
    /usr/local/go/src/bufio/bufio.go:151 +0x60
net/http.(*conn).serve(0x1400332b8c0, {0x103737ae8, 0x14000ed3800})
    /usr/local/go/src/net/http/server.go:2044 +0x64c
created by net/http.(*Server).Serve in goroutine 30
    /usr/local/go/src/net/http/server.go:3086 +0x4cc

goroutine 47153 [IO wait]:
internal/poll.runtime_pollWait(0x12c57cf78, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x14002883380?, 0x14002d0b000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14002883380, {0x14002d0b000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x14002883380, {0x14002d0b000?, 0x140030fba58?, 0x10062ce7c?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x14002340260, {0x14002d0b000?, 0x1402791cd4f?, 0x140033258c8?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*connReader).Read(0x140033258c0, {0x14002d0b000, 0x1000, 0x1000})
    /usr/local/go/src/net/http/server.go:791 +0x224
bufio.(*Reader).fill(0x1400128c000)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).Peek(0x1400128c000, 0x4)
    /usr/local/go/src/bufio/bufio.go:151 +0x60
net/http.(*conn).serve(0x140008e5a70, {0x103737ae8, 0x14000ed3800})
    /usr/local/go/src/net/http/server.go:2044 +0x64c
created by net/http.(*Server).Serve in goroutine 30
    /usr/local/go/src/net/http/server.go:3086 +0x4cc

goroutine 46917 [IO wait]:
internal/poll.runtime_pollWait(0x12c57c5c8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x140023d8300?, 0x1400287d000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x140023d8300, {0x1400287d000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x140023d8300, {0x1400287d000?, 0x14002e53a58?, 0x10062ce7c?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x1400012c048, {0x1400287d000?, 0x1402791102a?, 0x140016360f8?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*connReader).Read(0x140016360f0, {0x1400287d000, 0x1000, 0x1000})
    /usr/local/go/src/net/http/server.go:791 +0x224
bufio.(*Reader).fill(0x14000a68000)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).Peek(0x14000a68000, 0x4)
    /usr/local/go/src/bufio/bufio.go:151 +0x60
net/http.(*conn).serve(0x140021f8000, {0x103737ae8, 0x14000ed3800})
    /usr/local/go/src/net/http/server.go:2044 +0x64c
created by net/http.(*Server).Serve in goroutine 30
    /usr/local/go/src/net/http/server.go:3086 +0x4cc

goroutine 47186 [IO wait]:
internal/poll.runtime_pollWait(0x12c57c3d8, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x14002421000?, 0x1400261e000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14002421000, {0x1400261e000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x14002421000, {0x1400261e000?, 0x1400241fa58?, 0x10062ce7c?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x1400180e328, {0x1400261e000?, 0x14026f258d3?, 0x14002329f58?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*connReader).Read(0x14002329f50, {0x1400261e000, 0x1000, 0x1000})
    /usr/local/go/src/net/http/server.go:791 +0x224
bufio.(*Reader).fill(0x14000bdc3c0)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).Peek(0x14000bdc3c0, 0x4)
    /usr/local/go/src/bufio/bufio.go:151 +0x60
net/http.(*conn).serve(0x1400332b4d0, {0x103737ae8, 0x14000ed3800})
    /usr/local/go/src/net/http/server.go:2044 +0x64c
created by net/http.(*Server).Serve in goroutine 30
    /usr/local/go/src/net/http/server.go:3086 +0x4cc

goroutine 46310 [IO wait]:
internal/poll.runtime_pollWait(0x12c57c6c0, 0x72)
    /usr/local/go/src/runtime/netpoll.go:343 +0xa0
internal/poll.(*pollDesc).wait(0x14002883280?, 0x140038e2000?, 0x0)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x28
internal/poll.(*pollDesc).waitRead(...)
    /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0x14002883280, {0x140038e2000, 0x1000, 0x1000})
    /usr/local/go/src/internal/poll/fd_unix.go:164 +0x200
net.(*netFD).Read(0x14002883280, {0x140038e2000?, 0x0?, 0x14000bae340?})
    /usr/local/go/src/net/fd_posix.go:55 +0x28
net.(*conn).Read(0x140008920f0, {0x140038e2000?, 0x0?, 0x0?})
    /usr/local/go/src/net/net.go:179 +0x34
net/http.(*persistConn).Read(0x14002db9e60, {0x140038e2000?, 0x1005418d0?, 0x140026dd7a0?})
    /usr/local/go/src/net/http/transport.go:1954 +0x50
bufio.(*Reader).fill(0x14000b4a660)
    /usr/local/go/src/bufio/bufio.go:113 +0xf8
bufio.(*Reader).Peek(0x14000b4a660, 0x1)
    /usr/local/go/src/bufio/bufio.go:151 +0x60
net/http.(*persistConn).readLoop(0x14002db9e60)
    /usr/local/go/src/net/http/transport.go:2118 +0x14c
created by net/http.(*Transport).dialConn in goroutine 46407
    /usr/local/go/src/net/http/transport.go:1776 +0x1144
vhadianto commented 9 months ago

Duplicate of https://github.com/turbot/flowpipe/issues/571