search

cshum / imagor

Fast, secure image processing server and Go library, using libvips
Apache License 2.0
3.36k stars 129 forks source link

Imagor crashes sometimes #162

Closed cocorossello closed 2 years ago

cocorossello commented 2 years ago

Hi, from time to time (every day or every to days) imagor crashes and restarts.

I cannot reproduce the error, I can see this traces in the log:

{"log":"fatal error: unexpected signal during runtime execution\n","stream":"stderr","time":"2022-09-07T10:26:21.691272038Z"}
{"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x125 pc=0x7f749a91b3a0]\n","stream":"stderr","time":"2022-09-07T10:26:21.693398049Z"}
{"log":"\n","stream":"stderr","time":"2022-09-07T10:26:21.69341131Z"}
{"log":"runtime stack:\n","stream":"stderr","time":"2022-09-07T10:26:21.693413756Z"}
{"log":"runtime.throw({0xf780d7?, 0x3?})\n","stream":"stderr","time":"2022-09-07T10:26:21.693730377Z"}
{"log":"\u0009/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f7332c75a60 sp=0x7f7332c75a30 pc=0x43b77d\n","stream":"stderr","time":"2022-09-07T10:26:21.693734236Z"}
{"log":"runtime.sigpanic()\n","stream":"stderr","time":"2022-09-07T10:26:21.693737062Z"}
{"log":"\u0009/usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x7f7332c75ab0 sp=0x7f7332c75a60 pc=0x451689\n","stream":"stderr","time":"2022-09-07T10:26:21.693739123Z"}
{"log":"\n","stream":"stderr","time":"2022-09-07T10:26:21.693741245Z"}
{"log":"goroutine 5715652 [syscall]:\n","stream":"stderr","time":"2022-09-07T10:26:21.693743343Z"}
{"log":"runtime.cgocall(0xd0e560, 0xc000ff4f40)\n","stream":"stderr","time":"2022-09-07T10:26:21.693754002Z"}
{"log":"\u0009/usr/local/go/src/runtime/cgocall.go:158 +0x5c fp=0xc000ff4f18 sp=0xc000ff4ee0 pc=0x4096dc\n","stream":"stderr","time":"2022-09-07T10:26:21.693757499Z"}
{"log":"github.com/cshum/imagor/vips._Cfunc_thumbnail_source_with_option(0x7f7434207270, 0xc0013bc010, 0x1f4, 0x1388, 0x0, 0x2, 0x7f742d60d780)\n","stream":"stderr","time":"2022-09-07T10:26:21.694102171Z"}
{"log":"\u0009_cgo_gotypes.go:1402 +0x4c fp=0xc000ff4f40 sp=0xc000ff4f18 pc=0xcfea0c\n","stream":"stderr","time":"2022-09-07T10:26:21.694106296Z"}
{"log":"github.com/cshum/imagor/vips.vipsThumbnailFromSource.func2(0xc00184e020?, 0x9?, 0x1f4, 0x1388, 0x0, 0x2, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694108351Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/vips.go:99 +0xbc fp=0xc000ff4f98 sp=0xc000ff4f40 pc=0xd06e3c\n","stream":"stderr","time":"2022-09-07T10:26:21.694110387Z"}
{"log":"github.com/cshum/imagor/vips.vipsThumbnailFromSource(0x7f7496d85d28?, 0x8?, 0xc001d1e400?, 0xc0013bc008?, 0x0?, 0xc000ff50b0)\n","stream":"stderr","time":"2022-09-07T10:26:21.694112449Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/vips.go:99 +0x155 fp=0xc000ff5058 sp=0xc000ff4f98 pc=0xd06bf5\n","stream":"stderr","time":"2022-09-07T10:26:21.694117436Z"}
{"log":"github.com/cshum/imagor/vips.LoadThumbnailFromSource(0x1225d38?, 0xc000960420?, 0xc000ff51d8?, 0x41d46b?, 0x7f746f2b4100?, 0xc00042e8c0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.69413578Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/image.go:137 +0x86 fp=0xc000ff5180 sp=0xc000ff5058 pc=0xd019e6\n","stream":"stderr","time":"2022-09-07T10:26:21.694139392Z"}
{"log":"github.com/cshum/imagor/vips.(*Source).LoadThumbnail(...)\n","stream":"stderr","time":"2022-09-07T10:26:21.694503284Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/source.go:61\n","stream":"stderr","time":"2022-09-07T10:26:21.694507062Z"}
{"log":"github.com/cshum/imagor/vips.newThumbnailFromBlob({0x1225d38, 0xc000960420}, 0xc000988200, 0x203000?, 0xc000960420?, 0xc0009603f0?, 0xc000753860?, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694509083Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/processor.go:174 +0x1a5 fp=0xc000ff5208 sp=0xc000ff5180 pc=0xcfa3c5\n","stream":"stderr","time":"2022-09-07T10:26:21.69451134Z"}
{"log":"github.com/cshum/imagor/vips.(*Processor).NewThumbnail(0xdf86c0?, {0x1225d38, 0xc000960420}, 0xc000988200, 0xe12080?, 0xc0009603f0?, 0x0, 0x2, 0x1)\n","stream":"stderr","time":"2022-09-07T10:26:21.694513437Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/processor.go:219 +0x44f fp=0xc000ff5350 sp=0xc000ff5208 pc=0xcfa88f\n","stream":"stderr","time":"2022-09-07T10:26:21.694515589Z"}
{"log":"github.com/cshum/imagor/vips.(*Processor).Process(_, {_, _}, _, {0x0, {0xc001b180a8, 0x91}, {0xc002050300, 0x51}, 0x1, ...}, ...)\n","stream":"stderr","time":"2022-09-07T10:26:21.694554902Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/process.go:119 +0x32f fp=0xc000ff5738 sp=0xc000ff5350 pc=0xcf4c2f\n","stream":"stderr","time":"2022-09-07T10:26:21.694570019Z"}
{"log":"github.com/cshum/imagor.(*Imagor).Do.func2({0x1225d38, 0xc0022542d0}, 0xc001e02140)\n","stream":"stderr","time":"2022-09-07T10:26:21.694573673Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/imagor.go:306 +0x10a3 fp=0xc000ff5e58 sp=0xc000ff5738 pc=0x7581a3\n","stream":"stderr","time":"2022-09-07T10:26:21.694576598Z"}
{"log":"github.com/cshum/imagor.(*Imagor).suppress.func2()\n","stream":"stderr","time":"2022-09-07T10:26:21.694579764Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/imagor.go:496 +0xc7 fp=0xc000ff5f00 sp=0xc000ff5e58 pc=0x75b207\n","stream":"stderr","time":"2022-09-07T10:26:21.69458286Z"}
{"log":"golang.org/x/sync/singleflight.(*Group).doCall.func2(0xc000ff5f56, 0xc001152420, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694586323Z"}
{"log":"\u0009/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/singleflight/singleflight.go:193 +0x6f fp=0xc000ff5f38 sp=0xc000ff5f00 pc=0x75158f\n","stream":"stderr","time":"2022-09-07T10:26:21.694596584Z"}
{"log":"golang.org/x/sync/singleflight.(*Group).doCall(0xc000636480?, 0xc0011eefd0?, {0xc001b180a8?, 0xc0011ba060?}, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694603798Z"}
{"log":"\u0009/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/singleflight/singleflight.go:195 +0xa5 fp=0xc000ff5fa8 sp=0xc000ff5f38 pc=0x7514a5\n","stream":"stderr","time":"2022-09-07T10:26:21.694605955Z"}
{"log":"golang.org/x/sync/singleflight.(*Group).DoChan.func1()\n","stream":"stderr","time":"2022-09-07T10:26:21.694613684Z"}
{"log":"\u0009/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/singleflight/singleflight.go:133 +0x36 fp=0xc000ff5fe0 sp=0xc000ff5fa8 pc=0x7513d6\n","stream":"stderr","time":"2022-09-07T10:26:21.694615979Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-07T10:26:21.694618162Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000ff5fe8 sp=0xc000ff5fe0 pc=0x46cb81\n","stream":"stderr","time":"2022-09-07T10:26:21.694620142Z"}
{"log":"created by golang.org/x/sync/singleflight.(*Group).DoChan\n","stream":"stderr","time":"2022-09-07T10:26:21.69462219Z"}
{"log":"\u0009/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/singleflight/singleflight.go:133 +0x315\n","stream":"stderr","time":"2022-09-07T10:26:21.694624208Z"}
{"log":"\n","stream":"stderr","time":"2022-09-07T10:26:21.694626248Z"}
{"log":"goroutine 1 [chan receive, 861 minutes]:\n","stream":"stderr","time":"2022-09-07T10:26:21.694628117Z"}
{"log":"runtime.gopark(0x30cc80c?, 0x1890060?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694631229Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000a86d90 sp=0xc000a86d70 pc=0x43e456\n","stream":"stderr","time":"2022-09-07T10:26:21.694633251Z"}
{"log":"runtime.chanrecv(0xc0000d84e0, 0x0, 0x1)\n","stream":"stderr","time":"2022-09-07T10:26:21.694635648Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc000a86e20 sp=0xc000a86d90 pc=0x40c6db\n","stream":"stderr","time":"2022-09-07T10:26:21.694638531Z"}
{"log":"runtime.chanrecv1(0xc00019c420?, 0xf4ad58?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694642103Z"}
{"log":"\u0009/usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc000a86e48 sp=0xc000a86e20 pc=0x40c1d8\n","stream":"stderr","time":"2022-09-07T10:26:21.694645043Z"}
{"log":"github.com/cshum/imagor/server.(*Server).RunContext(0xc00019c420, {0x12264e0, 0xc0001d0a40})\n","stream":"stderr","time":"2022-09-07T10:26:21.69464934Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/server/server.go:83 +0x16b fp=0xc000a86ed0 sp=0xc000a86e48 pc=0x76548b\n","stream":"stderr","time":"2022-09-07T10:26:21.694652347Z"}
{"log":"github.com/cshum/imagor/server.(*Server).Run(0xc000036240?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694654428Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/server/server.go:71 +0xa5 fp=0xc000a86f28 sp=0xc000a86ed0 pc=0x7652c5\n","stream":"stderr","time":"2022-09-07T10:26:21.694656349Z"}
{"log":"main.main()\n","stream":"stderr","time":"2022-09-07T10:26:21.694658373Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/cmd/imagor/main.go:19 +0xb7 fp=0xc000a86f80 sp=0xc000a86f28 pc=0xd0d6b7\n","stream":"stderr","time":"2022-09-07T10:26:21.694660416Z"}
{"log":"runtime.main()\n","stream":"stderr","time":"2022-09-07T10:26:21.694663029Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:250 +0x212 fp=0xc000a86fe0 sp=0xc000a86f80 pc=0x43e092\n","stream":"stderr","time":"2022-09-07T10:26:21.69466493Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-07T10:26:21.694667047Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000a86fe8 sp=0xc000a86fe0 pc=0x46cb81\n","stream":"stderr","time":"2022-09-07T10:26:21.694669512Z"}
{"log":"\n","stream":"stderr","time":"2022-09-07T10:26:21.694671574Z"}
{"log":"goroutine 2 [force gc (idle), 861 minutes]:\n","stream":"stderr","time":"2022-09-07T10:26:21.694673803Z"}
{"log":"runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.6946842Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a0fb0 sp=0xc0000a0f90 pc=0x43e456\n","stream":"stderr","time":"2022-09-07T10:26:21.694692563Z"}
{"log":"runtime.goparkunlock(...)\n","stream":"stderr","time":"2022-09-07T10:26:21.69469481Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:369\n","stream":"stderr","time":"2022-09-07T10:26:21.694704646Z"}
{"log":"runtime.forcegchelper()\n","stream":"stderr","time":"2022-09-07T10:26:21.694706723Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:302 +0xad fp=0xc0000a0fe0 sp=0xc0000a0fb0 pc=0x43e2ed\n","stream":"stderr","time":"2022-09-07T10:26:21.694711758Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-07T10:26:21.694715028Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a0fe8 sp=0xc0000a0fe0 pc=0x46cb81\n","stream":"stderr","time":"2022-09-07T10:26:21.694718055Z"}
{"log":"created by runtime.init.6\n","stream":"stderr","time":"2022-09-07T10:26:21.694721206Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:290 +0x25\n","stream":"stderr","time":"2022-09-07T10:26:21.694725907Z"}
{"log":"\n","stream":"stderr","time":"2022-09-07T10:26:21.694728948Z"}
{"log":"goroutine 3 [GC sweep wait]:\n","stream":"stderr","time":"2022-09-07T10:26:21.694730856Z"}
{"log":"runtime.gopark(0x188fa01?, 0x0?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694732793Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a1790 sp=0xc0000a1770 pc=0x43e456\n","stream":"stderr","time":"2022-09-07T10:26:21.694735235Z"}
{"log":"runtime.goparkunlock(...)\n","stream":"stderr","time":"2022-09-07T10:26:21.694759617Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:369\n","stream":"stderr","time":"2022-09-07T10:26:21.694767234Z"}
{"log":"runtime.bgsweep(0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694770359Z"}
{"log":"\u0009/usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc0000a17c8 sp=0xc0000a1790 pc=0x42a897\n","stream":"stderr","time":"2022-09-07T10:26:21.694773261Z"}
{"log":"runtime.gcenable.func1()\n","stream":"stderr","time":"2022-09-07T10:26:21.694776628Z"}
{"log":"\u0009/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000a17e0 sp=0xc0000a17c8 pc=0x41f506\n","stream":"stderr","time":"2022-09-07T10:26:21.694779532Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-07T10:26:21.694782534Z"}
{"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a17e8 sp=0xc0000a17e0 pc=0x46cb81\n","stream":"stderr","time":"2022-09-07T10:26:21.694785472Z"}
{"log":"created by runtime.gcenable\n","stream":"stderr","time":"2022-09-07T10:26:21.694788725Z"}
{"log":"\u0009/usr/local/go/src/runtime/mgc.go:178 +0x6b\n","stream":"stderr","time":"2022-09-07T10:26:21.694792508Z"}
{"log":"\n","stream":"stderr","time":"2022-09-07T10:26:21.694795503Z"}
{"log":"goroutine 4 [GC scavenge wait]:\n","stream":"stderr","time":"2022-09-07T10:26:21.694798322Z"}
{"log":"runtime.gopark(0x1c9d63eb19ee6f?, 0x201f13?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694801343Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a1f70 sp=0xc0000a1f50 pc=0x43e456\n","stream":"stderr","time":"2022-09-07T10:26:21.694804511Z"}
{"log":"runtime.goparkunlock(...)\n","stream":"stderr","time":"2022-09-07T10:26:21.694807658Z"}
{"log":"\u0009/usr/local/go/src/runtime/proc.go:369\n","stream":"stderr","time":"2022-09-07T10:26:21.694810682Z"}
{"log":"runtime.(*scavengerState).park(0x18900e0)\n","stream":"stderr","time":"2022-09-07T10:26:21.694813669Z"}
{"log":"\u0009/usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc0000a1fa0 sp=0xc0000a1f70 pc=0x4288f3\n","stream":"stderr","time":"2022-09-07T10:26:21.694816562Z"}
{"log":"runtime.bgscavenge(0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694819748Z"}
{"log":"\u0009/usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc0000a1fc8 sp=0xc0000a1fa0 pc=0x428ee5\n","stream":"stderr","time":"2022-09-07T10:26:21.694823288Z"}
{"log":"runtime.gcenable.func2()\n","stream":"stderr","time":"2022-09-07T10:26:21.694827605Z"}
{"log":"\u0009/usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc0000a1fe0 sp=0xc0000a1fc8 pc=0x41f4a6\n","stream":"stderr","time":"2022-09-07T10:26:21.694830585Z"}
{"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-07T10:26:21.69483364Z"}
...

This is with 1.0.3. It was also happening in 0.9.7 (and others I guess). I think that it happens a bit more frequently in this version.

I'm not sure about it, but I think that related to this crashes I get the result storage populated with truncated images, like this one: https://cdn4.travelconline.com/unsafe/fit-in/800x0/filters:quality(75):format(jpg)/https%3A%2F%2Fi.travelapi.com%2Fhotels%2F22000000%2F21540000%2F21537500%2F21537420%2F8a95c8cf_z.jpg

(it looks fine in the browser but it's truncated and some utilities will crash when they read this image)

If I delete the file from the result storage the image works fine again

cshum commented 2 years ago

What kind of image types are you processing with imagor? Apart from JPEG?

Looking at [signal SIGSEGV: segmentation violation code=0x1 addr=0x125 pc=0x7f749a91b3a0], a speculation is some operations for some image types may not be as concurrency safe. But these are just pure guess.

cocorossello commented 2 years ago

We are using it to process supplier's images, so it can be anything. It should be jpg, png, gif and svg. Looking at the errors at the time of the crash, in the nginx access logs, I don't see anything really relevant. Some 500 errors with images with hosts that don't exist like

ttps://cdn4.travelconline.com/unsafe/fit-in/500x0/filters:quality(75):format(webp)/https%3A%2F%2Fveturis.azureedge.nethttps%3A%2F%2Fveturis.azureedge.net%2Fhotel-Wake-In-Aveiro-Guesthouse-F147429_1.jpg

And some 406 errors with jpg images that apparently look fine, but their CDN maybe was down at the time:

http://cdn4.travelconline.com/unsafe/fit-in/500x0/filters:quality(75):format(webp)/http%3A%2F%2Fwww.ixpira.it%2Fhandlers%2FImageRequest.ashx%3Fw%3D360%26h%3D270%26coded%3D1%26path%3D68747470733a2f2f63646e2e776f726c646f74612e6e65742f742f31303234783736382f636f6e74656e742f39362f34382f393634383133383366353632633135326663303134326137366435316461626531633962306161652e6a706567%26a%3D12674

cshum commented 2 years ago

Would you try the docker image of master branch, ghcr.io/cshum/imagor:master? I have pushed some tweaks lets see if it gives any difference. Please clear docker cache if you have pulled the image before.

cocorossello commented 2 years ago

I will deploy this tonight and report results, thanks

cocorossello commented 2 years ago

Hi, it crashed again, two times. I'm showing here the logs from the last crash, with imagor 1.1.0

{"log":"fatal error: unexpected signal during runtime execution\n","stream":"stderr","time":"2022-09-09T19:19:36.772132369Z"} {"log":"[signal SIGSEGV: segmentation violation code=0x1 addr=0x19 pc=0x7f330c3de3a0]\n","stream":"stderr","time":"2022-09-09T19:19:36.774259507Z"} {"log":"\n","stream":"stderr","time":"2022-09-09T19:19:36.774280345Z"} {"log":"runtime stack:\n","stream":"stderr","time":"2022-09-09T19:19:36.77428526Z"} {"log":"runtime.throw({0xf93a53?, 0x3?})\n","stream":"stderr","time":"2022-09-09T19:19:36.774288629Z"} {"log":"\u0009/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7f32d7ffba80 sp=0x7f32d7ffba50 pc=0x43b77d\n","stream":"stderr","time":"2022-09-09T19:19:36.774291889Z"} {"log":"runtime.sigpanic()\n","stream":"stderr","time":"2022-09-09T19:19:36.7742962Z"} {"log":"\u0009/usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x7f32d7ffbad0 sp=0x7f32d7ffba80 pc=0x451689\n","stream":"stderr","time":"2022-09-09T19:19:36.774299906Z"} {"log":"\n","stream":"stderr","time":"2022-09-09T19:19:36.774303933Z"} {"log":"goroutine 118460 [syscall]:\n","stream":"stderr","time":"2022-09-09T19:19:36.774307211Z"} {"log":"runtime.cgocall(0xd26da0, 0xc0011bec98)\n","stream":"stderr","time":"2022-09-09T19:19:36.774311072Z"} {"log":"\u0009/usr/local/go/src/runtime/cgocall.go:158 +0x5c fp=0xc0011bec70 sp=0xc0011bec38 pc=0x4096dc\n","stream":"stderr","time":"2022-09-09T19:19:36.774314407Z"} {"log":"github.com/cshum/imagor/vips._Cfunc_thumbnail(0x7f3247f8aa20, 0xc0003c0010, 0x1f4, 0x2328, 0x0, 0x2)\n","stream":"stderr","time":"2022-09-09T19:19:36.77431854Z"} {"log":"\u0009_cgogotypes.go:1348 +0x4c fp=0xc0011bec98 sp=0xc0011bec70 pc=0xd16e8c\n","stream":"stderr","time":"2022-09-09T19:19:36.774322528Z"} {"log":"github.com/cshum/imagor/vips.vipsThumbnailFromFile.func1(0xc000d50230?, 0x68?, 0x1f4, 0x2328, 0x0, 0x2)\n","stream":"stderr","time":"2022-09-09T19:19:36.774326304Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/vips/vips.go:121 +0x8c fp=0xc0011bece0 sp=0xc0011bec98 pc=0xd1fc8c\n","stream":"stderr","time":"2022-09-09T19:19:36.774329656Z"} {"log":"github.com/cshum/imagor/vips.vipsThumbnailFromFile({0xc000640420, 0x5d}, 0x4b80a6d3de690ac8?, 0x3b2d755d6ba5372a?, 0xbe2bb20559d564e9?, 0xb428b49255ad4acb?, 0xc0011bee20)\n","stream":"stderr","time":"2022-09-09T19:19:36.774340561Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/vips/vips.go:121 +0x14d fp=0xc0011beda0 sp=0xc0011bece0 pc=0xd1fa8d\n","stream":"stderr","time":"2022-09-09T19:19:36.774344622Z"} {"log":"github.com/cshum/imagor/vips.LoadThumbnailFromFile({0xc000640420, 0x5d}, 0xc0013a2d00?, 0x10000000075653d?, 0x0?, 0x476de6?, 0x0)\n","stream":"stderr","time":"2022-09-09T19:19:36.774348557Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/vips/image.go:193 +0xef fp=0xc0011beef0 sp=0xc0011beda0 pc=0xd1a98f\n","stream":"stderr","time":"2022-09-09T19:19:36.774351718Z"} {"log":"github.com/cshum/imagor/vips.newThumbnailFromBlob({0x124e8f8, 0xc00001e6f0}, 0xc0009065a0, 0xc0011befd0?, 0x41c92b?, 0x18f1440?, 0x4279a7?, 0x8?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774355138Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/vips/processor.go:166 +0x105 fp=0xc0011bef78 sp=0xc0011beef0 pc=0xd12de5\n","stream":"stderr","time":"2022-09-09T19:19:36.774358623Z"} {"log":"github.com/cshum/imagor/vips.(Processor).NewThumbnail(0xe11720?, {0x124e8f8, 0xc00001e6f0}, 0x4?, 0xe2b320?, 0xc00001e3f0?, 0x0, 0x2, 0x1)\n","stream":"stderr","time":"2022-09-09T19:19:36.774370427Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/vips/processor.go:217 +0x42a fp=0xc0011bf0c8 sp=0xc0011bef78 pc=0xd1332a\n","stream":"stderr","time":"2022-09-09T19:19:36.774374296Z"} {"log":"github.com/cshum/imagor/vips.(Processor).Process(, {, }, _, {0x0, {0xc000128328, 0x8e}, {0xc0004ae230, 0x4e}, 0x1, ...}, ...)\n","stream":"stderr","time":"2022-09-09T19:19:36.77438578Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/vips/process.go:125 +0x3a5 fp=0xc0011bf4a8 sp=0xc0011bf0c8 pc=0xd0d6e5\n","stream":"stderr","time":"2022-09-09T19:19:36.774389701Z"} {"log":"github.com/cshum/imagor.(Imagor).Do.func2({0x124e8f8, 0xc0014a4690}, 0xc00057c190)\n","stream":"stderr","time":"2022-09-09T19:19:36.774393076Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/imagor.go:321 +0x1127 fp=0xc0011bfe58 sp=0xc0011bf4a8 pc=0x759b07\n","stream":"stderr","time":"2022-09-09T19:19:36.774396219Z"} {"log":"github.com/cshum/imagor.(Imagor).suppress.func2()\n","stream":"stderr","time":"2022-09-09T19:19:36.774399496Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/imagor.go:536 +0xc7 fp=0xc0011bff00 sp=0xc0011bfe58 pc=0x75cd07\n","stream":"stderr","time":"2022-09-09T19:19:36.774402627Z"} {"log":"golang.org/x/sync/singleflight.(Group).doCall.func2(0xc0011bff56, 0xc000c1a3c0, 0xc0034aaf00?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774405768Z"} {"log":"\u0009/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/singleflight/singleflight.go:193 +0x6f fp=0xc0011bff38 sp=0xc0011bff00 pc=0x751f4f\n","stream":"stderr","time":"2022-09-09T19:19:36.774409273Z"} {"log":"golang.org/x/sync/singleflight.(Group).doCall(0xc000186900?, 0xc00012e8a0?, {0xc000128328?, 0x75a44a?}, 0xc000e7c780?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774430504Z"} {"log":"\u0009/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/singleflight/singleflight.go:195 +0xa5 fp=0xc0011bffa8 sp=0xc0011bff38 pc=0x751e65\n","stream":"stderr","time":"2022-09-09T19:19:36.77444459Z"} {"log":"golang.org/x/sync/singleflight.(Group).DoChan.func1()\n","stream":"stderr","time":"2022-09-09T19:19:36.774447846Z"} {"log":"\u0009/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/singleflight/singleflight.go:133 +0x36 fp=0xc0011bffe0 sp=0xc0011bffa8 pc=0x751d96\n","stream":"stderr","time":"2022-09-09T19:19:36.774450384Z"} {"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-09T19:19:36.774452847Z"} {"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0011bffe8 sp=0xc0011bffe0 pc=0x46cb81\n","stream":"stderr","time":"2022-09-09T19:19:36.774455241Z"} {"log":"created by golang.org/x/sync/singleflight.(Group).DoChan\n","stream":"stderr","time":"2022-09-09T19:19:36.77445757Z"} {"log":"\u0009/go/pkg/mod/golang.org/x/sync@v0.0.0-20220601150217-0de741cfad7f/singleflight/singleflight.go:133 +0x315\n","stream":"stderr","time":"2022-09-09T19:19:36.774459815Z"} {"log":"\n","stream":"stderr","time":"2022-09-09T19:19:36.774462082Z"} {"log":"goroutine 1 [chan receive, 11 minutes]:\n","stream":"stderr","time":"2022-09-09T19:19:36.774466837Z"} {"log":"runtime.gopark(0x181d4d1?, 0x18c7660?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774469143Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0006f0d90 sp=0xc0006f0d70 pc=0x43e456\n","stream":"stderr","time":"2022-09-09T19:19:36.774471445Z"} {"log":"runtime.chanrecv(0xc0000d8120, 0x0, 0x1)\n","stream":"stderr","time":"2022-09-09T19:19:36.774499807Z"} {"log":"\u0009/usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc0006f0e20 sp=0xc0006f0d90 pc=0x40c6db\n","stream":"stderr","time":"2022-09-09T19:19:36.774518005Z"} {"log":"runtime.chanrecv1(0xc000162160?, 0xf65f60?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774521479Z"} {"log":"\u0009/usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc0006f0e48 sp=0xc0006f0e20 pc=0x40c1d8\n","stream":"stderr","time":"2022-09-09T19:19:36.774523918Z"} {"log":"github.com/cshum/imagor/server.(Server).RunContext(0xc000162160, {0x124f0a0, 0xc0000e54c0})\n","stream":"stderr","time":"2022-09-09T19:19:36.774534708Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/server/server.go:83 +0x16b fp=0xc0006f0ed0 sp=0xc0006f0e48 pc=0x76772b\n","stream":"stderr","time":"2022-09-09T19:19:36.774537348Z"} {"log":"github.com/cshum/imagor/server.(Server).Run(0xc000036240?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774539878Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/server/server.go:71 +0xa5 fp=0xc0006f0f28 sp=0xc0006f0ed0 pc=0x767565\n","stream":"stderr","time":"2022-09-09T19:19:36.7745426Z"} {"log":"main.main()\n","stream":"stderr","time":"2022-09-09T19:19:36.774545992Z"} {"log":"\u0009/go/src/github.com/cshum/imagor/cmd/imagor/main.go:19 +0xb7 fp=0xc0006f0f80 sp=0xc0006f0f28 pc=0xd25fb7\n","stream":"stderr","time":"2022-09-09T19:19:36.774549499Z"} {"log":"runtime.main()\n","stream":"stderr","time":"2022-09-09T19:19:36.774556717Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:250 +0x212 fp=0xc0006f0fe0 sp=0xc0006f0f80 pc=0x43e092\n","stream":"stderr","time":"2022-09-09T19:19:36.774560363Z"} {"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-09T19:19:36.774567256Z"} {"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0006f0fe8 sp=0xc0006f0fe0 pc=0x46cb81\n","stream":"stderr","time":"2022-09-09T19:19:36.774570893Z"} {"log":"\n","stream":"stderr","time":"2022-09-09T19:19:36.7745732Z"} {"log":"goroutine 2 [force gc (idle), 11 minutes]:\n","stream":"stderr","time":"2022-09-09T19:19:36.77457555Z"} {"log":"runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774577804Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a0fb0 sp=0xc0000a0f90 pc=0x43e456\n","stream":"stderr","time":"2022-09-09T19:19:36.774579954Z"} {"log":"runtime.goparkunlock(...)\n","stream":"stderr","time":"2022-09-09T19:19:36.774582201Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:369\n","stream":"stderr","time":"2022-09-09T19:19:36.774584524Z"} {"log":"runtime.forcegchelper()\n","stream":"stderr","time":"2022-09-09T19:19:36.774587132Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:302 +0xad fp=0xc0000a0fe0 sp=0xc0000a0fb0 pc=0x43e2ed\n","stream":"stderr","time":"2022-09-09T19:19:36.774589267Z"} {"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-09T19:19:36.774591545Z"} {"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a0fe8 sp=0xc0000a0fe0 pc=0x46cb81\n","stream":"stderr","time":"2022-09-09T19:19:36.774594803Z"} {"log":"created by runtime.init.6\n","stream":"stderr","time":"2022-09-09T19:19:36.774597088Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:290 +0x25\n","stream":"stderr","time":"2022-09-09T19:19:36.774599208Z"} {"log":"\n","stream":"stderr","time":"2022-09-09T19:19:36.774601378Z"} {"log":"goroutine 3 [GC sweep wait]:\n","stream":"stderr","time":"2022-09-09T19:19:36.774603566Z"} {"log":"runtime.gopark(0x18c7001?, 0x0?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774605724Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a1790 sp=0xc0000a1770 pc=0x43e456\n","stream":"stderr","time":"2022-09-09T19:19:36.774607856Z"} {"log":"runtime.goparkunlock(...)\n","stream":"stderr","time":"2022-09-09T19:19:36.774610137Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:369\n","stream":"stderr","time":"2022-09-09T19:19:36.774613397Z"} {"log":"runtime.bgsweep(0x0?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774615525Z"} {"log":"\u0009/usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc0000a17c8 sp=0xc0000a1790 pc=0x42a897\n","stream":"stderr","time":"2022-09-09T19:19:36.774617646Z"} {"log":"runtime.gcenable.func1()\n","stream":"stderr","time":"2022-09-09T19:19:36.774619889Z"} {"log":"\u0009/usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000a17e0 sp=0xc0000a17c8 pc=0x41f506\n","stream":"stderr","time":"2022-09-09T19:19:36.774623679Z"} {"log":"runtime.goexit()\n","stream":"stderr","time":"2022-09-09T19:19:36.774631025Z"} {"log":"\u0009/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a17e8 sp=0xc0000a17e0 pc=0x46cb81\n","stream":"stderr","time":"2022-09-09T19:19:36.774639431Z"} {"log":"created by runtime.gcenable\n","stream":"stderr","time":"2022-09-09T19:19:36.774643261Z"} {"log":"\u0009/usr/local/go/src/runtime/mgc.go:178 +0x6b\n","stream":"stderr","time":"2022-09-09T19:19:36.774645407Z"} {"log":"\n","stream":"stderr","time":"2022-09-09T19:19:36.774647586Z"} {"log":"goroutine 4 [GC scavenge wait]:\n","stream":"stderr","time":"2022-09-09T19:19:36.774649639Z"} {"log":"runtime.gopark(0x1d57a70692a2fe?, 0x108d45?, 0x0?, 0x0?, 0x0?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774652713Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a1f70 sp=0xc0000a1f50 pc=0x43e456\n","stream":"stderr","time":"2022-09-09T19:19:36.774654964Z"} {"log":"runtime.goparkunlock(...)\n","stream":"stderr","time":"2022-09-09T19:19:36.774663293Z"} {"log":"\u0009/usr/local/go/src/runtime/proc.go:369\n","stream":"stderr","time":"2022-09-09T19:19:36.774665577Z"}

cocorossello commented 2 years ago

It has restarted 3 times in 25 minutes with v1.1.0, which is quite a lot.

I would say that this crashes are a lot more frequent since v>1.0.0. I have reverted to v0.9.3.

I don't know how to provide more information about this issue, if you have any indication I will be happy to help

cshum commented 2 years ago 
{"log":"github.com/cshum/imagor/vips._Cfunc_thumbnail_source_with_option(0x7f7434207270, 0xc0013bc010, 0x1f4, 0x1388, 0x0, 0x2, 0x7f742d60d780)\n","stream":"stderr","time":"2022-09-07T10:26:21.694102171Z"}
{"log":"\u0009_cgo_gotypes.go:1402 +0x4c fp=0xc000ff4f40 sp=0xc000ff4f18 pc=0xcfea0c\n","stream":"stderr","time":"2022-09-07T10:26:21.694106296Z"}
{"log":"github.com/cshum/imagor/vips.vipsThumbnailFromSource.func2(0xc00184e020?, 0x9?, 0x1f4, 0x1388, 0x0, 0x2, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694108351Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/vips.go:99 +0xbc fp=0xc000ff4f98 sp=0xc000ff4f40 pc=0xd06e3c\n","stream":"stderr","time":"2022-09-07T10:26:21.694110387Z"}
{"log":"github.com/cshum/imagor/vips.vipsThumbnailFromSource(0x7f7496d85d28?, 0x8?, 0xc001d1e400?, 0xc0013bc008?, 0x0?, 0xc000ff50b0)\n","stream":"stderr","time":"2022-09-07T10:26:21.694112449Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/vips.go:99 +0x155 fp=0xc000ff5058 sp=0xc000ff4f98 pc=0xd06bf5\n","stream":"stderr","time":"2022-09-07T10:26:21.694117436Z"}
{"log":"github.com/cshum/imagor/vips.LoadThumbnailFromSource(0x1225d38?, 0xc000960420?, 0xc000ff51d8?, 0x41d46b?, 0x7f746f2b4100?, 0xc00042e8c0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.69413578Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/image.go:137 +0x86 fp=0xc000ff5180 sp=0xc000ff5058 pc=0xd019e6\n","stream":"stderr","time":"2022-09-07T10:26:21.694139392Z"}
{"log":"github.com/cshum/imagor/vips.(*Source).LoadThumbnail(...)\n","stream":"stderr","time":"2022-09-07T10:26:21.694503284Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/source.go:61\n","stream":"stderr","time":"2022-09-07T10:26:21.694507062Z"}
{"log":"github.com/cshum/imagor/vips.newThumbnailFromBlob({0x1225d38, 0xc000960420}, 0xc000988200, 0x203000?, 0xc000960420?, 0xc0009603f0?, 0xc000753860?, 0x0?)\n","stream":"stderr","time":"2022-09-07T10:26:21.694509083Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/processor.go:174 +0x1a5 fp=0xc000ff5208 sp=0xc000ff5180 pc=0xcfa3c5\n","stream":"stderr","time":"2022-09-07T10:26:21.69451134Z"}
{"log":"github.com/cshum/imagor/vips.(*Processor).NewThumbnail(0xdf86c0?, {0x1225d38, 0xc000960420}, 0xc000988200, 0xe12080?, 0xc0009603f0?, 0x0, 0x2, 0x1)\n","stream":"stderr","time":"2022-09-07T10:26:21.694513437Z"}
{"log":"github.com/cshum/imagor/vips.Cfunc_thumbnail(0x7f3247f8aa20, 0xc0003c0010, 0x1f4, 0x2328, 0x0, 0x2)\n","stream":"stderr","time":"2022-09-09T19:19:36.77431854Z"}
{"log":"\u0009_cgo_gotypes.go:1348 +0x4c fp=0xc0011bec98 sp=0xc0011bec70 pc=0xd16e8c\n","stream":"stderr","time":"2022-09-09T19:19:36.774322528Z"}
{"log":"github.com/cshum/imagor/vips.vipsThumbnailFromFile.func1(0xc000d50230?, 0x68?, 0x1f4, 0x2328, 0x0, 0x2)\n","stream":"stderr","time":"2022-09-09T19:19:36.774326304Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/vips.go:121 +0x8c fp=0xc0011bece0 sp=0xc0011bec98 pc=0xd1fc8c\n","stream":"stderr","time":"2022-09-09T19:19:36.774329656Z"}
{"log":"github.com/cshum/imagor/vips.vipsThumbnailFromFile({0xc000640420, 0x5d}, 0x4b80a6d3de690ac8?, 0x3b2d755d6ba5372a?, 0xbe2bb20559d564e9?, 0xb428b49255ad4acb?, 0xc0011bee20)\n","stream":"stderr","time":"2022-09-09T19:19:36.774340561Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/vips.go:121 +0x14d fp=0xc0011beda0 sp=0xc0011bece0 pc=0xd1fa8d\n","stream":"stderr","time":"2022-09-09T19:19:36.774344622Z"}
{"log":"github.com/cshum/imagor/vips.LoadThumbnailFromFile({0xc000640420, 0x5d}, 0xc0013a2d00?, 0x10000000075653d?, 0x0?, 0x476de6?, 0x0)\n","stream":"stderr","time":"2022-09-09T19:19:36.774348557Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/image.go:193 +0xef fp=0xc0011beef0 sp=0xc0011beda0 pc=0xd1a98f\n","stream":"stderr","time":"2022-09-09T19:19:36.774351718Z"}
{"log":"github.com/cshum/imagor/vips.newThumbnailFromBlob({0x124e8f8, 0xc00001e6f0}, 0xc0009065a0, 0xc0011befd0?, 0x41c92b?, 0x18f1440?, 0x4279a7?, 0x8?)\n","stream":"stderr","time":"2022-09-09T19:19:36.774355138Z"}
{"log":"\u0009/go/src/github.com/cshum/imagor/vips/processor.go:166 +0x105 fp=0xc0011bef78 sp=0xc0011beef0 pc=0xd12de5\n","stream":"stderr","time":"2022-09-09T19:19:36.774358623Z"}
{"log":"github.com/cshum/imagor/vips.(*Processor).NewThumbnail(0xe11720?, {0x124e8f8, 0xc00001e6f0}, 0x4?, 0xe2b320?, 0xc00001e3f0?, 0x0, 0x2, 0x1)\n","stream":"stderr","time":"2022-09-09T19:19:36.774370427Z"}

Looking by the logs it seems all crashes has something to do with vips_thumbnail.

I have created a docker branch ghcr.io/cshum/imagor:vips-thumbnail-disable

https://github.com/cshum/imagor/pkgs/container/imagor/41106830?tag=vips-thumbnail-disable

This is basically v1.1.0 with vips_thumbnail turned off in almost all cases.

vips_thumbnail provides a lot of optimizations for speeding up the image operations. But if this attributes to the crash then it makes sense to have option to turn it off.

But I want to be sure before committing into that route. Would you test it out?

cocorossello commented 2 years ago

Sure, I just deployed it. I will let you know if it reboots.

cocorossello commented 2 years ago

It just crashed with this image:

{"level":"warn","ts":1662846989.998617,"caller":"imagor/imagor.go:482","msg":"save","key":"https://sunhotels.azureedge.net/Sunhotels.net/HotelInfo/hotelImage.aspx?id=17044669&full=1","error":"Get \"https://sunhotels.azureedge.net/Sunhotels.net/HotelInfo/hotelImage.aspx?id=17044669&full=1\": context canceled"}
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x44f6 pc=0x7fa8142830b4]

runtime stack:
runtime.throw({0xf93a53?, 0x7fa7b000dc00?})
        /usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7fa7b6c74090 sp=0x7fa7b6c74060 pc=0x43b77d
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x7fa7b6c740e0 sp=0x7fa7b6c74090 pc=0x451689

goroutine 72337 [syscall]:
runtime.cgocall(0xd265e0, 0xc00011c680)
        /usr/local/go/src/runtime/cgocall.go:158 +0x5c fp=0xc00011c658 sp=0xc00011c620 pc=0x4096dc
github.com/cshum/imagor/vips._Cfunc_clear_image(0xc0016b4000)
        _cgo_gotypes.go:697 +0x45 fp=0xc00011c680 sp=0xc00011c658 pc=0xd14365
github.com/cshum/imagor/vips.clearImage.func1(0x1?)
        /go/src/github.com/cshum/imagor/vips/vips.go:130 +0x45 fp=0xc00011c6b8 sp=0xc00011c680 pc=0xd1fe25
github.com/cshum/imagor/vips.clearImage(0x7fa7631a11a0)
        /go/src/github.com/cshum/imagor/vips/vips.go:130 +0x5b fp=0xc00011c6e8 sp=0xc00011c6b8 pc=0xd1fdbb
github.com/cshum/imagor/vips.(*Image).Close(0xc000af8040)
        /go/src/github.com/cshum/imagor/vips/image.go:234 +0x65 fp=0xc00011c738 sp=0xc00011c6e8 pc=0xd1acc5
github.com/cshum/imagor/vips.(*Image).Close-fm()
        <autogenerated>:1 +0x26 fp=0xc00011c750 sp=0xc00011c738 pc=0xd24286
github.com/cshum/imagor/vips/vipscontext.(*contextRef).Done(0xc000ec0990)
        /go/src/github.com/cshum/imagor/vips/vipscontext/context.go:28 +0x9d fp=0xc00011c788 sp=0xc00011c750 pc=0xd06cdd
github.com/cshum/imagor/vips/vipscontext.Done({0x124e8f8?, 0xc000ec09c0?})
        /go/src/github.com/cshum/imagor/vips/vipscontext/context.go:50 +0x4d fp=0xc00011c7b0 sp=0xc00011c788 pc=0xd06e2d
github.com/cshum/imagor/vips.(*Processor).Process.func1()
        /go/src/github.com/cshum/imagor/vips/process.go:50 +0x45 fp=0xc00011c7e0 sp=0xc00011c7b0 pc=0xd0f3a5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011c7e8 sp=0xc00011c7e0 pc=0x46cb81
created by github.com/cshum/imagor/vips.(*Processor).Process
        /go/src/github.com/cshum/imagor/vips/process.go:47 +0x139

goroutine 1 [chan receive, 9 minutes]:
runtime.gopark(0x1c8ea26?, 0x18c7660?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000fa0d90 sp=0xc000fa0d70 pc=0x43e456
runtime.chanrecv(0xc000172360, 0x0, 0x1)
        /usr/local/go/src/runtime/chan.go:583 +0x49b fp=0xc000fa0e20 sp=0xc000fa0d90 pc=0x40c6db
runtime.chanrecv1(0xdbc520?, 0xf65f60?)
        /usr/local/go/src/runtime/chan.go:442 +0x18 fp=0xc000fa0e48 sp=0xc000fa0e20 pc=0x40c1d8
github.com/cshum/imagor/server.(*Server).RunContext(0xc00072c2c0, {0x124f0a0, 0xc0003e2240})
        /go/src/github.com/cshum/imagor/server/server.go:83 +0x16b fp=0xc000fa0ed0 sp=0xc000fa0e48 pc=0x76772b
github.com/cshum/imagor/server.(*Server).Run(0xc000036240?)
        /go/src/github.com/cshum/imagor/server/server.go:71 +0xa5 fp=0xc000fa0f28 sp=0xc000fa0ed0 pc=0x767565
main.main()
        /go/src/github.com/cshum/imagor/cmd/imagor/main.go:19 +0xb7 fp=0xc000fa0f80 sp=0xc000fa0f28 pc=0xd25fb7
runtime.main()
        /usr/local/go/src/runtime/proc.go:250 +0x212 fp=0xc000fa0fe0 sp=0xc000fa0f80 pc=0x43e092
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000fa0fe8 sp=0xc000fa0fe0 pc=0x46cb81

goroutine 2 [force gc (idle), 9 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a0fb0 sp=0xc0000a0f90 pc=0x43e456
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:302 +0xad fp=0xc0000a0fe0 sp=0xc0000a0fb0 pc=0x43e2ed
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a0fe8 sp=0xc0000a0fe0 pc=0x46cb81
created by runtime.init.6
        /usr/local/go/src/runtime/proc.go:290 +0x25

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a1790 sp=0xc0000a1770 pc=0x43e456
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.bgsweep(0x0?)
        /usr/local/go/src/runtime/mgcsweep.go:297 +0xd7 fp=0xc0000a17c8 sp=0xc0000a1790 pc=0x42a897
runtime.gcenable.func1()
        /usr/local/go/src/runtime/mgc.go:178 +0x26 fp=0xc0000a17e0 sp=0xc0000a17c8 pc=0x41f506
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a17e8 sp=0xc0000a17e0 pc=0x46cb81
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:178 +0x6b

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x1daece81cf46ad?, 0x2020e6?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a1f70 sp=0xc0000a1f50 pc=0x43e456
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.(*scavengerState).park(0x18c76e0)
        /usr/local/go/src/runtime/mgcscavenge.go:389 +0x53 fp=0xc0000a1fa0 sp=0xc0000a1f70 pc=0x4288f3
runtime.bgscavenge(0x0?)
        /usr/local/go/src/runtime/mgcscavenge.go:622 +0x65 fp=0xc0000a1fc8 sp=0xc0000a1fa0 pc=0x428ee5
runtime.gcenable.func2()
        /usr/local/go/src/runtime/mgc.go:179 +0x26 fp=0xc0000a1fe0 sp=0xc0000a1fc8 pc=0x41f4a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a1fe8 sp=0xc0000a1fe0 pc=0x46cb81
created by runtime.gcenable
        /usr/local/go/src/runtime/mgc.go:179 +0xaa

goroutine 5 [finalizer wait]:
runtime.gopark(0x0?, 0xfc8658?, 0x80?, 0xe0?, 0x1000000010?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a0628 sp=0xc0000a0608 pc=0x43e456
runtime.goparkunlock(...)
        /usr/local/go/src/runtime/proc.go:369
runtime.runfinq()
        /usr/local/go/src/runtime/mfinal.go:180 +0x10f fp=0xc0000a07e0 sp=0xc0000a0628 pc=0x41e58f
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a07e8 sp=0xc0000a07e0 pc=0x46cb81
created by runtime.createfing
        /usr/local/go/src/runtime/mfinal.go:157 +0x45

goroutine 6 [GC worker (idle), 9 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a2750 sp=0xc0000a2730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000a27e0 sp=0xc0000a2750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a27e8 sp=0xc0000a27e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 18 [GC worker (idle), 9 minutes]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009c750 sp=0xc00009c730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009c7e0 sp=0xc00009c750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009c7e8 sp=0xc00009c7e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x18f92e0?, 0x1?, 0x67?, 0xbc?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000114750 sp=0xc000114730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001147e0 sp=0xc000114750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001147e8 sp=0xc0001147e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x1daece81b92ae1?, 0x1?, 0x97?, 0xa8?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009cf50 sp=0xc00009cf30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009cfe0 sp=0xc00009cf50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009cfe8 sp=0xc00009cfe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x1daece81b8e2a2?, 0x1?, 0x2b?, 0x55?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000114f50 sp=0xc000114f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000114fe0 sp=0xc000114f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000114fe8 sp=0xc000114fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x18f92e0?, 0x3?, 0xe5?, 0x3e?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000115750 sp=0xc000115730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001157e0 sp=0xc000115750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001157e8 sp=0xc0001157e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x18f92e0?, 0x3?, 0x58?, 0xd3?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009d750 sp=0xc00009d730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009d7e0 sp=0xc00009d750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009d7e8 sp=0xc00009d7e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x1daece81b8b312?, 0x3?, 0xd0?, 0x68?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009df50 sp=0xc00009df30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009dfe0 sp=0xc00009df50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009dfe8 sp=0xc00009dfe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 37 [GC worker (idle)]:
runtime.gopark(0x1daece81b8dbda?, 0x1?, 0x92?, 0xaa?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000115f50 sp=0xc000115f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000115fe0 sp=0xc000115f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000115fe8 sp=0xc000115fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x1daece81b8ae71?, 0x3?, 0xca?, 0xc6?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009e750 sp=0xc00009e730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009e7e0 sp=0xc00009e750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009e7e8 sp=0xc00009e7e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 38 [GC worker (idle)]:
runtime.gopark(0x1daece81b973cf?, 0x1?, 0xc4?, 0x4b?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000116750 sp=0xc000116730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001167e0 sp=0xc000116750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001167e8 sp=0xc0001167e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 23 [GC worker (idle)]:
runtime.gopark(0x1daecaf6b0f17c?, 0x3?, 0x35?, 0xd0?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009ef50 sp=0xc00009ef30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009efe0 sp=0xc00009ef50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009efe8 sp=0xc00009efe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 39 [GC worker (idle)]:
runtime.gopark(0x1daece81b9640c?, 0x3?, 0x62?, 0xa9?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000116f50 sp=0xc000116f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000116fe0 sp=0xc000116f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000116fe8 sp=0xc000116fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 24 [GC worker (idle)]:
runtime.gopark(0x1daece81b97336?, 0x1?, 0xa6?, 0xf7?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009f750 sp=0xc00009f730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009f7e0 sp=0xc00009f750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009f7e8 sp=0xc00009f7e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 40 [GC worker (idle)]:
runtime.gopark(0x1daece81b900d1?, 0x3?, 0x32?, 0x35?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000117750 sp=0xc000117730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001177e0 sp=0xc000117750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001177e8 sp=0xc0001177e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 25 [GC worker (idle)]:
runtime.gopark(0x1daece18583d08?, 0x1?, 0x87?, 0xdc?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00009ff50 sp=0xc00009ff30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc00009ffe0 sp=0xc00009ff50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00009ffe8 sp=0xc00009ffe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 41 [GC worker (idle)]:
runtime.gopark(0x1daece37814611?, 0x3?, 0x55?, 0x12?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000117f50 sp=0xc000117f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000117fe0 sp=0xc000117f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000117fe8 sp=0xc000117fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 26 [GC worker (idle)]:
runtime.gopark(0x1daece18583ccb?, 0x3?, 0x26?, 0x9?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000110750 sp=0xc000110730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001107e0 sp=0xc000110750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001107e8 sp=0xc0001107e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 42 [GC worker (idle)]:
runtime.gopark(0x18f92e0?, 0x1?, 0xd5?, 0x85?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000120750 sp=0xc000120730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001207e0 sp=0xc000120750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001207e8 sp=0xc0001207e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 27 [GC worker (idle)]:
runtime.gopark(0x1daece1858572c?, 0x3?, 0x43?, 0xb4?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000110f50 sp=0xc000110f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000110fe0 sp=0xc000110f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000110fe8 sp=0xc000110fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 43 [GC worker (idle)]:
runtime.gopark(0x18f92e0?, 0x1?, 0x62?, 0x2?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000120f50 sp=0xc000120f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000120fe0 sp=0xc000120f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000120fe8 sp=0xc000120fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 28 [GC worker (idle)]:
runtime.gopark(0x1daecaf6b0f0ce?, 0x3?, 0xca?, 0xb1?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000111750 sp=0xc000111730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001117e0 sp=0xc000111750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001117e8 sp=0xc0001117e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 44 [GC worker (idle), 1 minutes]:
runtime.gopark(0x1daeb4022060c1?, 0x3?, 0x8c?, 0x64?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000121750 sp=0xc000121730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001217e0 sp=0xc000121750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001217e8 sp=0xc0001217e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 29 [GC worker (idle)]:
runtime.gopark(0x1daece81b8b0ae?, 0x3?, 0xa3?, 0x8b?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000111f50 sp=0xc000111f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000111fe0 sp=0xc000111f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000111fe8 sp=0xc000111fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 45 [GC worker (idle)]:
runtime.gopark(0x1daece81b91e88?, 0x3?, 0x67?, 0x9a?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000121f50 sp=0xc000121f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000121fe0 sp=0xc000121f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000121fe8 sp=0xc000121fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 30 [GC worker (idle)]:
runtime.gopark(0x1daece1858544d?, 0x1?, 0x78?, 0x79?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000112750 sp=0xc000112730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001127e0 sp=0xc000112750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001127e8 sp=0xc0001127e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 46 [GC worker (idle)]:
runtime.gopark(0x1daece81b91d71?, 0x3?, 0xbd?, 0x66?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000122750 sp=0xc000122730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001227e0 sp=0xc000122750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001227e8 sp=0xc0001227e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 31 [GC worker (idle)]:
runtime.gopark(0x1daece81b92ba7?, 0x1?, 0x2e?, 0x2a?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000112f50 sp=0xc000112f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000112fe0 sp=0xc000112f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000112fe8 sp=0xc000112fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 47 [GC worker (idle)]:
runtime.gopark(0x18f92e0?, 0x1?, 0x73?, 0xa5?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000122f50 sp=0xc000122f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc000122fe0 sp=0xc000122f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000122fe8 sp=0xc000122fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x1daece81b8ed21?, 0x3?, 0x3f?, 0x62?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a2f50 sp=0xc0000a2f30 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000a2fe0 sp=0xc0000a2f50 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a2fe8 sp=0xc0000a2fe0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 32 [GC worker (idle)]:
runtime.gopark(0x1daece81b8dcf1?, 0x3?, 0x86?, 0x62?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000113750 sp=0xc000113730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0001137e0 sp=0xc000113750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001137e8 sp=0xc0001137e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x18f92e0?, 0x3?, 0x29?, 0xfe?, 0x0?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000a3750 sp=0xc0000a3730 pc=0x43e456
runtime.gcBgMarkWorker()
        /usr/local/go/src/runtime/mgc.go:1235 +0xf1 fp=0xc0000a37e0 sp=0xc0000a3750 pc=0x421651
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000a37e8 sp=0xc0000a37e0 pc=0x46cb81
created by runtime.gcBgMarkStartWorkers
        /usr/local/go/src/runtime/mgc.go:1159 +0x25

goroutine 48 [select]:
runtime.gopark(0xc000123788?, 0x3?, 0x40?, 0x2b?, 0xc000123772?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0001235f8 sp=0xc0001235d8 pc=0x43e456
runtime.selectgo(0xc000123788, 0xc00012376c, 0xc000118080?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc000123738 sp=0xc0001235f8 pc=0x44d8dc
go.opencensus.io/stats/view.(*worker).start(0xc000118080)
        /go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xad fp=0xc0001237c8 sp=0xc000123738 pc=0xc0b1cd
go.opencensus.io/stats/view.init.0.func1()
        /go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x26 fp=0xc0001237e0 sp=0xc0001237c8 pc=0xc0a446
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0001237e8 sp=0xc0001237e0 pc=0x46cb81
created by go.opencensus.io/stats/view.init.0
        /go/pkg/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x8d

goroutine 50 [select, 9 minutes, locked to thread]:
runtime.gopark(0xc00011dfa8?, 0x2?, 0xa0?, 0x61?, 0xc00011dfa4?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00011de18 sp=0xc00011ddf8 pc=0x43e456
runtime.selectgo(0xc00011dfa8, 0xc00011dfa0, 0x0?, 0x0, 0x8?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc00011df58 sp=0xc00011de18 pc=0x44d8dc
runtime.ensureSigM.func1()
        /usr/local/go/src/runtime/signal_unix.go:991 +0x1b0 fp=0xc00011dfe0 sp=0xc00011df58 pc=0x451b30
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00011dfe8 sp=0xc00011dfe0 pc=0x46cb81
created by runtime.ensureSigM
        /usr/local/go/src/runtime/signal_unix.go:974 +0xbd

goroutine 9 [syscall, 9 minutes]:
runtime.notetsleepg(0x0?, 0x0?)
        /usr/local/go/src/runtime/lock_futex.go:236 +0x34 fp=0xc000531fa0 sp=0xc000531f68 pc=0x411ad4
os/signal.signal_recv()
        /usr/local/go/src/runtime/sigqueue.go:152 +0x2f fp=0xc000531fc0 sp=0xc000531fa0 pc=0x46924f
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 fp=0xc000531fe0 sp=0xc000531fc0 pc=0x764ef9
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000531fe8 sp=0xc000531fe0 pc=0x46cb81
created by os/signal.Notify.func1.1
        /usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 51 [select, 9 minutes]:
runtime.gopark(0xc00052dfb0?, 0x2?, 0x0?, 0x0?, 0xc00052df9c?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00052de28 sp=0xc00052de08 pc=0x43e456
runtime.selectgo(0xc00052dfb0, 0xc00052df98, 0x0?, 0x0, 0x0?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc00052df68 sp=0xc00052de28 pc=0x44d8dc
os/signal.NotifyContext.func1()
        /usr/local/go/src/os/signal/signal.go:288 +0x70 fp=0xc00052dfe0 sp=0xc00052df68 pc=0x764ab0
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00052dfe8 sp=0xc00052dfe0 pc=0x46cb81
created by os/signal.NotifyContext
        /usr/local/go/src/os/signal/signal.go:287 +0x16b

goroutine 52 [IO wait]:
runtime.gopark(0x200000003?, 0xc0006431e0?, 0x0?, 0xa5?, 0xc0000b1b90?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0000b1b20 sp=0xc0000b1b00 pc=0x43e456
runtime.netpollblock(0xc0000b1b70?, 0xb1be8?, 0xc0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0000b1b58 sp=0xc0000b1b20 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7e54d5b98, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0000b1b78 sp=0xc0000b1b58 pc=0x467229
internal/poll.(*pollDesc).wait(0xc0005a4000?, 0x6?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0000b1ba0 sp=0xc0000b1b78 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc0005a4000)
        /usr/local/go/src/internal/poll/fd_unix.go:614 +0x234 fp=0xc0000b1c38 sp=0xc0000b1ba0 pc=0x4f0194
net.(*netFD).accept(0xc0005a4000)
        /usr/local/go/src/net/fd_unix.go:172 +0x35 fp=0xc0000b1cf0 sp=0xc0000b1c38 pc=0x5e74f5
net.(*TCPListener).accept(0xc0000126f0)
        /usr/local/go/src/net/tcpsock_posix.go:142 +0x28 fp=0xc0000b1d20 sp=0xc0000b1cf0 pc=0x5ffd48
net.(*TCPListener).Accept(0xc0000126f0)
        /usr/local/go/src/net/tcpsock.go:288 +0x3d fp=0xc0000b1d50 sp=0xc0000b1d20 pc=0x5fef1d
net/http.(*onceCloseListener).Accept(0xc00143c000?)
        <autogenerated>:1 +0x2a fp=0xc0000b1d68 sp=0xc0000b1d50 pc=0x7090aa
net/http.(*Server).Serve(0xc00072c2c0, {0x124dcb0, 0xc0000126f0})
        /usr/local/go/src/net/http/server.go:3070 +0x385 fp=0xc0000b1e98 sp=0xc0000b1d68 pc=0x6e45c5
net/http.(*Server).ListenAndServe(0xc00072c2c0)
        /usr/local/go/src/net/http/server.go:2999 +0x7d fp=0xc0000b1ec8 sp=0xc0000b1e98 pc=0x6e41fd
github.com/cshum/imagor/server.(*Server).listenAndServe(0x0?)
        /go/src/github.com/cshum/imagor/server/server.go:112 +0x4e fp=0xc0000b1f00 sp=0xc0000b1ec8 pc=0x767f8e
github.com/cshum/imagor/server.(*Server).RunContext.func1()
        /go/src/github.com/cshum/imagor/server/server.go:78 +0x34 fp=0xc0000b1fe0 sp=0xc0000b1f00 pc=0x7677b4
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0000b1fe8 sp=0xc0000b1fe0 pc=0x46cb81
created by github.com/cshum/imagor/server.(*Server).RunContext
        /go/src/github.com/cshum/imagor/server/server.go:77 +0x7c

goroutine 105 [IO wait]:
runtime.gopark(0x9ffb?, 0xb?, 0x0?, 0x0?, 0x8?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000442618 sp=0xc0004425f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x1f8b?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc000442650 sp=0xc000442618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7e54d59b8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc000442670 sp=0xc000442650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc0005a4380?, 0xc000adc000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000442698 sp=0xc000442670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005a4380, {0xc000adc000, 0xa000, 0xa000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000442718 sp=0xc000442698 pc=0x4ec27a
net.(*netFD).Read(0xc0005a4380, {0xc000adc000?, 0xc000733a80?, 0xc000adc005?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000442760 sp=0xc000442718 pc=0x5e53e9
net.(*conn).Read(0xc000014008, {0xc000adc000?, 0xa000?, 0xc000733a80?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0004427a8 sp=0xc000442760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc000012918, {0xc000adc000?, 0x0?, 0xc000442860?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0004427f0 sp=0xc0004427a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc000730278, {0x1244740, 0xc000012918})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc000442848 sp=0xc0004427f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc000730000, {0x1246e80?, 0xc000014008}, 0xa000?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc000442888 sp=0xc000442848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc000730000, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc000442c10 sp=0xc000442888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc000730000, {0xc0009be000, 0x1000, 0x6aca20?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc000442c80 sp=0xc000442c10 pc=0x638aaf
bufio.(*Reader).Read(0xc0009956e0, {0xc0009aa4a0, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000442cb8 sp=0xc000442c80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc0009956e0}, {0xc0009aa4a0, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000442d00 sp=0xc000442cb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc0009aa4a0?, 0x9?, 0xc000ec0420?}, {0x1243b60?, 0xc0009956e0?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc000442d50 sp=0xc000442d00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc0009aa460)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc000442e00 sp=0xc000442d50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc000442f98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc000442f60 sp=0xc000442e00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc0005b8300)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc000442fc8 sp=0xc000442f60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc000442fe0 sp=0xc000442fc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000442fe8 sp=0xc000442fe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 1231 [IO wait]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x11?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000446af8 sp=0xc000446ad8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x4de5d7?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc000446b30 sp=0xc000446af8 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7e54d57d8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc000446b50 sp=0xc000446b30 pc=0x467229
internal/poll.(*pollDesc).wait(0xc0001ba800?, 0xc000f2f000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000446b78 sp=0xc000446b50 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001ba800, {0xc000f2f000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000446bf8 sp=0xc000446b78 pc=0x4ec27a
net.(*netFD).Read(0xc0001ba800, {0xc000f2f000?, 0x40bc31?, 0x4?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000446c40 sp=0xc000446bf8 pc=0x5e53e9
net.(*conn).Read(0xc000014638, {0xc000f2f000?, 0x0?, 0x0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000446c88 sp=0xc000446c40 pc=0x5f70a5
net/http.(*persistConn).Read(0xc0004c86c0, {0xc000f2f000?, 0xc0000e1260?, 0xc000446d30?})
        /usr/local/go/src/net/http/transport.go:1929 +0x4e fp=0xc000446ce8 sp=0xc000446c88 pc=0x6f8cee
bufio.(*Reader).fill(0xc0006cb1a0)
        /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc000446d20 sp=0xc000446ce8 pc=0x51c57f
bufio.(*Reader).Peek(0xc0006cb1a0, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc000446d40 sp=0xc000446d20 pc=0x51c6dd
net/http.(*persistConn).readLoop(0xc0004c86c0)
        /usr/local/go/src/net/http/transport.go:2093 +0x1ac fp=0xc000446fc8 sp=0xc000446d40 pc=0x6f9b0c
net/http.(*Transport).dialConn.func5()
        /usr/local/go/src/net/http/transport.go:1751 +0x26 fp=0xc000446fe0 sp=0xc000446fc8 pc=0x6f82e6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000446fe8 sp=0xc000446fe0 pc=0x46cb81
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1751 +0x173e

goroutine 53521 [select]:
runtime.gopark(0xc00114df90?, 0x2?, 0xd8?, 0xdd?, 0xc00114df24?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00114dd90 sp=0xc00114dd70 pc=0x43e456
runtime.selectgo(0xc00114df90, 0xc00114df20, 0xc000a8a5c0?, 0x0, 0xc000ec1560?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc00114ded0 sp=0xc00114dd90 pc=0x44d8dc
net/http.(*persistConn).writeLoop(0xc0004c9200)
        /usr/local/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00114dfc8 sp=0xc00114ded0 pc=0x6fb795
net/http.(*Transport).dialConn.func6()
        /usr/local/go/src/net/http/transport.go:1752 +0x26 fp=0xc00114dfe0 sp=0xc00114dfc8 pc=0x6f8286
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00114dfe8 sp=0xc00114dfe0 pc=0x46cb81
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1752 +0x1791

goroutine 56341 [select]:
runtime.gopark(0xc00114ff90?, 0x2?, 0xd8?, 0xfd?, 0xc00114ff24?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00114fd90 sp=0xc00114fd70 pc=0x43e456
runtime.selectgo(0xc00114ff90, 0xc00114ff20, 0xc0009f2340?, 0x0, 0xc00061e930?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc00114fed0 sp=0xc00114fd90 pc=0x44d8dc
net/http.(*persistConn).writeLoop(0xc000728d80)
        /usr/local/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00114ffc8 sp=0xc00114fed0 pc=0x6fb795
net/http.(*Transport).dialConn.func6()
        /usr/local/go/src/net/http/transport.go:1752 +0x26 fp=0xc00114ffe0 sp=0xc00114ffc8 pc=0x6f8286
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00114ffe8 sp=0xc00114ffe0 pc=0x46cb81
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1752 +0x1791

goroutine 1223 [IO wait]:
runtime.gopark(0x50000c0001a73c8?, 0xb?, 0x0?, 0x0?, 0xf?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0008ad540 sp=0xc0008ad520 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x1a7340?, 0xc0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0008ad578 sp=0xc0008ad540 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7e54d5058, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0008ad598 sp=0xc0008ad578 pc=0x467229
internal/poll.(*pollDesc).wait(0xc0001ba700?, 0xc000af0000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0008ad5c0 sp=0xc0008ad598 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0001ba700, {0xc000af0000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0008ad640 sp=0xc0008ad5c0 pc=0x4ec27a
net.(*netFD).Read(0xc0001ba700, {0xc000af0000?, 0xc0001a7340?, 0x4771be?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0008ad688 sp=0xc0008ad640 pc=0x5e53e9
net.(*conn).Read(0xc0000145f0, {0xc000af0000?, 0x6d01cc?, 0xc000b98080?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0008ad6d0 sp=0xc0008ad688 pc=0x5f70a5
net/http.(*connReader).Read(0xc00066a8a0, {0xc000af0000, 0x1000, 0x1000})
        /usr/local/go/src/net/http/server.go:786 +0x171 fp=0xc0008ad720 sp=0xc0008ad6d0 pc=0x6d94f1
bufio.(*Reader).fill(0xc000b18240)
        /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc0008ad758 sp=0xc0008ad720 pc=0x51c57f
bufio.(*Reader).ReadSlice(0xc000b18240, 0x0?)
        /usr/local/go/src/bufio/bufio.go:372 +0x2f fp=0xc0008ad7a8 sp=0xc0008ad758 pc=0x51d16f
bufio.(*Reader).ReadLine(0xc000b18240)
        /usr/local/go/src/bufio/bufio.go:401 +0x27 fp=0xc0008ad7f8 sp=0xc0008ad7a8 pc=0x51d3c7
net/textproto.(*Reader).readLineSlice(0xc0010e8150)
        /usr/local/go/src/net/textproto/reader.go:56 +0x99 fp=0xc0008ad8e8 sp=0xc0008ad7f8 pc=0x6708f9
net/textproto.(*Reader).ReadLine(...)
        /usr/local/go/src/net/textproto/reader.go:37
net/http.readRequest(0xc0000145f0?)
        /usr/local/go/src/net/http/request.go:1036 +0x79 fp=0xc0008ad9f0 sp=0xc0008ad8e8 pc=0x6d4879
net/http.(*conn).readRequest(0xc0003af680, {0x124e850, 0xc000af88c0})
        /usr/local/go/src/net/http/server.go:994 +0x24a fp=0xc0008adb78 sp=0xc0008ad9f0 pc=0x6daa8a
net/http.(*conn).serve(0xc0003af680, {0x124e8f8, 0xc0006a0ba0})
        /usr/local/go/src/net/http/server.go:1916 +0x345 fp=0xc0008adfb8 sp=0xc0008adb78 pc=0x6defc5
net/http.(*Server).Serve.func3()
        /usr/local/go/src/net/http/server.go:3102 +0x2e fp=0xc0008adfe0 sp=0xc0008adfb8 pc=0x6e49ae
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0008adfe8 sp=0xc0008adfe0 pc=0x46cb81
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3102 +0x4db

goroutine 1232 [select]:
runtime.gopark(0xc00105cf90?, 0x2?, 0xd8?, 0xcd?, 0xc00105cf24?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00105cd90 sp=0xc00105cd70 pc=0x43e456
runtime.selectgo(0xc00105cf90, 0xc00105cf20, 0xc000af8c40?, 0x0, 0xc0010e8570?, 0x1)
        /usr/local/go/src/runtime/select.go:328 +0x7bc fp=0xc00105ced0 sp=0xc00105cd90 pc=0x44d8dc
net/http.(*persistConn).writeLoop(0xc0004c86c0)
        /usr/local/go/src/net/http/transport.go:2392 +0xf5 fp=0xc00105cfc8 sp=0xc00105ced0 pc=0x6fb795
net/http.(*Transport).dialConn.func6()
        /usr/local/go/src/net/http/transport.go:1752 +0x26 fp=0xc00105cfe0 sp=0xc00105cfc8 pc=0x6f8286
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00105cfe8 sp=0xc00105cfe0 pc=0x46cb81
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1752 +0x1791

goroutine 58414 [IO wait]:
runtime.gopark(0x3898?, 0xb?, 0x0?, 0x0?, 0x1b?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0015c3618 sp=0xc0015c35f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0xa?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0015c3650 sp=0xc0015c3618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7da8804b8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0015c3670 sp=0xc0015c3650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000fb0c00?, 0xc000a08000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0015c3698 sp=0xc0015c3670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000fb0c00, {0xc000a08000, 0xa000, 0xa000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0015c3718 sp=0xc0015c3698 pc=0x4ec27a
net.(*netFD).Read(0xc000fb0c00, {0xc000a08000?, 0xc000840540?, 0xc000a0e768?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0015c3760 sp=0xc0015c3718 pc=0x5e53e9
net.(*conn).Read(0xc00218c000, {0xc000a08000?, 0x5fea?, 0xc000840540?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0015c37a8 sp=0xc0015c3760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc0007262e8, {0xc000a08000?, 0x0?, 0x518028?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0015c37f0 sp=0xc0015c37a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc000a12278, {0x1244740, 0xc0007262e8})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0015c3848 sp=0xc0015c37f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc000a12000, {0x1246e80?, 0xc00218c000}, 0x389d?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0015c3888 sp=0xc0015c3848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc000a12000, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc0015c3c10 sp=0xc0015c3888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc000a12000, {0xc000b32000, 0x1000, 0x476ac9?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc0015c3c80 sp=0xc0015c3c10 pc=0x638aaf
bufio.(*Reader).Read(0xc00070f920, {0xc0009aaba0, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0015c3cb8 sp=0xc0015c3c80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc00070f920}, {0xc0009aaba0, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0015c3d00 sp=0xc0015c3cb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc0009aaba0?, 0x9?, 0xc000d584b0?}, {0x1243b60?, 0xc00070f920?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc0015c3d50 sp=0xc0015c3d00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc0009aab60)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc0015c3e00 sp=0xc0015c3d50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc0015c3f98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc0015c3f60 sp=0xc0015c3e00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc000e7a180)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc0015c3fc8 sp=0xc0015c3f60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc0015c3fe0 sp=0xc0015c3fc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0015c3fe8 sp=0xc0015c3fe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 68468 [IO wait]:
runtime.gopark(0x3becf457a2a5d585?, 0xb?, 0x0?, 0x0?, 0x23?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0013bb618 sp=0xc0013bb5f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x5?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0013bb650 sp=0xc0013bb618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7dc746bb8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0013bb670 sp=0xc0013bb650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000118300?, 0xc000e14000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0013bb698 sp=0xc0013bb670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000118300, {0xc000e14000, 0xa000, 0xa000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0013bb718 sp=0xc0013bb698 pc=0x4ec27a
net.(*netFD).Read(0xc000118300, {0xc000e14000?, 0xe7eb01?, 0xc000456000?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0013bb760 sp=0xc0013bb718 pc=0x5e53e9
net.(*conn).Read(0xc000e880a0, {0xc000e14000?, 0x1fd4?, 0xc000456000?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0013bb7a8 sp=0xc0013bb760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc000f20828, {0xc000e14000?, 0x0?, 0x518028?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0013bb7f0 sp=0xc0013bb7a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc001013778, {0x1244740, 0xc000f20828})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0013bb848 sp=0xc0013bb7f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc001013500, {0x1246e80?, 0xc000e880a0}, 0xc000e1c031?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0013bb888 sp=0xc0013bb848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc001013500, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc0013bbc10 sp=0xc0013bb888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc001013500, {0xc000b33000, 0x1000, 0x6aca20?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc0013bbc80 sp=0xc0013bbc10 pc=0x638aaf
bufio.(*Reader).Read(0xc00168b440, {0xc00083e200, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0013bbcb8 sp=0xc0013bbc80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc00168b440}, {0xc00083e200, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0013bbd00 sp=0xc0013bbcb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc00083e200?, 0x9?, 0xc000ec1d10?}, {0x1243b60?, 0xc00168b440?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc0013bbd50 sp=0xc0013bbd00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc00083e1c0)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc0013bbe00 sp=0xc0013bbd50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc0013bbf98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc0013bbf60 sp=0xc0013bbe00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc000342780)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc0013bbfc8 sp=0xc0013bbf60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc0013bbfe0 sp=0xc0013bbfc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0013bbfe8 sp=0xc0013bbfe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 70038 [IO wait]:
runtime.gopark(0x6ab008603a648241?, 0xb?, 0x0?, 0x0?, 0x18?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000e27618 sp=0xc000e275f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x5?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc000e27650 sp=0xc000e27618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7da9016d0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc000e27670 sp=0xc000e27650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000118d00?, 0xc000e6e000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000e27698 sp=0xc000e27670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000118d00, {0xc000e6e000, 0xa000, 0xa000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000e27718 sp=0xc000e27698 pc=0x4ec27a
net.(*netFD).Read(0xc000118d00, {0xc000e6e000?, 0xe7eb01?, 0xc001794680?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000e27760 sp=0xc000e27718 pc=0x5e53e9
net.(*conn).Read(0xc000e88000, {0xc000e6e000?, 0x5fea?, 0xc001794680?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc000e277a8 sp=0xc000e27760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc0005661e0, {0xc000e6e000?, 0x0?, 0xc000e27860?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc000e277f0 sp=0xc000e277a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc001012978, {0x1244740, 0xc0005661e0})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc000e27848 sp=0xc000e277f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc001012700, {0x1246e80?, 0xc000e88000}, 0xc000e7201b?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc000e27888 sp=0xc000e27848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc001012700, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc000e27c10 sp=0xc000e27888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc001012700, {0xc0008a3000, 0x1000, 0x6aca20?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc000e27c80 sp=0xc000e27c10 pc=0x638aaf
bufio.(*Reader).Read(0xc0007490e0, {0xc00059e660, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000e27cb8 sp=0xc000e27c80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc0007490e0}, {0xc00059e660, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000e27d00 sp=0xc000e27cb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc00059e660?, 0x9?, 0xc000d58390?}, {0x1243b60?, 0xc0007490e0?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc000e27d50 sp=0xc000e27d00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc00059e620)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc000e27e00 sp=0xc000e27d50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc000e27f98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc000e27f60 sp=0xc000e27e00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc000342600)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc000e27fc8 sp=0xc000e27f60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc000e27fe0 sp=0xc000e27fc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000e27fe8 sp=0xc000e27fe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 62291 [IO wait, 1 minutes]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0xd?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00145eaf8 sp=0xc00145ead8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0xafe000?, 0xc0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc00145eb30 sp=0xc00145eaf8 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7dc675e30, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc00145eb50 sp=0xc00145eb30 pc=0x467229
internal/poll.(*pollDesc).wait(0xc002ae8b80?, 0xc000906000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00145eb78 sp=0xc00145eb50 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc002ae8b80, {0xc000906000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00145ebf8 sp=0xc00145eb78 pc=0x4ec27a
net.(*netFD).Read(0xc002ae8b80, {0xc000906000?, 0xc00145ec80?, 0xc00145ec80?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc00145ec40 sp=0xc00145ebf8 pc=0x5e53e9
net.(*conn).Read(0xc000aee118, {0xc000906000?, 0x0?, 0x0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc00145ec88 sp=0xc00145ec40 pc=0x5f70a5
net/http.(*persistConn).Read(0xc0007297a0, {0xc000906000?, 0x40b3a6?, 0x60?})
        /usr/local/go/src/net/http/transport.go:1929 +0x4e fp=0xc00145ece8 sp=0xc00145ec88 pc=0x6f8cee
bufio.(*Reader).fill(0xc00052b8c0)
        /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc00145ed20 sp=0xc00145ece8 pc=0x51c57f
bufio.(*Reader).Peek(0xc00052b8c0, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc00145ed40 sp=0xc00145ed20 pc=0x51c6dd
net/http.(*persistConn).readLoop(0xc0007297a0)
        /usr/local/go/src/net/http/transport.go:2093 +0x1ac fp=0xc00145efc8 sp=0xc00145ed40 pc=0x6f9b0c
net/http.(*Transport).dialConn.func5()
        /usr/local/go/src/net/http/transport.go:1751 +0x26 fp=0xc00145efe0 sp=0xc00145efc8 pc=0x6f82e6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00145efe8 sp=0xc00145efe0 pc=0x46cb81
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1751 +0x173e

goroutine 64424 [IO wait, 1 minutes]:
runtime.gopark(0x16d4?, 0xb?, 0x0?, 0x0?, 0xe?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00171f618 sp=0xc00171f5f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0xa?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc00171f650 sp=0xc00171f618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7da940ed0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc00171f670 sp=0xc00171f650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc0005a4580?, 0xc000ed8000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00171f698 sp=0xc00171f670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0005a4580, {0xc000ed8000, 0x2500, 0x2500})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00171f718 sp=0xc00171f698 pc=0x4ec27a
net.(*netFD).Read(0xc0005a4580, {0xc000ed8000?, 0xc0004565a0?, 0xc000ed8e2c?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc00171f760 sp=0xc00171f718 pc=0x5e53e9
net.(*conn).Read(0xc0000145d8, {0xc000ed8000?, 0x2500?, 0xc0004565a0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc00171f7a8 sp=0xc00171f760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc000012ed0, {0xc000ed8000?, 0x0?, 0x41c92b?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc00171f7f0 sp=0xc00171f7a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc0010125f8, {0x1244740, 0xc000012ed0})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc00171f848 sp=0xc00171f7f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc001012380, {0x1246e80?, 0xc0000145d8}, 0x16d9?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc00171f888 sp=0xc00171f848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc001012380, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc00171fc10 sp=0xc00171f888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc001012380, {0xc000f2e000, 0x1000, 0x6aca20?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc00171fc80 sp=0xc00171fc10 pc=0x638aaf
bufio.(*Reader).Read(0xc00210f140, {0xc0008d29e0, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc00171fcb8 sp=0xc00171fc80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc00210f140}, {0xc0008d29e0, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc00171fd00 sp=0xc00171fcb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc0008d29e0?, 0x9?, 0xc000af7b00?}, {0x1243b60?, 0xc00210f140?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc00171fd50 sp=0xc00171fd00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc0008d29a0)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc00171fe00 sp=0xc00171fd50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc00171ff98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc00171ff60 sp=0xc00171fe00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc000342c00)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc00171ffc8 sp=0xc00171ff60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc00171ffe0 sp=0xc00171ffc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00171ffe8 sp=0xc00171ffe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 62189 [IO wait, 1 minutes]:
runtime.gopark(0xc31?, 0xb?, 0x0?, 0x0?, 0x17?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc000613618 sp=0xc0006135f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x9?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc000613650 sp=0xc000613618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7dc675980, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc000613670 sp=0xc000613650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000118900?, 0xc0007f4a00?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc000613698 sp=0xc000613670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000118900, {0xc0007f4a00, 0x2500, 0x2500})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc000613718 sp=0xc000613698 pc=0x4ec27a
net.(*netFD).Read(0xc000118900, {0xc0007f4a00?, 0xc0008406e0?, 0xc0007f62cf?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc000613760 sp=0xc000613718 pc=0x5e53e9
net.(*conn).Read(0xc000b880d8, {0xc0007f4a00?, 0xc000613800?, 0x4460b1?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0006137a8 sp=0xc000613760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc00221e288, {0xc0007f4a00?, 0x0?, 0x467ddb?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0006137f0 sp=0xc0006137a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc000a12978, {0x1244740, 0xc00221e288})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc000613848 sp=0xc0006137f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc000a12700, {0x1246e80?, 0xc000b880d8}, 0xc3e?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc000613888 sp=0xc000613848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc000a12700, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc000613c10 sp=0xc000613888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc000a12700, {0xc001374000, 0x1000, 0x10001?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc000613c80 sp=0xc000613c10 pc=0x638aaf
bufio.(*Reader).Read(0xc000995560, {0xc0008d2200, 0x9, 0x6c9b06?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc000613cb8 sp=0xc000613c80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc000995560}, {0xc0008d2200, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc000613d00 sp=0xc000613cb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc0008d2200?, 0x9?, 0xc0009b4c00?}, {0x1243b60?, 0xc000995560?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc000613d50 sp=0xc000613d00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc0008d21c0)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc000613e00 sp=0xc000613d50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc000613f98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc000613f60 sp=0xc000613e00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc0005b8f00)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc000613fc8 sp=0xc000613f60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc000613fe0 sp=0xc000613fc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000613fe8 sp=0xc000613fe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 56340 [IO wait]:
runtime.gopark(0x0?, 0xb?, 0x0?, 0x0?, 0x16?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0015e7af8 sp=0xc0015e7ad8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x4de5d7?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0015e7b30 sp=0xc0015e7af8 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7e54d58c8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0015e7b50 sp=0xc0015e7b30 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000fb0000?, 0xc000ce5000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0015e7b78 sp=0xc0015e7b50 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000fb0000, {0xc000ce5000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0015e7bf8 sp=0xc0015e7b78 pc=0x4ec27a
net.(*netFD).Read(0xc000fb0000, {0xc000ce5000?, 0x0?, 0x4?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0015e7c40 sp=0xc0015e7bf8 pc=0x5e53e9
net.(*conn).Read(0xc000e88070, {0xc000ce5000?, 0x0?, 0x0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0015e7c88 sp=0xc0015e7c40 pc=0x5f70a5
net/http.(*persistConn).Read(0xc000728d80, {0xc000ce5000?, 0x44e660?, 0xc0015e7ec8?})
        /usr/local/go/src/net/http/transport.go:1929 +0x4e fp=0xc0015e7ce8 sp=0xc0015e7c88 pc=0x6f8cee
bufio.(*Reader).fill(0xc001d5f080)
        /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc0015e7d20 sp=0xc0015e7ce8 pc=0x51c57f
bufio.(*Reader).Peek(0xc001d5f080, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc0015e7d40 sp=0xc0015e7d20 pc=0x51c6dd
net/http.(*persistConn).readLoop(0xc000728d80)
        /usr/local/go/src/net/http/transport.go:2093 +0x1ac fp=0xc0015e7fc8 sp=0xc0015e7d40 pc=0x6f9b0c
net/http.(*Transport).dialConn.func5()
        /usr/local/go/src/net/http/transport.go:1751 +0x26 fp=0xc0015e7fe0 sp=0xc0015e7fc8 pc=0x6f82e6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0015e7fe8 sp=0xc0015e7fe0 pc=0x46cb81
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1751 +0x173e

goroutine 53520 [IO wait]:
runtime.gopark(0x9ff3?, 0xb?, 0x0?, 0x0?, 0x12?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc00114e620 sp=0xc00114e600 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x4000?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc00114e658 sp=0xc00114e620 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7dc676010, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc00114e678 sp=0xc00114e658 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000118100?, 0xc001364000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc00114e6a0 sp=0xc00114e678 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000118100, {0xc001364000, 0xa000, 0xa000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc00114e720 sp=0xc00114e6a0 pc=0x4ec27a
net.(*netFD).Read(0xc000118100, {0xc001364000?, 0xc001794520?, 0xc00136400d?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc00114e768 sp=0xc00114e720 pc=0x5e53e9
net.(*conn).Read(0xc000b88038, {0xc001364000?, 0x58?, 0x7fa80e9b31d8?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc00114e7b0 sp=0xc00114e768 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc000726330, {0xc001364000?, 0x0?, 0x51808f?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc00114e7f8 sp=0xc00114e7b0 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc0003e4cf8, {0x1244740, 0xc000726330})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc00114e850 sp=0xc00114e7f8 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc0003e4a80, {0x1246e80?, 0xc000b88038}, 0xe44?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc00114e890 sp=0xc00114e850 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc0003e4a80, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc00114ec18 sp=0xc00114e890 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc0003e4a80, {0xc0000eb000, 0x1000, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc00114ec88 sp=0xc00114ec18 pc=0x638aaf
net/http.(*persistConn).Read(0xc0004c9200, {0xc0000eb000?, 0xc000fe4180?, 0xc00114ed30?})
        /usr/local/go/src/net/http/transport.go:1929 +0x4e fp=0xc00114ece8 sp=0xc00114ec88 pc=0x6f8cee
bufio.(*Reader).fill(0xc00168a060)
        /usr/local/go/src/bufio/bufio.go:106 +0xff fp=0xc00114ed20 sp=0xc00114ece8 pc=0x51c57f
bufio.(*Reader).Peek(0xc00168a060, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d fp=0xc00114ed40 sp=0xc00114ed20 pc=0x51c6dd
net/http.(*persistConn).readLoop(0xc0004c9200)
        /usr/local/go/src/net/http/transport.go:2093 +0x1ac fp=0xc00114efc8 sp=0xc00114ed40 pc=0x6f9b0c
net/http.(*Transport).dialConn.func5()
        /usr/local/go/src/net/http/transport.go:1751 +0x26 fp=0xc00114efe0 sp=0xc00114efc8 pc=0x6f82e6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc00114efe8 sp=0xc00114efe0 pc=0x46cb81
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1751 +0x173e

goroutine 62103 [IO wait, 1 minutes]:
runtime.gopark(0x12f3?, 0xb?, 0x0?, 0x0?, 0x1e?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0016ed618 sp=0xc0016ed5f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x9?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0016ed650 sp=0xc0016ed618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7da940de0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0016ed670 sp=0xc0016ed650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000118400?, 0xc000ea7900?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0016ed698 sp=0xc0016ed670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000118400, {0xc000ea7900, 0x1300, 0x1300})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0016ed718 sp=0xc0016ed698 pc=0x4ec27a
net.(*netFD).Read(0xc000118400, {0xc000ea7900?, 0xc001b2c360?, 0xc000ea790d?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0016ed760 sp=0xc0016ed718 pc=0x5e53e9
net.(*conn).Read(0xc000aee1b8, {0xc000ea7900?, 0x2000?, 0x0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0016ed7a8 sp=0xc0016ed760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc0007266f0, {0xc000ea7900?, 0x0?, 0x475?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0016ed7f0 sp=0xc0016ed7a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc001013af8, {0x1244740, 0xc0007266f0})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0016ed848 sp=0xc0016ed7f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc001013880, {0x1246e80?, 0xc000aee1b8}, 0x1300?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0016ed888 sp=0xc0016ed848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc001013880, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc0016edc10 sp=0xc0016ed888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc001013880, {0xc000470000, 0x1000, 0x6aca20?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc0016edc80 sp=0xc0016edc10 pc=0x638aaf
bufio.(*Reader).Read(0xc00278b3e0, {0xc001044040, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0016edcb8 sp=0xc0016edc80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc00278b3e0}, {0xc001044040, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0016edd00 sp=0xc0016edcb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc001044040?, 0x9?, 0xc002b06e40?}, {0x1243b60?, 0xc00278b3e0?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc0016edd50 sp=0xc0016edd00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc001044000)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc0016ede00 sp=0xc0016edd50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc0016edf98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc0016edf60 sp=0xc0016ede00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc0005b8a80)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc0016edfc8 sp=0xc0016edf60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc0016edfe0 sp=0xc0016edfc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0016edfe8 sp=0xc0016edfe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 33635 [IO wait]:
runtime.gopark(0x1f95?, 0xb?, 0x0?, 0x0?, 0x39?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0026ca618 sp=0xc0026ca5f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x1b82?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0026ca650 sp=0xc0026ca618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7da880b48, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0026ca670 sp=0xc0026ca650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000118700?, 0xc001ef0000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0026ca698 sp=0xc0026ca670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000118700, {0xc001ef0000, 0xa000, 0xa000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0026ca718 sp=0xc0026ca698 pc=0x4ec27a
net.(*netFD).Read(0xc000118700, {0xc001ef0000?, 0xc0008403a0?, 0x0?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0026ca760 sp=0xc0026ca718 pc=0x5e53e9
net.(*conn).Read(0xc000b88020, {0xc001ef0000?, 0x1fd4?, 0xc0008403a0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0026ca7a8 sp=0xc0026ca760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc000012630, {0xc001ef0000?, 0xc000d5cea0?, 0x6ab?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0026ca7f0 sp=0xc0026ca7a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc000983078, {0x1244740, 0xc000012630})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0026ca848 sp=0xc0026ca7f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc000982e00, {0x1246e80?, 0xc000b88020}, 0x1f9a?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0026ca888 sp=0xc0026ca848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc000982e00, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc0026cac10 sp=0xc0026ca888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc000982e00, {0xc001018000, 0x1000, 0x476ac9?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc0026cac80 sp=0xc0026cac10 pc=0x638aaf
bufio.(*Reader).Read(0xc001bff0e0, {0xc00083e9e0, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0026cacb8 sp=0xc0026cac80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc001bff0e0}, {0xc00083e9e0, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0026cad00 sp=0xc0026cacb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc00083e9e0?, 0x9?, 0xc00066b7d0?}, {0x1243b60?, 0xc001bff0e0?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc0026cad50 sp=0xc0026cad00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc00083e9a0)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc0026cae00 sp=0xc0026cad50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc0026caf98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc0026caf60 sp=0xc0026cae00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc0008c2480)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc0026cafc8 sp=0xc0026caf60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc0026cafe0 sp=0xc0026cafc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0026cafe8 sp=0xc0026cafe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 64066 [IO wait, 1 minutes]:
runtime.gopark(0x340?, 0xb?, 0x0?, 0x0?, 0xa?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0026cc618 sp=0xc0026cc5f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0xa?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0026cc650 sp=0xc0026cc618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7da9017c0, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0026cc670 sp=0xc0026cc650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc00121a180?, 0xc000cb3000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0026cc698 sp=0xc0026cc670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00121a180, {0xc000cb3000, 0x7000, 0x7000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0026cc718 sp=0xc0026cc698 pc=0x4ec27a
net.(*netFD).Read(0xc00121a180, {0xc000cb3000?, 0xc0000bf1c0?, 0xc000cb9cc0?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0026cc760 sp=0xc0026cc718 pc=0x5e53e9
net.(*conn).Read(0xc00098a000, {0xc000cb3000?, 0x2fea?, 0xc0000bf1c0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0026cc7a8 sp=0xc0026cc760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc0005664b0, {0xc000cb3000?, 0x0?, 0x51808f?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0026cc7f0 sp=0xc0026cc7a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc0003e5af8, {0x1244740, 0xc0005664b0})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0026cc848 sp=0xc0026cc7f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc0003e5880, {0x1246e80?, 0xc00098a000}, 0x345?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0026cc888 sp=0xc0026cc848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc0003e5880, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc0026ccc10 sp=0xc0026cc888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc0003e5880, {0xc000a5f000, 0x1000, 0x6aca20?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc0026ccc80 sp=0xc0026ccc10 pc=0x638aaf
bufio.(*Reader).Read(0xc00052aba0, {0xc00076a200, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0026cccb8 sp=0xc0026ccc80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc00052aba0}, {0xc00076a200, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0026ccd00 sp=0xc0026cccb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc00076a200?, 0x9?, 0xc00086ee70?}, {0x1243b60?, 0xc00052aba0?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc0026ccd50 sp=0xc0026ccd00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc00076a1c0)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc0026cce00 sp=0xc0026ccd50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc0026ccf98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc0026ccf60 sp=0xc0026cce00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc0004a2780)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc0026ccfc8 sp=0xc0026ccf60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc0026ccfe0 sp=0xc0026ccfc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0026ccfe8 sp=0xc0026ccfe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 68917 [IO wait]:
runtime.gopark(0x282a?, 0xb?, 0x0?, 0x0?, 0x14?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0010a2618 sp=0xc0010a25f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0xa?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0010a2650 sp=0xc0010a2618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7e54d5418, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0010a2670 sp=0xc0010a2650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc00121ad00?, 0xc000028800?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0010a2698 sp=0xc0010a2670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00121ad00, {0xc000028800, 0x4800, 0x4800})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0010a2718 sp=0xc0010a2698 pc=0x4ec27a
net.(*netFD).Read(0xc00121ad00, {0xc000028800?, 0xc000456420?, 0xc00002a7d6?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0010a2760 sp=0xc0010a2718 pc=0x5e53e9
net.(*conn).Read(0xc000014678, {0xc000028800?, 0x4800?, 0xc000456420?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0010a27a8 sp=0xc0010a2760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc00221e0f0, {0xc000028800?, 0x0?, 0xc0010a2860?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0010a27f0 sp=0xc0010a27a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc000982978, {0x1244740, 0xc00221e0f0})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0010a2848 sp=0xc0010a27f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc000982700, {0x1246e80?, 0xc000014678}, 0x282f?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0010a2888 sp=0xc0010a2848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc000982700, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc0010a2c10 sp=0xc0010a2888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc000982700, {0xc000de5000, 0x1000, 0x6aca20?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc0010a2c80 sp=0xc0010a2c10 pc=0x638aaf
bufio.(*Reader).Read(0xc0021aae40, {0xc0008d2660, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc0010a2cb8 sp=0xc0010a2c80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc0021aae40}, {0xc0008d2660, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc0010a2d00 sp=0xc0010a2cb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc0008d2660?, 0x9?, 0xc00247e390?}, {0x1243b60?, 0xc0021aae40?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc0010a2d50 sp=0xc0010a2d00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc0008d2620)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc0010a2e00 sp=0xc0010a2d50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc0010a2f98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc0010a2f60 sp=0xc0010a2e00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc000342a80)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc0010a2fc8 sp=0xc0010a2f60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc0010a2fe0 sp=0xc0010a2fc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0010a2fe8 sp=0xc0010a2fe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 63927 [IO wait, 1 minutes]:
runtime.gopark(0xd3a?, 0xb?, 0x0?, 0x0?, 0x10?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc001d37618 sp=0xc001d375f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0xa?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc001d37650 sp=0xc001d37618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7da901b80, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc001d37670 sp=0xc001d37650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc00121a700?, 0xc0003be380?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc001d37698 sp=0xc001d37670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc00121a700, {0xc0003be380, 0xd80, 0xd80})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc001d37718 sp=0xc001d37698 pc=0x4ec27a
net.(*netFD).Read(0xc00121a700, {0xc0003be380?, 0xc000a805a0?, 0xc0003be3c6?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc001d37760 sp=0xc001d37718 pc=0x5e53e9
net.(*conn).Read(0xc0016b4118, {0xc0003be380?, 0xd80?, 0xc000a805a0?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc001d377a8 sp=0xc001d37760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc000012dc8, {0xc0003be380?, 0x0?, 0x41c92b?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc001d377f0 sp=0xc001d377a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc0003e45f8, {0x1244740, 0xc000012dc8})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc001d37848 sp=0xc001d377f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc0003e4380, {0x1246e80?, 0xc0016b4118}, 0xd3f?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc001d37888 sp=0xc001d37848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc0003e4380, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc001d37c10 sp=0xc001d37888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:582
crypto/tls.(*Conn).Read(0xc0003e4380, {0xc00046d000, 0x1000, 0x6aca20?})
        /usr/local/go/src/crypto/tls/conn.go:1287 +0x16f fp=0xc001d37c80 sp=0xc001d37c10 pc=0x638aaf
bufio.(*Reader).Read(0xc001d5e960, {0xc0008d24a0, 0x9, 0x6c9905?})
        /usr/local/go/src/bufio/bufio.go:237 +0x1bb fp=0xc001d37cb8 sp=0xc001d37c80 pc=0x51cb5b
io.ReadAtLeast({0x1243b60, 0xc001d5e960}, {0xc0008d24a0, 0x9, 0x9}, 0x9)
        /usr/local/go/src/io/io.go:332 +0x9a fp=0xc001d37d00 sp=0xc001d37cb8 pc=0x47887a
io.ReadFull(...)
        /usr/local/go/src/io/io.go:351
net/http.http2readFrameHeader({0xc0008d24a0?, 0x9?, 0xc00091f140?}, {0x1243b60?, 0xc001d5e960?})
        /usr/local/go/src/net/http/h2_bundle.go:1565 +0x6e fp=0xc001d37d50 sp=0xc001d37d00 pc=0x6a356e
net/http.(*http2Framer).ReadFrame(0xc0008d2460)
        /usr/local/go/src/net/http/h2_bundle.go:1829 +0x95 fp=0xc001d37e00 sp=0xc001d37d50 pc=0x6a3dd5
net/http.(*http2clientConnReadLoop).run(0xc001d37f98)
        /usr/local/go/src/net/http/h2_bundle.go:8872 +0x130 fp=0xc001d37f60 sp=0xc001d37e00 pc=0x6c67b0
net/http.(*http2ClientConn).readLoop(0xc0004a2180)
        /usr/local/go/src/net/http/h2_bundle.go:8768 +0x6f fp=0xc001d37fc8 sp=0xc001d37f60 pc=0x6c5ccf
net/http.(*http2Transport).newClientConn.func1()
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0x26 fp=0xc001d37fe0 sp=0xc001d37fc8 pc=0x6be9a6
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc001d37fe8 sp=0xc001d37fe0 pc=0x46cb81
created by net/http.(*http2Transport).newClientConn
        /usr/local/go/src/net/http/h2_bundle.go:7475 +0xaaa

goroutine 70995 [IO wait]:
runtime.gopark(0x47fb?, 0xb?, 0x0?, 0x0?, 0x13?)
        /usr/local/go/src/runtime/proc.go:363 +0xd6 fp=0xc0011db618 sp=0xc0011db5f8 pc=0x43e456
runtime.netpollblock(0x4d09a5?, 0x3582?, 0x0?)
        /usr/local/go/src/runtime/netpoll.go:526 +0xf7 fp=0xc0011db650 sp=0xc0011db618 pc=0x436cd7
internal/poll.runtime_pollWait(0x7fa7da880e18, 0x72)
        /usr/local/go/src/runtime/netpoll.go:305 +0x89 fp=0xc0011db670 sp=0xc0011db650 pc=0x467229
internal/poll.(*pollDesc).wait(0xc000118600?, 0xc00002d000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32 fp=0xc0011db698 sp=0xc0011db670 pc=0x4eaf12
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000118600, {0xc00002d000, 0x4800, 0x4800})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a fp=0xc0011db718 sp=0xc0011db698 pc=0x4ec27a
net.(*netFD).Read(0xc000118600, {0xc00002d000?, 0xc0000bf760?, 0xc00002d005?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29 fp=0xc0011db760 sp=0xc0011db718 pc=0x5e53e9
net.(*conn).Read(0xc00218c018, {0xc00002d000?, 0x2697?, 0xc0000bf760?})
        /usr/local/go/src/net/net.go:183 +0x45 fp=0xc0011db7a8 sp=0xc0011db760 pc=0x5f70a5
crypto/tls.(*atLeastReader).Read(0xc000e87260, {0xc00002d000?, 0x0?, 0x51808f?})
        /usr/local/go/src/crypto/tls/conn.go:787 +0x3d fp=0xc0011db7f0 sp=0xc0011db7a8 pc=0x63581d
bytes.(*Buffer).ReadFrom(0xc000a125f8, {0x1244740, 0xc000e87260})
        /usr/local/go/src/bytes/buffer.go:202 +0x98 fp=0xc0011db848 sp=0xc0011db7f0 pc=0x5184f8
crypto/tls.(*Conn).readFromUntil(0xc000a12380, {0x1246e80?, 0xc00218c018}, 0x4800?)
        /usr/local/go/src/crypto/tls/conn.go:809 +0xe5 fp=0xc0011db888 sp=0xc0011db848 pc=0x635a05
crypto/tls.(*Conn).readRecordOrCCS(0xc000a12380, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:616 +0x116 fp=0xc0011dbc10 sp=0xc0011db888 pc=0x632e56
crypto/tls.(*Conn).readRecord(...)
cshum commented 2 years ago

Has this vips._Cfunc_clear_image been showing previously, or is it a new one?

cocorossello commented 2 years ago

I don't really know. I can make it crash again in 1.1.0 and get the whole thread dump to compare. It's much more frequent now, but I have no idea why, traffic has increased but not that much. We are getting about 20 request per second, with peaks of maybe 80

cshum commented 2 years ago

Another speculation, perhaps in a high memory pressure environment the Go garbage collector kicks in prematurely sometimes.

Made some fixes based on that assumption, would you test the docker image?

shumc/imagor@sha256:c0c46ac09488e22bdd878e9cc3a68585b500a4f50ef5ea9c498a84405ca865b9

Its the docker image from the same branch vips-thumbnail-disable, Using sha digest this time so it would hit the old image cache https://github.com/cshum/imagor/pkgs/container/imagor/41530990?tag=vips-thumbnail-disable

cocorossello commented 2 years ago

I tried this image and it crashed 3 times in 15 minutes. I have reverted to 0.9.3 that crashes way less frequently.

Logs here: https://gist.github.com/cocorossello/3ebe654eb8afb1ab1458720a48243aae

I suspect that it has something to do with GC. I'm having 2 types of failures:

(healthcheckimage is another container inside the same compose unit that only has a sample image)

cshum commented 2 years ago

Can you share your docker environment variables (except secret key, access key etc)? What is the docker memory/cpu limit it is run on?

By now it is probably not a good idea just guessing things, better reproduce in a load tested environment first.

cocorossello commented 2 years ago

Last docker-compose.yml:

version: '3'
services:
  imagor:
    container_name: imagor
    image: ghcr.io/cshum/imagor@sha256:018179c4b602370be70d412866c3a9672fa5eab694558b4646c56c70a63af0e7
    restart: always
    environment:
      IMAGOR_UNSAFE: 1
      VIPS_MAX_WIDTH: 9000
      VIPS_MAX_HEIGHT: 9000
      VIPS_MAX_RESOLUTION: 100000000
      VIPS_MAX_ANIMATION_FRAMES: 1
      FILE_STORAGE_BASE_DIR: /data/storage
      FILE_STORAGE_EXPIRATION: 167h
      FILE_RESULT_STORAGE_BASE_DIR: /data/result-storage
      FILE_RESULT_STORAGE_EXPIRATION: 167h
      IMAGOR_RESULT_STORAGE_PATH_STYLE: suffix
      SERVER_ACCESS_LOG: 0
    volumes:
      - /services/images/imagor:/data
    ports:
      - 8088:8000
    logging:
      options:
        max-size: 20m
  imagor-delete-cache:
    build: .
    container_name: imagor-delete-cache
    volumes:
      - /services/images/imagor:/data
    logging:
      options:
        max-size: 10m
  healthcheck-image:
    image: flashspys/nginx-static
    volumes:
      - ./staticimage:/static

imagor-delete-cache is just a sh that deletes files in /data older than 7 days

cocorossello commented 2 years ago

I tried to make a load test with the images requested between the first and the second crash, but it doesn't crash in my local.

https://github.com/cocorossello/imagor-load-test

cshum commented 2 years ago

I tried the URLs from the repo and it seems to be doing normal in my local machine.

If there is only 1 instance, there is always a finite number of concurrent requests it can handle.

What about trying to add back 

IMAGOR_PROCESS_CONCURRENCY: 64

which queue up the pending request if it goes pass the concurrency. A sensible number would be the number of CPUsx2.

Another number is 

IMAGOR_PROCESS_QUEUE_SIZE: 64

Which result a 429 error if the queue size exceeded the specified amount.

cocorossello commented 2 years ago

I just deployed 1.1.0 with these two settings. Let's see what happens

cocorossello commented 2 years ago

No luck, 3 reboots in less than 2 hours

cshum commented 2 years ago

@cocorossello would you also try 2 more images?

 ghcr.io/cshum/imagor:sha256:def15109c6bc3134dfd31f9fca3cff2ad5c743c0b2beea087560e72466693715

https://github.com/cshum/imagor/pkgs/container/imagor/41796186?tag=master

This is from the master branch and attempts to fix some of the 406 VipsJpeg: Premature end of input file errors. Though I am not sure if it helps with segmentation violation errors...

 ghcr.io/cshum/imagor:sha256:1b971175a892b4f1ba244621bf686fb1dc0847f77c2e7a06754a4aebfc546d74

https://github.com/cshum/imagor/pkgs/container/imagor/41816160?tag=vips-thumbnail-disable

This disable streaming altogether, which resulted the same image loading mechanism as v0.9. Lets see if it helps. If it does I will expose this changes as an option when merged.

cocorossello commented 2 years ago

With this last vips-thumbnail-disable image alone I had 4 crashes, so I guess it does not improve. https://gist.github.com/cocorossello/258517cf9848625716789f6d04aa899d

I have put in front a nginx image with a cache and I haven't had a reboot for 4 hours now, but it's still not enough time.

I haven't tested the other image yet, sorry. Anyway, most of the 406 VipsJpeg: Premature end of input file errors I have I guess that come from corrupt storage files that are left when imagor reboots, let's tackle one problem and then the other one

cshum commented 2 years ago

Thats super strange if the rollback of stream part makes no difference. Thats pretty much the biggest change from v0.9.x to v1.

in which specific version you started to see more crashes? v1.0.0? Or the crashes already happened more frequently within v0.9.x?

Is v0.9.3 the more stable version thur far? What other v0.9.x versions you have previously deployed and kinda feel the same as v0.9.3?

cshum commented 2 years ago

Tbh i can never reproduce the same crash frequency as you did. Though the machines I tested are around 4/8 CPUs.

In my production there were indeed the segmentation violation crashes occurred in v1.0.3, but not anymore in v1.1.0 / master branch. The instances are also 4/8 CPUs but they do scale up after certain CPU/memory threshold.

cocorossello commented 2 years ago

It was also crashing with 0.9.x but less frequently.

I have now removed (file) storage and result storage and it has been working for 10 hours without a crash with vips-thumbnail-disable branch, with the nginx cache in front.

I will try now with v1.1.1, I think that it will also work as well.

cshum commented 2 years ago

Having heard that I may have better clue of what went wrong, at least some part of it. It is not the streaming part but mainly on file system, along with some context cleanup/synchronization bugs.

I don't really use file system in my production setup, which I have probably overlooked some of the file system edge cases under stressed load.

cocorossello commented 2 years ago

I tried 1.1.1 and I had 1 crash, then I went back to vips-thumbnail-disable and I also had 1 crash.

It crashes way more frequently with file storage (also with an increased load since we didn't have the nginx cache), but it also crashes from time to time.

cshum commented 2 years ago

Lets focus on the file system related issues first.

Would you try the current master branch with file storage enabled?

 ghcr.io/cshum/imagor:sha256:b6bc1dade720429a81e153b27497c7decdee4255cbfdc5a483562a3f28841741

https://github.com/cshum/imagor/pkgs/container/imagor/42123058?tag=master

cshum commented 2 years ago

Update: please test with v1.1.3 instead

cocorossello commented 2 years ago

I had 1.1.3 running for 4 hours and it crashed once.

cshum commented 2 years ago

Is it with or without storage?

cocorossello commented 2 years ago

With storage

cshum commented 2 years ago

Updated vips-thumbnail-disable to be based on 1.1.3, see if it makes any difference

ghcr.io/cshum/imagor:sha256:88b8075c90be6f7843ce78db5b2c043b3c0bdef148fcd266d796298ac25d4ad2

https://github.com/cshum/imagor/pkgs/container/imagor/42893514?tag=vips-thumbnail-disable

cocorossello commented 2 years ago

I have deployed all tags (1.1.3, 1.1.4 and vips-thumbnail-disable) and I had crashes with all of them.

Right now I have deployed 1.1.4 without storage and it has crashed once in 18 hours.

I'm not really sure that storage makes a difference. The number of crashes seems very random.

I don't really know how to provide feedback or be useful with this issue, other than saying that it has crashed x times and providing the stacktraces.

Other than 1 or 2 crashes per day, we are quite happy and we can live with this errors

cshum commented 2 years ago

The feedback did help discovered a number of bugs, especially with file storage related. Let's close this issue for now. Lets create a new issue if you see a different patterns more than 1-2 times per day.

cocorossello commented 1 year ago

I haven't had a reboot in 3 days now (since I deployed 1.1.7), so that really looks good. Usually it was crashing once or maybe twice per day

I will post here if there is a crash.

cshum commented 1 year ago

1.1.7 improved memory buffer allocations and it seems to make a difference. Looking good.

cshum commented 1 year ago

1.2.0 should improve it slightly further

tholden92 commented 1 year ago

We have random restarts also, but I have not activated DEBUG in production yet, so I am not sure what is causing it. But so far, we have had 34 restarts. Will report back when I have some logs to reference.