search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
31.05k stars 2.95k forks source link

[Bug]: build diskann failed in cluster mode #25961

Closed lbj1110 closed 1 year ago

lbj1110 commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus version:2.2.6
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka): Kafka
- SDK version(e.g. pymilvus v2.0.0rc2):2.2.6
- OS(Ubuntu or CentOS): ubuntu
- CPU/Memory: unrestricted
- GPU: 
- Others:

Current Behavior

When I create a diskann on a dataset of 5 million, the dataCoord node keeps saying, "there is no IndexNode available or etc is not serviceable" and "index builder peek client error, there is no available", and with the dataNode constantly restarting frequently, etcd node status is normal

Expected Behavior

No response

Steps To Reproduce

1. Map the /milvus/data paths on dataNodes and queryNode nodes to ssd of the server, and complete the ssd resource mounting; 
2. MILVUS service cluster deployment to K8S platform; 
3. Insert 5 million data into MILVUS; 
4. Establish the diskAnn index

Milvus Log

indexCoord: [2023/07/27 09:04:09.048 +00:00] [INFO] [log/mlogger.go:72] ["index builder peek client error, there is no available"] [2023/07/27 09:04:09.048 +00:00] [INFO] [indexcoord/index_builder.go:178] ["there is no IndexNode available or etcd is not serviceable, wait a minute..."]

etcd: {"level":"warn","ts":"2023-07-27T08:32:37.091Z","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"2.367137918s","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"by-dev/meta/config\" range_end:\"by-dev/meta/confih\" ","response":"range_response_count:0 size:6"}

Anything else?

nothing

yanliang567 commented 1 year ago

sounds like the milvus service is not running healthy. could you please double check the etcd service is working? /assign @lbj1110 /unassign

lbj1110 commented 1 year ago

There are still several problems to determine:

  1. Whether the data path of indexNode and queryNode is /var/lib/milvus/data or /milvus/data, the official website says the former, but this path is not found in the actual indexNode service container;
  2. What does it mean that the memory index like ivf_sq8 can be established, but the diskann cannot be established?
  3. Can you provide a direction for troubleshooting? I don't have a clear troubleshooting direction now, if you need me to provide any logs and other information, I can provide it
xiaofan-luan commented 1 year ago

There are still several problems to determine:

  1. Whether the data path of indexNode and queryNode is /var/lib/milvus/data or /milvus/data, the official website says the former, but this path is not found in the actual indexNode service container;
  2. What does it mean that the memory index like ivf_sq8 can be established, but the diskann cannot be established?
  3. Can you provide a direction for troubleshooting? I don't have a clear troubleshooting direction now, if you need me to provide any logs and other information, I can provide it

DiskANN index build require much larger memory than IVF indexes and takes longer. For serving diskann, you will also need to have high performance local ssds. In order to use diskANN, your indexNode need to have at least 16GB memory or more. Try to increate indexnode memory or reduce segment size to see if it helps

lbj1110 commented 1 year ago

I increased the ssd space of indexNode, the current configuration is about (ssd: 40G, memory: 16G) * 4 nodes, now indexCoord will not report the error "there is no IndexNode available or etc is not serviceable", but the indexNode node will still restart from time to time, why? I'll attach the indexNode error log and also want to ask, how long does it take about 5 million to create a diskann index? error log: 2023/07/28 03:28:20 maxprocs: Leaving GOMAXPROCS=128: CPU quota undefined

__  _________ _   ____  ______

/ |/ / _/ /| | / / / / / /
/ /|/ // // /| |/ / // /\ \
// ///__/_/__/_/

Welcome to use Milvus! Version: v2.2.6-1-g0a1a5ba8 Built: Fri Apr 14 04:43:52 UTC 2023 GitCommit: 0a1a5ba8 GoVersion: go version go1.18.3 linux/amd64

open pid file: /run/milvus/indexnode.pid lock pid file: /run/milvus/indexnode.pid [2023/07/28 03:28:20.957 +00:00] [INFO] [roles/roles.go:192] ["starting running Milvus components"] [2023/07/28 03:28:20.957 +00:00] [INFO] [roles/roles.go:134] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/libjemalloc.so] [2023/07/28 03:28:20.957 +00:00] [INFO] [management/server.go:68] ["management listen"] [addr=:9091] [2023/07/28 03:28:20.985 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/07/28 03:28:20.985 +00:00] [INFO] [paramtable/quota_param.go:532] ["init disk quota"] [diskQuota(MB)=+inf] [2023/07/28 03:28:20.985 +00:00] [INFO] [paramtable/component_param.go:1402] ["init segment max idle time"] [value=10m0s] [2023/07/28 03:28:20.985 +00:00] [INFO] [paramtable/component_param.go:1407] ["init segment min size from idle to sealed"] [value=16] [2023/07/28 03:28:20.985 +00:00] [INFO] [paramtable/component_param.go:1412] ["init segment max binlog file to sealed"] [value=32] [2023/07/28 03:28:20.988 +00:00] [INFO] [paramtable/base_table.go:142] ["cannot find etcd.endpoints"] [2023/07/28 03:28:20.988 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}] [2023/07/28 03:28:20.988 +00:00] [INFO] [indexnode/indexnode.go:107] ["New IndexNode ..."] [2023/07/28 03:28:21.017 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/07/28 03:28:21.018 +00:00] [DEBUG] [paramtable/grpc_param.go:153] [initServerMaxSendSize] [role=indexnode] [grpc.serverMaxSendSize=536870912] [2023/07/28 03:28:21.018 +00:00] [DEBUG] [paramtable/grpc_param.go:175] [initServerMaxRecvSize] [role=indexnode] [grpc.serverMaxRecvSize=536870912] [2023/07/28 03:28:21.019 +00:00] [INFO] [indexnode/service.go:85] [IndexNode] ["network address"=10.233.94.255:21121] ["network port: "=21121] [2023/07/28 03:28:21.150 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/07/28 03:28:21.150 +00:00] [INFO] [paramtable/quota_param.go:532] ["init disk quota"] [diskQuota(MB)=+inf] [2023/07/28 03:28:21.150 +00:00] [INFO] [paramtable/component_param.go:1402] ["init segment max idle time"] [value=10m0s] [2023/07/28 03:28:21.151 +00:00] [INFO] [paramtable/component_param.go:1407] ["init segment min size from idle to sealed"] [value=16] [2023/07/28 03:28:21.151 +00:00] [INFO] [paramtable/component_param.go:1412] ["init segment max binlog file to sealed"] [value=32] [2023/07/28 03:28:21.153 +00:00] [INFO] [paramtable/base_table.go:142] ["cannot find etcd.endpoints"] [2023/07/28 03:28:21.153 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}] [2023/07/28 03:28:21.154 +00:00] [DEBUG] [indexnode/indexnode.go:193] ["IndexNode init"] [State=Initializing] [2023/07/28 03:28:21.154 +00:00] [INFO] [sessionutil/session_util.go:197] ["Session try to connect to etcd"] [2023/07/28 03:28:21.155 +00:00] [INFO] [sessionutil/session_util.go:212] ["Session connect to etcd success"] [2023/07/28 03:28:21.159 +00:00] [INFO] [sessionutil/session_util.go:294] ["Session get serverID success"] [key=id] [ServerId=1208] [2023/07/28 03:28:21.192 +00:00] [INFO] [config/etcd_source.go:145] ["start refreshing configurations"] [2023/07/28 03:28:21.192 +00:00] [INFO] [paramtable/quota_param.go:532] ["init disk quota"] [diskQuota(MB)=+inf] [2023/07/28 03:28:21.193 +00:00] [INFO] [paramtable/component_param.go:1402] ["init segment max idle time"] [value=10m0s] [2023/07/28 03:28:21.193 +00:00] [INFO] [paramtable/component_param.go:1407] ["init segment min size from idle to sealed"] [value=16] [2023/07/28 03:28:21.193 +00:00] [INFO] [paramtable/component_param.go:1412] ["init segment max binlog file to sealed"] [value=32] [2023/07/28 03:28:21.195 +00:00] [INFO] [paramtable/base_table.go:142] ["cannot find etcd.endpoints"] [2023/07/28 03:28:21.195 +00:00] [INFO] [paramtable/hook_config.go:19] ["hook config"] [hook={}] [2023/07/28 03:28:21.195 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=] [2023/07/28 03:28:21.195 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=] 2023-07-28 03:28:21,196 INFO [default] [KNOWHERE][SetBlasThreshold][milvus] Set faiss::distance_compute_blas_threshold to 16384 2023-07-28 03:28:21,196 INFO [default] [KNOWHERE][SetEarlyStopThreshold][milvus] Set faiss::early_stop_threshold to 0 2023-07-28 03:28:21,196 INFO [default] [KNOWHERE][SetStatisticsLevel][milvus] Set knowhere::STATISTICS_LEVEL to 0 2023-07-28 03:28:21,197 | DEBUG | default | [SERVER][operator()][milvus] Config easylogging with yaml file: /milvus/configs/easylogging.yaml 2023-07-28 03:28:21,197 | INFO | default | [KNOWHERE][SetSimdType][milvus] FAISS expect simdType::AUTO 2023-07-28 03:28:21,197 | INFO | default | [KNOWHERE][SetSimdType][milvus] FAISS hook AVX512 2023-07-28 03:28:21,197 | DEBUG | default | [SEGCORE][SetIndexSliceSize][milvus] set config index slice size: 16 2023-07-28 03:28:21,197 | DEBUG | default | [SEGCORE][SetThreadCoreCoefficient][milvus] set thread pool core coefficient: 10 [2023/07/28 03:28:21.197 +00:00] [INFO] [indexnode/service.go:73] ["IndexNode init done ..."] [2023/07/28 03:28:21.197 +00:00] [INFO] [sessionutil/session_util.go:344] ["service begin to register to etcd"] [serverName=indexnode] [ServerID=1208] [2023/07/28 03:28:21.198 +00:00] [INFO] [indexnode/task_scheduler.go:240] ["IndexNode TaskScheduler start build loop ..."] [2023/07/28 03:28:21.201 +00:00] [INFO] [sessionutil/session_util.go:381] ["put session key into etcd"] [key=by-dev/meta/session/indexnode-1208] [value="{\"ServerID\":1208,\"ServerName\":\"indexnode\",\"Address\":\"10.233.94.255:21121\",\"TriggerKill\":true,\"Version\":\"2.2.5-pre+dev\"}"] [2023/07/28 03:28:21.201 +00:00] [INFO] [sessionutil/session_util.go:392] ["Service registered successfully"] [ServerName=indexnode] [serverID=1208] [2023/07/28 03:28:21.201 +00:00] [INFO] [indexnode/indexnode.go:131] ["IndexNode Register Finished"] [2023/07/28 03:28:21.203 +00:00] [INFO] [indexnode/service.go:193] ["IndexNode Register etcd success"] [2023/07/28 03:28:21.203 +00:00] [INFO] [indexnode/service.go:77] ["IndexNode start done ..."] [2023/07/28 03:28:21.216 +00:00] [INFO] [sessionutil/session_util.go:734] ["register session success"] [role=indexnode] [key=by-dev/meta/session/indexnode-1208] [2023/07/28 03:28:21.392 +00:00] [INFO] [indexnode/indexnode_service.go:164] ["drop index build jobs"] [traceID=3e958cf379b5c2a2] [ClusterID=by-dev] [IndexBuildIDs="[443157959289281492]"] [2023/07/28 03:28:21.748 +00:00] [INFO] [indexnode/indexnode_service.go:209] ["Get Index Job Stats"] [traceID=381567b71580bad5] [Unissued=0] [Active=0] [Slot=1] [2023/07/28 03:28:21.755 +00:00] [INFO] [indexnode/indexnode_service.go:47] ["IndexNode building index ..."] [traceID=78d2104a0eeb5657] [ClusterID=by-dev] [IndexBuildID=443157959289281492] [IndexID=0] [IndexName=] [IndexFilePrefix=file/index_files] [IndexVersion=4] [DataPaths="[file/insert_log/443157959289280462/443157959289280463/443157959289280842/101/443157959289280853]"] [TypeParams="[{\"key\":\"dim\",\"value\":\"512\"}]"] [IndexParams="[{\"key\":\"max_degree\",\"value\":\"56\"},{\"key\":\"num_build_thread_ratio\",\"value\":\"1.000000\"},{\"key\":\"search_cache_budget_gb_ratio\",\"value\":\"0.100000\"},{\"key\":\"pq_code_budget_gb_ratio\",\"value\":\"0.125000\"},{\"key\":\"num_load_thread_ratio\",\"value\":\"8.000000\"},{\"key\":\"beamwidth_ratio\",\"value\":\"4.000000\"},{\"key\":\"metric_type\",\"value\":\"IP\"},{\"key\":\"index_type\",\"value\":\"DISKANN\"},{\"key\":\"search_list_size\",\"value\":\"100\"}]"] [num_rows=14908] [2023/07/28 03:28:21.779 +00:00] [INFO] [storage/minio_chunk_manager.go:135] ["minio chunk manager init success."] [bucketname=milvus-bucket] [root=file] [2023/07/28 03:28:21.779 +00:00] [INFO] [indexnode/chunk_mgr_factory.go:33] ["index node successfully init chunk manager"] [2023/07/28 03:28:21.779 +00:00] [INFO] [indexnode/indexnode_service.go:112] ["IndexNode successfully scheduled"] [traceID=78d2104a0eeb5657] [IndexBuildID=443157959289281492] [ClusterID=by-dev] [indexName=] [2023/07/28 03:28:21.779 +00:00] [INFO] [indexnode/task.go:155] ["Begin to prepare indexBuildTask"] [buildID=443157959289281492] [Collection=0] [SegmentIf=0] [2023/07/28 03:28:21.779 +00:00] [INFO] [indexnode/task.go:182] ["Successfully prepare indexBuildTask"] [buildID=443157959289281492] [Collection=0] [SegmentIf=0] [2023/07/28 03:28:22.261 +00:00] [INFO] [indexnode/task.go:237] ["Successfully load data"] [buildID=443157959289281492] [Collection=443157959289280462] [SegmentIf=443157959289280842] [2023/07/28 03:28:22.275 +00:00] [INFO] [indexnode/task.go:359] ["disk index params are ready"] [buildID=443157959289281492] ["index params"="{\"beamwidth_ratio\":\"4.000000\",\"build_dram_budget_gb\":\"503.221619\",\"collection_id\":\"443157959289280462\",\"dim\":\"512\",\"field_id\":\"101\",\"index_build_id\":\"443157959289281492\",\"index_id\":\"0\",\"index_type\":\"DISKANN\",\"index_version\":\"4\",\"max_degree\":\"56\",\"metric_type\":\"IP\",\"num_build_thread\":\"128\",\"num_build_thread_ratio\":\"1.000000\",\"num_load_thread_ratio\":\"8.000000\",\"partition_id\":\"443157959289280463\",\"pq_code_budget_gb\":\"0.003554\",\"pq_code_budget_gb_ratio\":\"0.125000\",\"search_cache_budget_gb_ratio\":\"0.100000\",\"search_list_size\":\"100\",\"segment_id\":\"443157959289280842\"}"] 2023-07-28 03:28:24,287 | INFO | default | [SEGCORE][N6milvus7storage17MinioChunkManagerE::MinioChunkManager][milvus] init MinioChunkManager with parameter[endpoint: 'xxx-xxx-minio:9000', default_bucket_name:'milvus-bucket', use_secure:'false']xxx-xxx 2023-07-28 03:28:24,311 | INFO | default | Using Inner Product search, so need to pre-process base data into temp file. Please ensure there is additional (n(d+1)4) bytes for storing pre-processed base vectors, apart from the intermin indices and final index. 2023-07-28 03:28:24,400 | INFO | default | Starting index build: R=56 L=100 Query RAM budget: 0.003554(GiB) Indexing ram budget: 503.222(GiB) T: 128 2023-07-28 03:28:24,400 | INFO | default | Compressing 513-dimensional data into 255 bytes per vector. 2023-07-28 03:28:24,522 | INFO | default | Loaded train data from raw data file.#train_size: 14908, #train_dim: 513 2023-07-28 03:28:24,522 | INFO | default | Training pivots of PQ chunks. 2023-07-28 03:28:25,605 | INFO | default | Processing PQ data from pivots. 2023-07-28 03:28:25,884 | INFO | default | Training PQ codes cost: 1.36241s 2023-07-28 03:28:25,887 | INFO | default | Full index fits in RAM budget, should consume at most 0.0369474GiBs, so building in one shot 2023-07-28 03:28:25,908 | INFO | default | Building start. Loading only first 14908 from file.. 2023-07-28 03:28:25,908 | INFO | default | Generate graph: 2023-07-28 03:28:35,457 | INFO | default | Completed Pass 0 of data using L=80 and alpha=1. Stats: search+prune_time=2.35807s, inter_time=4.91845s, inter_count=6408 [2023/07/28 03:28:35.749 +00:00] [INFO] [indexnode/indexnode_service.go:209] ["Get Index Job Stats"] [traceID=6f1fbd20623f65a0] [Unissued=0] [Active=1] [Slot=0] 2023-07-28 03:28:49,852 | INFO | default | Completed Pass 1 of data using L=100 and alpha=1.2. Stats: search+prune_time=3.60779s, inter_time=8.01097s, inter_count=17034 2023-07-28 03:28:51,311 | INFO | default | Index built with degree: max:56 avg:56 min:56 count(deg<2):0 2023-07-28 03:28:51,321 | INFO | default | Training graph cost: 25.4341s 2023-07-28 03:28:51,340 | INFO | default | Vamana index file size: 3399048 2023-07-28 03:28:51,427 | INFO | default | Indexing time: 27.0265s 2023-07-28 03:28:51,427 | INFO | default | [SEGCORE][N6milvus10ThreadPoolE::ThreadPool][milvus] Thread pool's worker num:1280 runtime/cgo: pthread_create failed: Resource temporarily unavailable SIGABRT: abort PC=0x7fbd11c9c00b m=3 sigcode=18446744073709551610

goroutine 0 [idle]: runtime: unknown pc 0x7fbd11c9c00b stack: frame={sp:0x7fbae4c75440, fp:0x0} stack=[0x7fbae4475eb8,0x7fbae4c75ab8) 0x00007fbae4c75340: 0x0000000000000000 0x0000000000000007 0x00007fbae4c75350: 0x0000000000000007 0x0000000000000007 0x00007fbae4c75360: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75370: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75380: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75390: 0x0000000000000000 0x000000000142de4d <runtime.park_m+0x000000000000014d> 0x00007fbae4c753a0: 0x0000000000000006 0x0000000000000000 0x00007fbae4c753b0: 0x00007fbae4c756d0 0x000000000141b36a <runtime.mProf_Malloc+0x000000000000006a> 0x00007fbae4c753c0: 0x0000000000000000 0x0000000000000000 0x00007fbae4c753d0: 0x0000000000000000 0x0000000000000000 0x00007fbae4c753e0: 0x0000000000000000 0x0000000001457463 <runtime.mcall+0x0000000000000043> 0x00007fbae4c753f0: 0x0000000800026370 0x0000000000000000 0x00007fbae4c75400: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75410: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75420: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75430: 0x0000000000000000 0x00000000013fa987 <runtime.newobject+0x0000000000000027> 0x00007fbae4c75440: <0x0000000000000000 0x0000000000000000 0x00007fbae4c75450: 0x0000000000000000 0x000000000142d365 <runtime.resetspinning+0x0000000000000045> 0x00007fbae4c75460: 0x00000018000187c8 0x000000000142de4c <runtime.park_m+0x000000000000014c> 0x00007fbae4c75470: 0xffffffff00018d57 0x000000000141b36a <runtime.mProf_Malloc+0x000000000000006a> 0x00007fbae4c75480: 0x00000168000123cf 0x000000000142bb9a <runtime.wakep+0x000000000000005a> 0x00007fbae4c75490: 0x0000001800017e1a 0x0000000000000000 0x00007fbae4c754a0: 0x0000000000000000 0x000000000142de4d <runtime.park_m+0x000000000000014d> 0x00007fbae4c754b0: 0x0000002800018cf2 0x000000000142d91d <runtime.schedule+0x000000000000025d> 0x00007fbae4c754c0: 0xfffffffe7fffffff 0xffffffffffffffff 0x00007fbae4c754d0: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c754e0: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c754f0: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c75500: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c75510: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c75520: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c75530: 0xffffffffffffffff 0xffffffffffffffff runtime: unknown pc 0x7fbd11c9c00b stack: frame={sp:0x7fbae4c75440, fp:0x0} stack=[0x7fbae4475eb8,0x7fbae4c75ab8) 0x00007fbae4c75340: 0x0000000000000000 0x0000000000000007 0x00007fbae4c75350: 0x0000000000000007 0x0000000000000007 0x00007fbae4c75360: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75370: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75380: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75390: 0x0000000000000000 0x000000000142de4d <runtime.park_m+0x000000000000014d> 0x00007fbae4c753a0: 0x0000000000000006 0x0000000000000000 0x00007fbae4c753b0: 0x00007fbae4c756d0 0x000000000141b36a <runtime.mProf_Malloc+0x000000000000006a> 0x00007fbae4c753c0: 0x0000000000000000 0x0000000000000000 0x00007fbae4c753d0: 0x0000000000000000 0x0000000000000000 0x00007fbae4c753e0: 0x0000000000000000 0x0000000001457463 <runtime.mcall+0x0000000000000043> 0x00007fbae4c753f0: 0x0000000800026370 0x0000000000000000 0x00007fbae4c75400: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75410: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75420: 0x0000000000000000 0x0000000000000000 0x00007fbae4c75430: 0x0000000000000000 0x00000000013fa987 <runtime.newobject+0x0000000000000027> 0x00007fbae4c75440: <0x0000000000000000 0x0000000000000000 0x00007fbae4c75450: 0x0000000000000000 0x000000000142d365 <runtime.resetspinning+0x0000000000000045> 0x00007fbae4c75460: 0x00000018000187c8 0x000000000142de4c <runtime.park_m+0x000000000000014c> 0x00007fbae4c75470: 0xffffffff00018d57 0x000000000141b36a <runtime.mProf_Malloc+0x000000000000006a> 0x00007fbae4c75480: 0x00000168000123cf 0x000000000142bb9a <runtime.wakep+0x000000000000005a> 0x00007fbae4c75490: 0x0000001800017e1a 0x0000000000000000 0x00007fbae4c754a0: 0x0000000000000000 0x000000000142de4d <runtime.park_m+0x000000000000014d> 0x00007fbae4c754b0: 0x0000002800018cf2 0x000000000142d91d <runtime.schedule+0x000000000000025d> 0x00007fbae4c754c0: 0xfffffffe7fffffff 0xffffffffffffffff 0x00007fbae4c754d0: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c754e0: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c754f0: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c75500: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c75510: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c75520: 0xffffffffffffffff 0xffffffffffffffff 0x00007fbae4c75530: 0xffffffffffffffff 0xffffffffffffffff

goroutine 1 [chan receive, 2 minutes]: github.com/milvus-io/milvus/cmd/roles.(MilvusRoles).Run(0xc00091fe58, 0x0, {0x0, 0x0}) /go/src/github.com/milvus-io/milvus/cmd/roles/roles.go:317 +0xb8d github.com/milvus-io/milvus/cmd/milvus.(run).execute(0xc0009bfe60, {0xc000296060?, 0x3, 0x3}, 0xc0008ea6c0) /go/src/github.com/milvus-io/milvus/cmd/milvus/run.go:112 +0x66e github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000296060?, 0x3, 0x3}) /go/src/github.com/milvus-io/milvus/cmd/milvus/milvus.go:60 +0x21e main.main() /go/src/github.com/milvus-io/milvus/cmd/main.go:26 +0x2e

goroutine 142 [chan receive, 2 minutes]: github.com/panjf2000/ants/v2.(*Pool).purgePeriodically(0xc00047a770) /go/pkg/mod/github.com/panjf2000/ants/v2@v2.4.8/pool.go:69 +0x8b created by github.com/panjf2000/ants/v2.NewPool /go/pkg/mod/github.com/panjf2000/ants/v2@v2.4.8/pool.go:137 +0x34a

goroutine 223 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19ec8, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc0008d2b00?, 0xc000064500?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Accept(0xc0008d2b00) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x22c net.(netFD).accept(0xc0008d2b00) /usr/local/go/src/net/fd_unix.go:172 +0x35 net.(TCPListener).accept(0xc0009f8540) /usr/local/go/src/net/tcpsock_posix.go:139 +0x28 net.(TCPListener).Accept(0xc0009f8540) /usr/local/go/src/net/tcpsock.go:288 +0x3d net/http.(Server).Serve(0xc0008dc1c0, {0x426baa0, 0xc0009f8540}) /usr/local/go/src/net/http/server.go:3039 +0x385 net/http.(Server).ListenAndServe(0xc0008dc1c0) /usr/local/go/src/net/http/server.go:2968 +0x7d net/http.ListenAndServe(...) /usr/local/go/src/net/http/server.go:3222 github.com/milvus-io/milvus/internal/management.ServeHTTP.func1() /go/src/github.com/milvus-io/milvus/internal/management/server.go:69 +0x151 created by github.com/milvus-io/milvus/internal/management.ServeHTTP /go/src/github.com/milvus-io/milvus/internal/management/server.go:66 +0x25

goroutine 225 [select, 2 minutes]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00107f280) /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x73 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22a

goroutine 310 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19928, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc0010c6180?, 0xc00092a000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc0010c6180, {0xc00092a000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc0010c6180, {0xc00092a000?, 0x7fbd0da90618?, 0x1427807?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc000622008, {0xc00092a000?, 0x7fbd0db15a68?, 0x801010601?}) /usr/local/go/src/net/net.go:183 +0x45 bufio.(Reader).Read(0xc000b0a1e0, {0xc000428120, 0x9, 0x18?}) /usr/local/go/src/bufio/bufio.go:236 +0x1b4 io.ReadAtLeast({0x4251aa0, 0xc000b0a1e0}, {0xc000428120, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 golang.org/x/net/http2.readFrameHeader({0xc000428120?, 0x9?, 0x1bd0e5ff9b?}, {0x4251aa0?, 0xc000b0a1e0?}) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e golang.org/x/net/http2.(Framer).ReadFrame(0xc0004280e0) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95 google.golang.org/grpc/internal/transport.(http2Client).reader(0xc0012c5680) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:1498 +0x414 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:365 +0x193f

goroutine 324 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19ce8, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc0008d2d80?, 0xc00121a000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc0008d2d80, {0xc00121a000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc0008d2d80, {0xc00121a000?, 0x3cb9520?, 0x200000001?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc0008a07a0, {0xc00121a000?, 0x0?, 0x802010601?}) /usr/local/go/src/net/net.go:183 +0x45 bufio.(Reader).Read(0xc000215d40, {0xc0008dc2e0, 0x9, 0x18?}) /usr/local/go/src/bufio/bufio.go:236 +0x1b4 io.ReadAtLeast({0x4251aa0, 0xc000215d40}, {0xc0008dc2e0, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 golang.org/x/net/http2.readFrameHeader({0xc0008dc2e0?, 0x9?, 0x1bf6a4bd25?}, {0x4251aa0?, 0xc000215d40?}) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e golang.org/x/net/http2.(Framer).ReadFrame(0xc0008dc2a0) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95 google.golang.org/grpc/internal/transport.(http2Client).reader(0xc001232000) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:1498 +0x414 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:365 +0x193f

goroutine 308 [select, 2 minutes]: github.com/milvus-io/milvus/internal/config.(EtcdSource).refreshConfigurationsPeriodically(0xc0010c7f00) /go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:147 +0x9f created by github.com/milvus-io/milvus/internal/config.(EtcdSource).GetConfigurations.func1 /go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:98 +0x5a

goroutine 325 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc0007c00f0, 0x1) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x115 google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc001234000) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85 google.golang.org/grpc/internal/transport.newHTTP2Client.func3() /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:415 +0x65 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:413 +0x1f91

goroutine 326 [syscall, 2 minutes]: os/signal.signal_recv() /usr/local/go/src/runtime/sigqueue.go:151 +0x2f os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x19 created by os/signal.Notify.func1.1 /usr/local/go/src/os/signal/signal.go:151 +0x2a

goroutine 327 [select, 2 minutes]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0007bc940) /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x73 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22a

goroutine 357 [select, 2 minutes]: github.com/milvus-io/milvus/internal/config.(EtcdSource).refreshConfigurationsPeriodically(0xc0007d6580) /go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:147 +0x9f created by github.com/milvus-io/milvus/internal/config.(EtcdSource).GetConfigurations.func1 /go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:98 +0x5a

goroutine 331 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5200) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 164 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19bf8, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc00037cb00?, 0xc0005b4000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc00037cb00, {0xc0005b4000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc00037cb00, {0xc0005b4000?, 0x7fbd0da8e750?, 0x7?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc00029db10, {0xc0005b4000?, 0x7fbd0db15f18?, 0x802010601?}) /usr/local/go/src/net/net.go:183 +0x45 bufio.(Reader).Read(0xc0012342a0, {0xc0008dc3c0, 0x9, 0x18?}) /usr/local/go/src/bufio/bufio.go:236 +0x1b4 io.ReadAtLeast({0x4251aa0, 0xc0012342a0}, {0xc0008dc3c0, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 golang.org/x/net/http2.readFrameHeader({0xc0008dc3c0?, 0x9?, 0x1bf888b235?}, {0x4251aa0?, 0xc0012342a0?}) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e golang.org/x/net/http2.(Framer).ReadFrame(0xc0008dc380) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95 google.golang.org/grpc/internal/transport.(http2Client).reader(0xc0000001e0) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:1498 +0x414 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:365 +0x193f

goroutine 165 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc000ad60f0, 0x1) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x115 google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc001234300) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85 google.golang.org/grpc/internal/transport.newHTTP2Client.func3() /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:415 +0x65 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:413 +0x1f91

goroutine 156 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc000a566e0, 0x1) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x115 google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc000b72360) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85 google.golang.org/grpc/internal/transport.newHTTP2Client.func3() /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:415 +0x65 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:413 +0x1f91

goroutine 148 [select, 2 minutes]: github.com/milvus-io/milvus/internal/config.(EtcdSource).refreshConfigurationsPeriodically(0xc0008d2200) /go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:147 +0x9f created by github.com/milvus-io/milvus/internal/config.(EtcdSource).GetConfigurations.func1 /go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:98 +0x5a

goroutine 149 [select, 2 minutes]: github.com/uber/jaeger-client-go.(RemotelyControlledSampler).pollControllerWithTicker(0xc000298c30, 0xc000ad61e0) /go/pkg/mod/github.com/uber/jaeger-client-go@v2.25.0+incompatible/sampler_remote.go:144 +0x89 github.com/uber/jaeger-client-go.(RemotelyControlledSampler).pollController(0xc000298c30) /go/pkg/mod/github.com/uber/jaeger-client-go@v2.25.0+incompatible/sampler_remote.go:139 +0x6d created by github.com/uber/jaeger-client-go.NewRemotelyControlledSampler /go/pkg/mod/github.com/uber/jaeger-client-go@v2.25.0+incompatible/sampler_remote.go:86 +0x15b

goroutine 151 [select, 2 minutes]: github.com/uber/jaeger-client-go/utils.(*reconnectingUDPConn).reconnectLoop(0xc000af4070, 0x0?) /go/pkg/mod/github.com/uber/jaeger-client-go@v2.25.0+incompatible/utils/reconnecting_udp_conn.go:70 +0xbc created by github.com/uber/jaeger-client-go/utils.newReconnectingUDPConn /go/pkg/mod/github.com/uber/jaeger-client-go@v2.25.0+incompatible/utils/reconnecting_udp_conn.go:60 +0x205

goroutine 152 [select, 2 minutes]: github.com/uber/jaeger-client-go.(*remoteReporter).processQueue(0xc000b722a0) /go/pkg/mod/github.com/uber/jaeger-client-go@v2.25.0+incompatible/reporter.go:296 +0xde created by github.com/uber/jaeger-client-go.NewRemoteReporter /go/pkg/mod/github.com/uber/jaeger-client-go@v2.25.0+incompatible/reporter.go:237 +0x245

goroutine 153 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19b08, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc000a07080?, 0xc000068f00?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Accept(0xc000a07080) /usr/local/go/src/internal/poll/fd_unix.go:614 +0x22c net.(netFD).accept(0xc000a07080) /usr/local/go/src/net/fd_unix.go:172 +0x35 net.(TCPListener).accept(0xc000b52168) /usr/local/go/src/net/tcpsock_posix.go:139 +0x28 net.(TCPListener).Accept(0xc000b52168) /usr/local/go/src/net/tcpsock.go:288 +0x3d google.golang.org/grpc.(Server).Serve(0xc0008bb180, {0x426baa0?, 0xc000b52168}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:780 +0x477 github.com/milvus-io/milvus/internal/distributed/indexnode.(Server).startGrpcLoop(0xc0008eb680, 0x5281) /go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:120 +0xa2b created by github.com/milvus-io/milvus/internal/distributed/indexnode.(*Server).init /go/src/github.com/milvus-io/milvus/internal/distributed/indexnode/service.go:151 +0x3c5

goroutine 111 [select, 2 minutes]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000ae1340) /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x73 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22a

goroutine 155 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19a18, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc00037ce00?, 0xc000a88000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc00037ce00, {0xc000a88000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc00037ce00, {0xc000a88000?, 0x3cb9520?, 0x1467301?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc000b36828, {0xc000a88000?, 0x0?, 0x802010601?}) /usr/local/go/src/net/net.go:183 +0x45 bufio.(Reader).Read(0xc000b72300, {0xc00067e040, 0x9, 0x18?}) /usr/local/go/src/bufio/bufio.go:236 +0x1b4 io.ReadAtLeast({0x4251aa0, 0xc000b72300}, {0xc00067e040, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 golang.org/x/net/http2.readFrameHeader({0xc00067e040?, 0x9?, 0x1c0080b6f3?}, {0x4251aa0?, 0xc000b72300?}) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e golang.org/x/net/http2.(Framer).ReadFrame(0xc00067e000) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95 google.golang.org/grpc/internal/transport.(http2Client).reader(0xc0012325a0) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:1498 +0x414 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:365 +0x193f

goroutine 362 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19838, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc0013c0280?, 0xc0013f6000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc0013c0280, {0xc0013f6000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc0013c0280, {0xc0013f6000?, 0x3cb9520?, 0x1467301?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc0011f2f18, {0xc0013f6000?, 0x0?, 0x802010601?}) /usr/local/go/src/net/net.go:183 +0x45 bufio.(Reader).Read(0xc000acfe00, {0xc0013d0040, 0x9, 0x18?}) /usr/local/go/src/bufio/bufio.go:236 +0x1b4 io.ReadAtLeast({0x4251aa0, 0xc000acfe00}, {0xc0013d0040, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 golang.org/x/net/http2.readFrameHeader({0xc0013d0040?, 0x9?, 0x1c02fe2ad3?}, {0x4251aa0?, 0xc000acfe00?}) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e golang.org/x/net/http2.(Framer).ReadFrame(0xc0013d0000) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95 google.golang.org/grpc/internal/transport.(http2Client).reader(0xc0013f0000) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:1498 +0x414 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:365 +0x193f

goroutine 363 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc0007c1bd0, 0x1) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x115 google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc000acfe60) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85 google.golang.org/grpc/internal/transport.newHTTP2Client.func3() /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:415 +0x65 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:413 +0x1f91

goroutine 354 [select, 2 minutes]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000a7ac80) /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x73 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22a

goroutine 312 [select, 2 minutes]: github.com/milvus-io/milvus/internal/config.(EtcdSource).refreshConfigurationsPeriodically(0xc0013c0000) /go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:147 +0x9f created by github.com/milvus-io/milvus/internal/config.(EtcdSource).GetConfigurations.func1 /go/src/github.com/milvus-io/milvus/internal/config/etcd_source.go:98 +0x5a

goroutine 358 [select, 2 minutes]: google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0011f8840) /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:112 +0x73 created by google.golang.org/grpc.newCCBalancerWrapper /go/pkg/mod/google.golang.org/grpc@v1.46.0/balancer_conn_wrappers.go:73 +0x22a

goroutine 311 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc0002cc370, 0x1) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x115 google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc000b0a2a0) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85 google.golang.org/grpc/internal/transport.newHTTP2Client.func3() /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:415 +0x65 created by google.golang.org/grpc/internal/transport.newHTTP2Client /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_client.go:413 +0x1f91

goroutine 173 [select, 2 minutes]: go.etcd.io/etcd/client/v3.(lessor).keepAliveCtxCloser(0xc000b0e460, {0x426d870, 0xc0008d5b40}, 0x20ba1a6?, 0xc0005b2fc0) /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:330 +0xd9 created by go.etcd.io/etcd/client/v3.(lessor).KeepAlive /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:297 +0x58a

goroutine 330 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5140) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 174 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(recvBufferReader).readClient(0xc0002cd4f0, {0xc00088ead8, 0x5, 0x5}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/transport.go:191 +0xa5 google.golang.org/grpc/internal/transport.(recvBufferReader).Read(0xc0002cd4f0, {0xc00088ead8?, 0xc001427e00?, 0xc00156fa28?}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/transport.go:171 +0x147 google.golang.org/grpc/internal/transport.(transportReader).Read(0xc0014c26f0, {0xc00088ead8?, 0xc00156faa0?, 0x1918687?}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/transport.go:485 +0x32 io.ReadAtLeast({0x4255c60, 0xc0014c26f0}, {0xc00088ead8, 0x5, 0x5}, 0x5) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 google.golang.org/grpc/internal/transport.(Stream).Read(0xc0014baea0, {0xc00088ead8, 0x5, 0x5}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/transport.go:469 +0xa5 google.golang.org/grpc.(parser).recvMsg(0xc00088eac8, 0x7fffffff) /go/pkg/mod/google.golang.org/grpc@v1.46.0/rpc_util.go:559 +0x47 google.golang.org/grpc.recvAndDecompress(0xc00156fc70?, 0xc0014baea0, {0x0, 0x0}, 0x7fffffff, 0x0, {0x0, 0x0}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/rpc_util.go:690 +0x66 google.golang.org/grpc.recv(0x1427840?, {0x7fbade8d2778, 0x58b8900}, 0xc00156fce0?, {0x0?, 0x0?}, {0x3d34620, 0xc0028ab700}, 0xc0005fc000?, 0x0, ...) /go/pkg/mod/google.golang.org/grpc@v1.46.0/rpc_util.go:756 +0x6e google.golang.org/grpc.(csAttempt).recvMsg(0xc0008f4d10, {0x3d34620?, 0xc0028ab700}, 0xc0028ab700?) /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:983 +0x2b0 google.golang.org/grpc.(clientStream).RecvMsg.func1(0x5858c00?) /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:834 +0x25 google.golang.org/grpc.(clientStream).withRetry(0xc0014bac60, 0xc00156fe90, 0xc00156fe60) /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:688 +0x2f6 google.golang.org/grpc.(clientStream).RecvMsg(0xc0014bac60, {0x3d34620?, 0xc0028ab700?}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:833 +0x11f go.etcd.io/etcd/api/v3/etcdserverpb.(leaseLeaseKeepAliveClient).Recv(0xc0014b4c50) /go/pkg/mod/go.etcd.io/etcd/api/v3@v3.5.5/etcdserverpb/rpc.pb.go:6853 +0x4c go.etcd.io/etcd/client/v3.(lessor).recvKeepAliveLoop(0xc000b0e460) /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:456 +0x28f created by go.etcd.io/etcd/client/v3.(lessor).KeepAlive.func1 /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:299 +0x65

goroutine 313 [semacquire, 2 minutes]: sync.runtime_Semacquire(0xc001854450?) /usr/local/go/src/runtime/sema.go:56 +0x25 sync.(WaitGroup).Wait(0xc00107f940?) /usr/local/go/src/sync/waitgroup.go:136 +0x52 github.com/milvus-io/milvus/internal/indexnode.(TaskScheduler).indexBuildLoop(0xc00107f940) /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:256 +0xa5 created by github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).Start /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:264 +0x6a

goroutine 175 [select, 2 minutes]: go.etcd.io/etcd/client/v3.(lessor).deadlineLoop(0xc000b0e460) /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:548 +0x85 created by go.etcd.io/etcd/client/v3.(lessor).KeepAlive.func1 /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:300 +0xa5

goroutine 176 [select, 2 minutes]: github.com/milvus-io/milvus/internal/util/sessionutil.(Session).processKeepAliveResponse.func1() /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:410 +0xf3 created by github.com/milvus-io/milvus/internal/util/sessionutil.(Session).processKeepAliveResponse /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:407 +0xbb

goroutine 314 [select, 2 minutes]: google.golang.org/grpc.newClientStreamWithParams.func4() /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:341 +0x92 created by google.golang.org/grpc.newClientStreamWithParams /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:340 +0xb70

goroutine 315 [select]: go.etcd.io/etcd/client/v3.(lessor).sendKeepAliveLoop(0xc000b0e460, {0x427b208, 0xc0014b4c50}) /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:588 +0x1ed created by go.etcd.io/etcd/client/v3.(lessor).resetRecv /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/lease.go:498 +0x2d8

goroutine 177 [select, 2 minutes]: context.propagateCancel.func1() /usr/local/go/src/context/context.go:279 +0x9e created by context.propagateCancel /usr/local/go/src/context/context.go:278 +0x1cb

goroutine 370 [select, 2 minutes]: go.etcd.io/etcd/client/v3.(watchGrpcStream).run(0xc000b44dd0) /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/watch.go:537 +0x354 created by go.etcd.io/etcd/client/v3.(watcher).newWatcherGrpcStream /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/watch.go:285 +0x34d

goroutine 373 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(recvBufferReader).readClient(0xc000ad7680, {0xc00000f390, 0x5, 0x5}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/transport.go:191 +0xa5 google.golang.org/grpc/internal/transport.(recvBufferReader).Read(0xc000ad7680, {0xc00000f390?, 0xc000af0408?, 0xc0005ada48?}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/transport.go:171 +0x147 google.golang.org/grpc/internal/transport.(transportReader).Read(0xc0005fe360, {0xc00000f390?, 0xc0005adac0?, 0x1918687?}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/transport.go:485 +0x32 io.ReadAtLeast({0x4255c60, 0xc0005fe360}, {0xc00000f390, 0x5, 0x5}, 0x5) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 google.golang.org/grpc/internal/transport.(Stream).Read(0xc0005defc0, {0xc00000f390, 0x5, 0x5}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/transport.go:469 +0xa5 google.golang.org/grpc.(parser).recvMsg(0xc00000f380, 0x7fffffff) /go/pkg/mod/google.golang.org/grpc@v1.46.0/rpc_util.go:559 +0x47 google.golang.org/grpc.recvAndDecompress(0xc00066ac90?, 0xc0005defc0, {0x0, 0x0}, 0x7fffffff, 0x0, {0x0, 0x0}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/rpc_util.go:690 +0x66 google.golang.org/grpc.recv(0x0?, {0x7fbade8d2778, 0x58b8900}, 0x0?, {0x0?, 0x0?}, {0x3d64fe0, 0xc0009809a0}, 0x0?, 0x0, ...) /go/pkg/mod/google.golang.org/grpc@v1.46.0/rpc_util.go:756 +0x6e google.golang.org/grpc.(csAttempt).recvMsg(0xc000787290, {0x3d64fe0?, 0xc0009809a0}, 0xc0009809a0?) /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:983 +0x2b0 google.golang.org/grpc.(clientStream).RecvMsg.func1(0xc001235a40?) /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:834 +0x25 google.golang.org/grpc.(clientStream).withRetry(0xc0005ded80, 0xc0005adeb0, 0xc0005ade80) /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:688 +0x2f6 google.golang.org/grpc.(clientStream).RecvMsg(0xc0005ded80, {0x3d64fe0?, 0xc0009809a0?}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:833 +0x11f go.etcd.io/etcd/api/v3/etcdserverpb.(watchWatchClient).Recv(0xc0005e28a0) /go/pkg/mod/go.etcd.io/etcd/api/v3@v3.5.5/etcdserverpb/rpc.pb.go:6714 +0x4c go.etcd.io/etcd/client/v3.(watchGrpcStream).serveWatchClient(0xc000b44dd0, {0x427b310, 0xc0005e28a0}) /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/watch.go:757 +0x6a created by go.etcd.io/etcd/client/v3.(watchGrpcStream).newWatchClient /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/watch.go:912 +0x5dc

goroutine 372 [select, 2 minutes]: google.golang.org/grpc.newClientStreamWithParams.func4() /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:341 +0x92 created by google.golang.org/grpc.newClientStreamWithParams /go/pkg/mod/google.golang.org/grpc@v1.46.0/stream.go:340 +0xb70

goroutine 374 [select, 2 minutes]: go.etcd.io/etcd/client/v3.(watchGrpcStream).serveSubstream(0xc000b44dd0, 0xc000787340, 0xc0005b33e0) /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/watch.go:803 +0x265 created by go.etcd.io/etcd/client/v3.(watchGrpcStream).run /go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.5/watch.go:554 +0xe65

goroutine 375 [select, 2 minutes]: github.com/milvus-io/milvus/internal/util/sessionutil.(Session).LivenessCheck.func1() /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:683 +0x165 created by github.com/milvus-io/milvus/internal/util/sessionutil.(Session).LivenessCheck /go/src/github.com/milvus-io/milvus/internal/util/sessionutil/session_util.go:680 +0xce

goroutine 365 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc0007c1cc0, 0x1) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x115 google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc001235c80) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:332 +0xc6 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:329 +0x19b3

goroutine 366 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001092680) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:1131 +0x233 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:341 +0x19f8

goroutine 367 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19748, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc0013c0380?, 0xc001580000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc0013c0380, {0xc001580000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc0013c0380, {0xc001580000?, 0xc0005f5d00?, 0x13f27ed?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc0011f2f20, {0xc001580000?, 0xbd0db155b8?, 0xc001384838?}) /usr/local/go/src/net/net.go:183 +0x45 bufio.(Reader).Read(0xc0002142a0, {0xc0013d0120, 0x9, 0xc0005f5d30?}) /usr/local/go/src/bufio/bufio.go:236 +0x1b4 io.ReadAtLeast({0x4251aa0, 0xc0002142a0}, {0xc0013d0120, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 golang.org/x/net/http2.readFrameHeader({0xc0013d0120?, 0x9?, 0x19592dc928?}, {0x4251aa0?, 0xc0002142a0?}) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e golang.org/x/net/http2.(Framer).ReadFrame(0xc0013d00e0) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95 google.golang.org/grpc/internal/transport.(http2Server).HandleStreams(0xc001092680, 0xc000b29190?, 0xc0012352c0?) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:611 +0xb2 google.golang.org/grpc.(Server).serveStreams(0xc0008bb180, {0x427e170?, 0xc001092680}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:906 +0x142 google.golang.org/grpc.(Server).handleRawConn.func1() /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:848 +0x46 created by google.golang.org/grpc.(*Server).handleRawConn /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:847 +0x185

goroutine 369 [syscall, 2 minutes]: github.com/milvus-io/milvus/internal/util/indexcgowrapper._Cfunc_BuildFloatVecIndex(0x7fbadd003060, 0x747800, 0xc009506000) _cgo_gotypes.go:130 +0x55 github.com/milvus-io/milvus/internal/util/indexcgowrapper.(CgoIndex).buildFloatVecIndex.func1(0x3a85fe0?, {0xc009506000, 0x747800, 0xb?}) /go/src/github.com/milvus-io/milvus/internal/util/indexcgowrapper/index.go:158 +0x69 github.com/milvus-io/milvus/internal/util/indexcgowrapper.(CgoIndex).buildFloatVecIndex(0xc0007ac220?, 0x0?) /go/src/github.com/milvus-io/milvus/internal/util/indexcgowrapper/index.go:158 +0x65 github.com/milvus-io/milvus/internal/util/indexcgowrapper.(CgoIndex).Build(0x426d870?, 0x3e5da0d?) /go/src/github.com/milvus-io/milvus/internal/util/indexcgowrapper/index.go:130 +0x1a6 github.com/milvus-io/milvus/internal/indexnode.(indexBuildTask).BuildDiskAnnIndex(0xc0005dcf20, {0x426d870, 0xc0017d87c0}) /go/src/github.com/milvus-io/milvus/internal/indexnode/task.go:367 +0xe4c github.com/milvus-io/milvus/internal/indexnode.(indexBuildTask).BuildIndex(0xc0005dcf20, {0x426d870, 0xc0017d87c0}) /go/src/github.com/milvus-io/milvus/internal/indexnode/task.go:248 +0xa5 github.com/milvus-io/milvus/internal/indexnode.(TaskScheduler).processTask.func1(0xc000521e00) /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:207 +0x82 github.com/milvus-io/milvus/internal/indexnode.(TaskScheduler).processTask(0xc00107f940, {0x427ff58, 0xc0005dcf20}, {0x0?, 0x0?}) /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:220 +0x40c github.com/milvus-io/milvus/internal/indexnode.(TaskScheduler).indexBuildLoop.func1(0x171fcb6?, {0x427ff58?, 0xc0005dcf20?}) /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:253 +0x6c created by github.com/milvus-io/milvus/internal/indexnode.(*TaskScheduler).indexBuildLoop /go/src/github.com/milvus-io/milvus/internal/indexnode/task_scheduler.go:251 +0x298

goroutine 332 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e52c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 333 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5380) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 334 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5440) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 335 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e55c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 336 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5680) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 337 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5740) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 418 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5800) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 419 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e58c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 420 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5980) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 421 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5a40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 422 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5b00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 423 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5bc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 424 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5c80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 425 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5d40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 426 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5e00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 427 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0002e5ec0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 428 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970000) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 429 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970180) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 430 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970240) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 431 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970300) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 432 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0009703c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 433 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970480) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 434 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970540) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 435 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970600) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 436 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0009706c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 437 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970780) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 438 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970840) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 439 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970900) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 440 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0009709c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 441 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970a80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 442 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970b40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 443 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970c00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 444 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970cc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 445 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970d80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 446 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970e40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 447 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970f00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 448 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000970fc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 449 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971080) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 450 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971140) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 451 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971200) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 452 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0009712c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 453 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971380) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 454 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971440) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 455 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971500) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 456 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0009715c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 457 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971680) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 458 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971740) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 459 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971800) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 460 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0009718c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 461 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971980) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 462 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971a40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 463 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971b00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 464 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971bc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 465 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971c80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 466 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971d40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 467 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971e00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 468 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000971ec0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 469 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0005340c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 470 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534240) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 471 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534300) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 472 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534480) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 473 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534540) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 474 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534600) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 475 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0005346c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 476 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534780) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 477 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534840) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 478 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534900) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 479 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0005349c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 480 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534a80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 481 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534b40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 482 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534c00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 483 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534cc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 484 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534d80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 485 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534e40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 486 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534f00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 487 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000534fc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 488 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535080) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 489 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535140) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 490 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535200) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 491 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0005352c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 492 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535380) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 493 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535440) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 494 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535500) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 495 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0005355c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 496 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535680) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 497 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535740) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 498 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535800) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 499 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0005358c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 500 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535980) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 501 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535a40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 502 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535b00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 503 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535bc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 504 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535c80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 505 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535d40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 506 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535e00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 507 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000535ec0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 508 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0008083c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 509 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808480) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 510 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808540) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 511 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808600) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 512 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0008086c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 513 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808780) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 514 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808840) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 515 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808900) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 516 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0008089c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 517 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808a80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 518 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808b40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 519 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808c00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 520 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808cc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 521 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808d80) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 522 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808e40) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 523 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808f00) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 524 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000808fc0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 525 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809080) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 526 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809140) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 527 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809200) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 528 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0008092c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 529 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809380) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 530 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809440) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 531 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809500) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 532 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0008095c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 533 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809680) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 534 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809740) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 535 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809800) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 536 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0008098c0) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 537 [chan receive, 2 minutes]: github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000809980) /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:211 +0x94 created by github.com/klauspost/compress/zstd.newBlockDec /go/pkg/mod/github.com/klauspost/compress@v1.14.2/zstd/blockdec.go:117 +0x185

goroutine 381 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19298, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc0008d2400?, 0xc0017ea000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc0008d2400, {0xc0017ea000, 0x8000, 0x8000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc0008d2400, {0xc0017ea000?, 0xc000bb1d00?, 0x13f27ed?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc000b36b60, {0xc0017ea000?, 0xbd0db15108?, 0xc000640ef8?}) /usr/local/go/src/net/net.go:183 +0x45 bufio.(Reader).Read(0xc0003e5320, {0xc00067e580, 0x9, 0xc000bb1d30?}) /usr/local/go/src/bufio/bufio.go:236 +0x1b4 io.ReadAtLeast({0x4251aa0, 0xc0003e5320}, {0xc00067e580, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:331 +0x9a io.ReadFull(...) /usr/local/go/src/io/io.go:350 golang.org/x/net/http2.readFrameHeader({0xc00067e580?, 0x9?, 0x1994c7c1bc?}, {0x4251aa0?, 0xc0003e5320?}) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:237 +0x6e golang.org/x/net/http2.(Framer).ReadFrame(0xc00067e540) /go/pkg/mod/golang.org/x/net@v0.0.0-20220127200216-cd36cc0744dd/http2/frame.go:498 +0x95 google.golang.org/grpc/internal/transport.(http2Server).HandleStreams(0xc0008ab1e0, 0xc00055ef38?, 0x261bba6?) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:611 +0xb2 google.golang.org/grpc.(Server).serveStreams(0xc0008bb180, {0x427e170?, 0xc0008ab1e0}) /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:906 +0x142 google.golang.org/grpc.(Server).handleRawConn.func1() /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:848 +0x46 created by google.golang.org/grpc.(*Server).handleRawConn /go/pkg/mod/google.golang.org/grpc@v1.46.0/server.go:847 +0x185

goroutine 538 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19388, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc0017f6680?, 0xc000b16000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc0017f6680, {0xc000b16000, 0x1000, 0x1000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc0017f6680, {0xc000b16000?, 0x182cc60?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc0011200a8, {0xc000b16000?, 0x8?, 0x11e?}) /usr/local/go/src/net/net.go:183 +0x45 net/http.(connReader).Read(0xc001011e00, {0xc000b16000, 0x1000, 0x1000}) /usr/local/go/src/net/http/server.go:780 +0x16d bufio.(Reader).fill(0xc0014e0600) /usr/local/go/src/bufio/bufio.go:106 +0x103 bufio.(Reader).ReadSlice(0xc0014e0600, 0x0?) /usr/local/go/src/bufio/bufio.go:371 +0x2f bufio.(Reader).ReadLine(0xc0014e0600) /usr/local/go/src/bufio/bufio.go:400 +0x27 net/textproto.(Reader).readLineSlice(0xc000c01d10) /usr/local/go/src/net/textproto/reader.go:57 +0x99 net/textproto.(Reader).ReadLine(...) /usr/local/go/src/net/textproto/reader.go:38 net/http.readRequest(0xc0011200a8?) /usr/local/go/src/net/http/request.go:1029 +0x79 net/http.(conn).readRequest(0xc000b0e0a0, {0x426d870, 0xc0008e4840}) /usr/local/go/src/net/http/server.go:988 +0x24a net/http.(conn).serve(0xc000b0e0a0, {0x426d918, 0xc000b11bc0}) /usr/local/go/src/net/http/server.go:1891 +0x32b created by net/http.(Server).Serve /usr/local/go/src/net/http/server.go:3071 +0x4db

goroutine 380 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0008ab1e0) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:1131 +0x233 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:341 +0x19f8

goroutine 394 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7fbd0da19478, 0x72) /usr/local/go/src/runtime/netpoll.go:302 +0x89 internal/poll.(pollDesc).wait(0xc0010c6300?, 0xc0018e5000?, 0x0) /usr/local/go/src/internal/poll/fd_poll_runtime.go:83 +0x32 internal/poll.(pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:88 internal/poll.(FD).Read(0xc0010c6300, {0xc0018e5000, 0x1000, 0x1000}) /usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a net.(netFD).Read(0xc0010c6300, {0xc0018e5000?, 0x182cc60?, 0x0?}) /usr/local/go/src/net/fd_posix.go:55 +0x29 net.(conn).Read(0xc0008a02a8, {0xc0018e5000?, 0x8?, 0x11e?}) /usr/local/go/src/net/net.go:183 +0x45 net/http.(connReader).Read(0xc001518a80, {0xc0018e5000, 0x1000, 0x1000}) /usr/local/go/src/net/http/server.go:780 +0x16d bufio.(Reader).fill(0xc0018d7980) /usr/local/go/src/bufio/bufio.go:106 +0x103 bufio.(Reader).ReadSlice(0xc0018d7980, 0x0?) /usr/local/go/src/bufio/bufio.go:371 +0x2f bufio.(Reader).ReadLine(0xc0018d7980) /usr/local/go/src/bufio/bufio.go:400 +0x27 net/textproto.(Reader).readLineSlice(0xc00146c840) /usr/local/go/src/net/textproto/reader.go:57 +0x99 net/textproto.(Reader).ReadLine(...) /usr/local/go/src/net/textproto/reader.go:38 net/http.readRequest(0xc0008a02a8?) /usr/local/go/src/net/http/request.go:1029 +0x79 net/http.(conn).readRequest(0xc0012fe960, {0x426d870, 0xc00150b3c0}) /usr/local/go/src/net/http/server.go:988 +0x24a net/http.(conn).serve(0xc0012fe960, {0x426d918, 0xc000b11bc0}) /usr/local/go/src/net/http/server.go:1891 +0x32b created by net/http.(Server).Serve /usr/local/go/src/net/http/server.go:3071 +0x4db

goroutine 379 [select, 2 minutes]: google.golang.org/grpc/internal/transport.(controlBuffer).get(0xc0014ae820, 0x1) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:407 +0x115 google.golang.org/grpc/internal/transport.(loopyWriter).run(0xc0014e0ea0) /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/controlbuf.go:534 +0x85 google.golang.org/grpc/internal/transport.NewServerTransport.func2() /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:332 +0xc6 created by google.golang.org/grpc/internal/transport.NewServerTransport /go/pkg/mod/google.golang.org/grpc@v1.46.0/internal/transport/http2_server.go:329 +0x19b3

rax 0x0 rbx 0x7fbae4c8c700 rcx 0xffffffffffffffff rdx 0x0 rdi 0x2 rsi 0x7fbae4c75440 rbp 0x42d3d28 rsp 0x7fbae4c75440 r8 0x0 r9 0x7fbae4c75440 r10 0x8 r11 0x246 r12 0x7fbae3800040 r13 0x178 r14 0xc000002ea0 r15 0x7fbae5820c75 rip 0x7fbd11c9c00b rflags 0x246 cs 0x33 fs 0x0 gs 0x0

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.