Not sure if it's practical(or worth the effort), but is it possible to introduce a mechanism similar to TCP rfc1323 (on slow-start and gradually optimize to proper batch size and can self adapted when storage handling capability changed), which helps optimize buffer size, buffer batch, etc. per each import activities to enable a speed close to the best out-of-box?
This could potentially decouple efforts on tuning those parameters for each nebula cluster(or even in different shape/workload from which the capability to handle import flow could vary)
Not sure if it's practical(or worth the effort), but is it possible to introduce a mechanism similar to TCP rfc1323 (on slow-start and gradually optimize to proper batch size and can self adapted when storage handling capability changed), which helps optimize buffer size, buffer batch, etc. per each import activities to enable
a speed close to the best out-of-box?This could potentially decouple efforts on tuning those parameters for each nebula cluster(or even in different shape/workload from which the capability to handle import flow could vary)