search

nasa / bplib

Apache License 2.0
27 stars 13 forks source link

Configuration changes for improving large file transfer with bpcat #153

Closed jphickey closed 2 years ago

jphickey commented 2 years ago

There are some configuration variables that are not ideal for large transfers with bpcat:

  1. Bundle size - currently set at 3KiB (3072 bytes) with 2552 bytes used for actual data (remainder is reserved for bundle blocks/encoding overhead). Increasing to 16KiB (16834) bytes with the same overhead improves throughput by putting more data into each bundle.
  2. Bundle age out time - currently set at 1 minute. This is the amount of time that the bundles remain in the storage index, before they are finally discarded. Particularly if using a smaller RAM pool, this means storage can hit contention with many bundle records in the deleted state, preventing new bundles from being made, and will not resume until 1 minute later once those records are finally discarded allowing new bundles to be generated again. Reducing this to a small number (e.g. 1 second) on a local transfer improves performance.