Set Bandwidth limit for event downloads

[nicolemah99]

• [x] Add option in config to set bandwidth rate limit for data_pipeline.sh (downloading event files)

Something like this:

#!/bin/bash

# Define FTP server and credentials
HOST='ftp.example.com'
USER='username'
PASS='password'
REMOTE_DIR='/path/to/remote/dir'
LOCAL_DIR='/path/to/local/dir'
BANDWIDTH_LIMIT='100k'

# Execute lftp command with bandwidth throttling
lftp -u $USER,$PASS $HOST <<EOF
set net:limit-rate $BANDWIDTH_LIMIT
mirror --continue --verbose $REMOTE_DIR $LOCAL_DIR
bye
EOF

• [x] Make a new release

  git tag v0.1.0
  git push --tags

• [x] Update cookbook for new release
  • Update release version
  • Update templates/databags for bandwith rate limit in config

[nicolemah99]

The bandwidth limit peeps over the limit but for example 200K/s doesn't go over 300K/s

Downloading event 10143

bandwidth_limit = 50k --> 3:09 bandwidth_limit = 100k --> 1:52 bandwidth_limit = 200k --> 1:16 bandwidth_limit = 300k --> 1:06

[nicolemah99]

new release v0.1.10