CRUD operations eat all the working memory

[MichaelUlbigPTB]

UseCase and effect

We are trying to send data to FIWARE via CRUD(Post/Patch) operations. We have tried it via the requests package from python and via a curl executions in bash script. The scripts patch about 10 data files every 10 seconds. The effect we see is linear increase of active memory usage over time.

Script Example

#!/bin/bash

BASE_URL="http://example:port/ngsi-ld/v1/entities"

update_l1_values() {
  local device_id=$1
  local apparent_power_l1=$2
  local current_l1=$3
  local phase_voltage_l1=$4
  local reactive_power_l1=$5

   local data=$(cat <<EOF
{
  "https://smartdatamodels.org/dataModel.Energy/apparentPower": {
    "type": "Property",
    "value": {
      "L1": $apparent_power_l1
    }
  },
  "https://smartdatamodels.org/dataModel.Energy/current": {
    "type": "Property",
    "value": {
      "L1": $current_l1
    }
  },
  "https://smartdatamodels.org/dataModel.Energy/phaseVoltage": {
    "type": "Property",
    "value": {
      "L1": $phase_voltage_l1
    }
  },
  "https://smartdatamodels.org/dataModel.Energy/reactivePower": {
    "type": "Property",
    "value": {
      "L1": $reactive_power_l1
    }
  }
}
EOF
  )

  curl -X PATCH "$BASE_URL/$device_id/attrs" -H 'Content-Type: application/json' -d "$data"
}

generate_random_value() {
  local value=$((RANDOM % 251))
  echo "$value"
}

device_ids=(
  "urn:ngsi-ld:TPM:1"
  "urn:ngsi-ld:TPM:2"
  "urn:ngsi-ld:TPM:3"
  "urn:ngsi-ld:TPM:4"
  "urn:ngsi-ld:TPM:5"
  "urn:ngsi-ld:TPM:6"
  "urn:ngsi-ld:TPM:7"
  "urn:ngsi-ld:TPM:8"
  "urn:ngsi-ld:TPM:9"
  "urn:ngsi-ld:TPM:10"
  "urn:ngsi-ld:TPM:11"
  "urn:ngsi-ld:TPM:12"
  "urn:ngsi-ld:TPM:13"
)

while true; do
  for device_id in "${device_ids[@]}"; do
    new_apparent_power_l1=$(generate_random_value)
    new_current_l1=$(generate_random_value)
    new_phase_voltage_l1=$(generate_random_value)
    new_reactive_power_l1=$(generate_random_value)

    update_l1_values "$device_id" "$new_apparent_power_l1" "$new_current_l1" "$new_phase_voltage_l1" "$new_reactive_power_l1"
    echo "$(date +'%Y-%m-%d %H:%M:%S') - Updated L1 values for $device_id"
  done

  sleep 10  # Pause for 10 seconds before next iteration
done 

FIWARE Details

 {
  "orionld version": "post-v1.5.1",
  "orion version": "1.15.0-next",
  "uptime": "0 d, 2 h, 43 m, 49 s",
  "git_hash": "nogitversion",
  "compile_time": "Sun Jan 28 08:20:37 UTC 2024",
  "compiled_by": "root",
  "compiled_in": "",
  "release_date": "Sun Jan 28 08:20:37 UTC 2024",
  "doc": "https://fiware-orion.readthedocs.org/en/master/"
}

Our troubleshooting so far

We have tried so far to excplicitly kill the connections with "Connection" : "Close" as well as response.close(). We also sed log=NONE and disabled the logging from the docker container as well. We also know created the bash script just to check if our python script is the problem. But we can see the same effect.

The onyl thing which seems so far to prevent Orion from crashing is to open up a session over the requests package in combination with a with statement:

with requests.Session() as fiware_session:
  response = fiware_session.patch(url = url, json = json_data_copy, headers = {'Content-Type': 'application/ld+json', 'Connection':'Close'})

But also using sessions is eating our active memory it just doesn't die at the end. Is this a known problem? What are we missing? We are aware of IoT-Agents but we are not sure if this is complient with our security concept and also we try to keep it as simple as possible.