search

aerosense-ai / data-gateway

Data influx for Aerosense.
https://www.aerosense.ai/
Other
3 stars 1 forks source link

Allow configurations of more than 8KiB #87

Closed thclark closed 2 years ago

thclark commented 2 years ago

Bug report

Ingress processing failing

We are experiencing problems getting custom metadata on google cloud storage objects in the cloud function. There may be two reasons for this:

  1. Race condition.

    • The metadata is uploaded to the new object after creation of the object, i.e. in a non-atomic way. the trigger of the cloud function to process the ingress could be occurring before completion of the metadata upload
    • WORKAROUND 1: Add retries to the trigger (although this will leave logs messy from failed first invocation)
    • WORKAROUND 2: Add time.sleep(10) at the start of the cloud function
    • WORKAROUND 3: Add retries within the cloud function, refetching the object until metadata is uploaded
    • SOLUTION: No idea; is there some way of adjusting the trigger?

  2. Metadata too large.

    • Metadata can be ~9kb in some cases, greater than the custom metadata size limit in google cloud. If the upload of the metadata is failing silently because it's too large, that may be why it's missing when we come to process the cloud function.
    • SOLUTION: upload configurations separately.

Example configuration:

{"gateway": {"baudrate": 2300000, "endian": "little", "initial_gateway_handles": {"64": "Local Info Message"}, "installation_reference": "aventa-turbine-test", "latitude": 0, "local_info_types": {"0": "Synchronization not ready as not every sensor node is connected", "1": "Time synchronization info", "2": "Time sync exception", "4": "Time sync coarse data record error", "8": "Time sync alignment error", "16": "Time sync coarse data time diff error", "32": "Device not connected", "64": "Select message destination successful", "128": "Time sync success", "129": "Coarse sync finish", "130": "Time sync msg sent"}, "longitude": 0, "serial_buffer_rx_size": 4095, "serial_buffer_tx_size": 1280, "turbine_id": "unknown", "receiver_firmware_version": "unknown", "packet_key": 254, "packet_key_offset": 245}, "nodes": {"2": {"acc_freq": 100, "acc_range": 16, "analog_freq": 16384, "baros_bm": 1023, "baros_freq": 100, "blade_id": "unknown", "constat_period": 45, "diff_baros_freq": 1000, "gyro_freq": 100, "gyro_range": 2000, "mag_freq": 12.5, "max_timestamp_slack": 0.005, "max_period_drift": 0.02, "mics_bm": 1023, "mics_freq": 15625, "node_firmware_version": "unknown", "decline_reason": {"0": "Bad block detection ongoing", "1": "Task already registered, cannot register again", "2": "Task is not registered, cannot de-register", "3": "Connection parameter update unfinished", "4": "Not ready to sleep", "5": "Not in sleep"}, "initial_node_handles": {"34": "Abs. baros", "36": "Diff. baros", "38": "Mic 0", "40": "Mic 1", "42": "IMU Accel", "44": "IMU Gyro", "46": "IMU Magnetometer", "48": "Analog1", "50": "Analog2", "52": "Constat", "54": "Cmd Decline", "56": "Sleep State", "58": "Remote Info Message", "60": "Timestamp Packet 0", "62": "Timestamp Packet 1", "64": "Local Info Message"}, "remote_info_type": {"0": "Battery info"}, "number_of_sensors": {"Mics": 10, "Baros_P": 40, "Baros_T": 40, "Diff_Baros": 5, "Acc": 3, "Gyro": 3, "Mag": 3, "Analog Vbat": 2, "Constat": 4}, "samples_per_packet": {"Mics": 8, "Diff_Baros": 24, "Baros_P": 1, "Baros_T": 1, "Acc": 40, "Gyro": 40, "Mag": 40, "Analog Vbat": 60, "Constat": 24}, "sensor_commands": {"start": ["startBaros", "startDiffBaros", "startIMU", "startMics"], "stop": ["stopBaros", "stopDiffBaros", "stopIMU", "stopMics"], "configuration": ["configBaros", "configAccel", "configGyro", "configMics"], "utilities": ["getBattery", "setConnInterval", "tpcBoostIncrease", "tpcBoostDecrease", "tpcBoostHeapMemThr1", "tpcBoostHeapMemThr2", "tpcBoostHeapMemThr4"]}, "sensor_conversion_constants": {"Mics": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], "Diff_Baros": [1, 1, 1, 1, 1], "Baros_P": [40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96], "Baros_T": [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100], "Acc": [1, 1, 1], "Gyro": [1, 1, 1], "Mag": [1, 1, 1], "Analog Vbat": [1, 1], "Constat": [1, 1, 1, 1]}, "sensor_names": ["Mics", "Baros_P", "Baros_T", "Diff_Baros", "Acc", "Gyro", "Mag", "Analog Vbat", "Constat"], "sleep_state": {"0": "Exiting sleep", "1": "Entering sleep"}, "sensor_coordinates": {"Mics": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], "Baros_P": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], "Baros_T": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], "Diff_Baros": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], "Acc": [[0, 0, 0], [0, 0, 0], [0, 0, 0]], "Gyro": [[0, 0, 0], [0, 0, 0], [0, 0, 0]], "Mag": [[0, 0, 0], [0, 0, 0], [0, 0, 0]], "Analog Vbat": [[0, 0, 0], [0, 0, 0]], "Constat": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]]}, "periods": {"Mics": 6.4e-05, "Baros_P": 0.01, "Baros_T": 0.01, "Diff_Baros": 0.001, "Acc": 0.01, "Gyro": 0.01, "Mag": 0.08, "Analog Vbat": 6.103515625e-05, "Constat": 0.045}}, "3": {"acc_freq": 100, "acc_range": 16, "analog_freq": 16384, "baros_bm": 1023, "baros_freq": 100, "blade_id": "unknown", "constat_period": 45, "diff_baros_freq": 1000, "gyro_freq": 100, "gyro_range": 2000, "mag_freq": 12.5, "max_timestamp_slack": 0.005, "max_period_drift": 0.02, "mics_bm": 1023, "mics_freq": 15625, "node_firmware_version": "unknown", "decline_reason": {"0": "Bad block detection ongoing", "1": "Task already registered, cannot register again", "2": "Task is not registered, cannot de-register", "3": "Connection parameter update unfinished", "4": "Not ready to sleep", "5": "Not in sleep"}, "initial_node_handles": {"34": "Abs. baros", "36": "Diff. baros", "38": "Mic 0", "40": "Mic 1", "42": "IMU Accel", "44": "IMU Gyro", "46": "IMU Magnetometer", "48": "Analog1", "50": "Analog2", "52": "Constat", "54": "Cmd Decline", "56": "Sleep State", "58": "Remote Info Message", "60": "Timestamp Packet 0", "62": "Timestamp Packet 1", "64": "Local Info Message"}, "remote_info_type": {"0": "Battery info"}, "number_of_sensors": {"Mics": 10, "Baros_P": 40, "Baros_T": 40, "Diff_Baros": 5, "Acc": 3, "Gyro": 3, "Mag": 3, "Analog Vbat": 2, "Constat": 4}, "samples_per_packet": {"Mics": 8, "Diff_Baros": 24, "Baros_P": 1, "Baros_T": 1, "Acc": 40, "Gyro": 40, "Mag": 40, "Analog Vbat": 60, "Constat": 24}, "sensor_commands": {"start": ["startBaros", "startDiffBaros", "startIMU", "startMics"], "stop": ["stopBaros", "stopDiffBaros", "stopIMU", "stopMics"], "configuration": ["configBaros", "configAccel", "configGyro", "configMics"], "utilities": ["getBattery", "setConnInterval", "tpcBoostIncrease", "tpcBoostDecrease", "tpcBoostHeapMemThr1", "tpcBoostHeapMemThr2", "tpcBoostHeapMemThr4"]}, "sensor_conversion_constants": {"Mics": [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], "Diff_Baros": [1, 1, 1, 1, 1], "Baros_P": [40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96, 40.96], "Baros_T": [100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100], "Acc": [1, 1, 1], "Gyro": [1, 1, 1], "Mag": [1, 1, 1], "Analog Vbat": [1, 1], "Constat": [1, 1, 1, 1]}, "sensor_names": ["Mics", "Baros_P", "Baros_T", "Diff_Baros", "Acc", "Gyro", "Mag", "Analog Vbat", "Constat"], "sleep_state": {"0": "Exiting sleep", "1": "Entering sleep"}, "sensor_coordinates": {"Mics": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], "Baros_P": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], "Baros_T": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], "Diff_Baros": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]], "Acc": [[0, 0, 0], [0, 0, 0], [0, 0, 0]], "Gyro": [[0, 0, 0], [0, 0, 0], [0, 0, 0]], "Mag": [[0, 0, 0], [0, 0, 0], [0, 0, 0]], "Analog Vbat": [[0, 0, 0], [0, 0, 0]], "Constat": [[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]]}, "periods": {"Mics": 6.4e-05, "Baros_P": 0.01, "Baros_T": 0.01, "Diff_Baros": 0.001, "Acc": 0.01, "Gyro": 0.01, "Mag": 0.08, "Analog Vbat": 6.103515625e-05, "Constat": 0.045}}}, "session": {"label": null, "notes": "Aventa wind turbine tests setup (in the lab)"}}
cortadocodes commented 2 years ago

Race condition This is solved in https://github.com/octue/octue-sdk-python/pull/494.

Metadata size limit I've just tested uploading a file with metadata that's the contents of a JSON configuration file 12 KB in size (greater than the 8 KiB limit) - it seems to have populated the custom metadata in the storage browser fine. Take a look here and let me know if you think it's still an issue. I would have thought we'd be getting some kind of error if we were hitting the (what seems to be different in practice than in theory) metadata size limit.