Error writing records to Timestream

andreicezar commented 2 months ago

Describe the bug

Hello! I try to take information from topic 'VirtualGateway' and populate a table in AWS Timestream using a lambda function. I get this error when I am running the test for my lambda function: Error writing records to Timestream: An error occurred (ValidationException) when calling the WriteRecords operation: Invalid time for record.

Expected Behavior

To publish data on Timestream Table 'VirtualGateway_CO2' from database 'VirtualGateway_DB'

Current Behavior

Test Event Name GHG_data_test_CO2

Response

{
  "statusCode": 200,
  "body": "\"Processing complete.\""
}

Received event: {"ghg_data": {"device_id": 3, "ghg_name": "CO2", "value": 131.68, "GWP_100_years": "-", "Unit": "ppb"}, "lorawan_data": {"DeviceEUI": "B9CC5668", "ApplicationEUI": "36618DD7", "GatewayEUI": "8327089B", "RSSI": -36, "SNR": 7.21}, "coordinates": {"latitude": 40.394902, "longitude": -73.853934}, "environmental_conditions": {"temperature": 3.3, "humidity": 73.6, "pressure": 1005.6}, "device_status": {"battery_level": 57.3, "status": "offline"}, "Other_information": {"time": 1723373577, "hostname": "Local Computer", "CPU %": 1.8, "Gas": "O3", "topic": "VirtualGateway"}} Error writing records to Timestream: An error occurred (ValidationException) when calling the WriteRecords operation: Invalid time for record.

Reproduction Steps

Here is the lambda function that I am using to extract data from topic 'VirtualGateway' :ExtractCO2Data-5d02a280-ab68-4389-b178-90ecb6928f15.zip Here is the python code where I am publishing my data: pubsub.zip Here is how a message on this topic looks like:

{
  "ghg_data": {
    "device_id": 3,
    "ghg_name": "CO2",
    "value": 131.68,
    "GWP_100_years": "-",
    "Unit": "ppb"
  },
  "lorawan_data": {
    "DeviceEUI": "B9CC5668",
    "ApplicationEUI": "36618DD7",
    "GatewayEUI": "8327089B",
    "RSSI": -36,
    "SNR": 7.21
  },
  "coordinates": {
    "latitude": 40.394902,
    "longitude": -73.853934
  },
  "environmental_conditions": {
    "temperature": 3.3,
    "humidity": 73.6,
    "pressure": 1005.6
  },
  "device_status": {
    "battery_level": 57.3,
    "status": "offline"
  },
  "Other_information": {
    "time": 1723373577,
    "hostname": "Local Computer",
    "CPU %": 1.8,
    "Gas": "O3",
    "topic": "VirtualGateway"
  }
}

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.34.158

Environment details (OS name and version, etc.)

Windows11

tim-finnigan commented 2 months ago

Thanks for reaching out. That error is coming back from the Timestream WriteRecords API (which the Boto3 write_records command invokes). I think what may be happening is that in your code, the timestamps are getting converted to an unsupported format, similar to what was reported here.

Can you print your generated timestamps or try testing a single record in isolation to narrow down the cause of this issue? If you can share your debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script, then that may also help with investigating the issue.

github-actions[bot] commented 2 months ago

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

boto / boto3