search

aws / aws-iot-fleetwise-edge

Reference Implementation for AWS IoT FleetWise
https://aws.amazon.com/iot-fleetwise/
Apache License 2.0
62 stars 43 forks source link

Error Occurs While Querying Timestream in demo.sh #54

Closed mttwise closed 1 year ago

mttwise commented 1 year ago

Describe the bug While runnning demo.sh following the quickstart readme, Everything runs fine until the "Querying Timestream..." step.

At this point, the following error is outputted to console:

An error occurred (ValidationException) when calling the Query operation: line 1:76: Column 'vehicleName' does not exist

Note: If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page. Please do not create a public github issue.

To Reproduce Follow the steps in the AWS IoT FleetWise cloud demo (https://github.com/aws/aws-iot-fleetwise-edge/blob/main/docs/dev-guide/edge-agent-dev-guide.md#prerequisites-for-quick-start-demo)

Expected behavior Timestream is populated w/ data and is queryable

Logs

[cloudshell-user@ip-xx-xx-xx-xxx ~]$ git clone https://github.com/aws/aws-iot-fleetwise-edge.git ~/aws-iot-fleetwise-edge \
>     && cd ~/aws-iot-fleetwise-edge/tools/cloud \
>     && pip3 install wrapt==1.10.0 plotly==5.3.1 pandas==1.3.5 cantools==36.4.0 boto3==1.18.60 fastparquet==0.8.1
Cloning into '/home/cloudshell-user/aws-iot-fleetwise-edge'...
remote: Enumerating objects: 2868, done.
remote: Counting objects: 100% (360/360), done.
remote: Compressing objects: 100% (199/199), done.
remote: Total 2868 (delta 216), reused 199 (delta 153), pack-reused 2508
Receiving objects: 100% (2868/2868), 2.30 MiB | 18.29 MiB/s, done.
Resolving deltas: 100% (1617/1617), done.
Defaulting to user installation because normal site-packages is not writeable
Collecting wrapt==1.10.0
  Downloading wrapt-1.10.0.tar.gz (24 kB)
Collecting plotly==5.3.1
  Downloading plotly-5.3.1-py2.py3-none-any.whl (23.9 MB)
     |████████████████████████████████| 23.9 MB 54 kB/s 
Collecting pandas==1.3.5
  Downloading pandas-1.3.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.3 MB)
     |████████████████████████████████| 11.3 MB 46.7 MB/s 
Collecting cantools==36.4.0
  Downloading cantools-36.4.0-py3-none-any.whl (102 kB)
     |████████████████████████████████| 102 kB 21.5 MB/s 
Collecting boto3==1.18.60
  Downloading boto3-1.18.60-py3-none-any.whl (131 kB)
     |████████████████████████████████| 131 kB 2.7 MB/s 
Collecting fastparquet==0.8.1
  Downloading fastparquet-0.8.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 43.6 MB/s 
Requirement already satisfied: six in /usr/local/lib/python3.7/site-packages (from plotly==5.3.1) (1.14.0)
Collecting tenacity>=6.2.0
  Downloading tenacity-8.2.2-py3-none-any.whl (24 kB)
Requirement already satisfied: pytz>=2017.3 in /usr/local/lib/python3.7/site-packages (from pandas==1.3.5) (2022.7.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.7/site-packages (from pandas==1.3.5) (2.8.2)
Collecting numpy>=1.17.3; platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10"
  Downloading numpy-1.21.6-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.7 MB)
     |████████████████████████████████| 15.7 MB 58.3 MB/s 
Collecting bitstruct>=6.0.0
  Downloading bitstruct-8.17.0.tar.gz (35 kB)
Collecting argparse-addons
  Downloading argparse_addons-0.12.0-py3-none-any.whl (3.3 kB)
Collecting textparser>=0.21.1
  Downloading textparser-0.24.0-py3-none-any.whl (8.3 kB)
Collecting diskcache
  Downloading diskcache-5.6.1-py3-none-any.whl (45 kB)
     |████████████████████████████████| 45 kB 11.9 MB/s 
Collecting python-can>=2.2.0
  Downloading python_can-4.2.1-py3-none-any.whl (256 kB)
     |████████████████████████████████| 256 kB 47.5 MB/s 
Collecting botocore<1.22.0,>=1.21.60
  Downloading botocore-1.21.65-py3-none-any.whl (8.0 MB)
     |████████████████████████████████| 8.0 MB 32.1 MB/s 
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.7/site-packages (from boto3==1.18.60) (0.10.0)
Collecting s3transfer<0.6.0,>=0.5.0
  Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
     |████████████████████████████████| 79 kB 23.3 MB/s 
Collecting cramjam>=2.3.0
  Downloading cramjam-2.6.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 86.6 MB/s 
Collecting fsspec
  Downloading fsspec-2023.1.0-py3-none-any.whl (143 kB)
     |████████████████████████████████| 143 kB 54.5 MB/s 
Requirement already satisfied: typing-extensions>=3.10.0.0 in /usr/local/lib/python3.7/site-packages (from python-can>=2.2.0->cantools==36.4.0) (4.4.0)
Collecting packaging
  Downloading packaging-23.1-py3-none-any.whl (48 kB)
     |████████████████████████████████| 48 kB 714 kB/s 
Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from python-can>=2.2.0->cantools==36.4.0) (49.1.3)
Collecting msgpack~=1.0.0; platform_system != "Windows"
  Downloading msgpack-1.0.5-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (299 kB)
     |████████████████████████████████| 299 kB 47.2 MB/s 
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.7/site-packages (from botocore<1.22.0,>=1.21.60->boto3==1.18.60) (1.26.15)
Building wheels for collected packages: wrapt, bitstruct
  Building wheel for wrapt (setup.py) ... done
  Created wheel for wrapt: filename=wrapt-1.10.0-py3-none-any.whl size=18395 sha256=1074a75ce85b3f14f648dc90d29d029ff58deba6db4137c3a25c6aeb68f1b0d2
  Stored in directory: /home/cloudshell-user/.cache/pip/wheels/5f/d0/32/01c1d4b7b9a16478d59c69f6215fbaab4bda0fb51a879f7432
  Building wheel for bitstruct (setup.py) ... done
  Created wheel for bitstruct: filename=bitstruct-8.17.0-py3-none-any.whl size=8845 sha256=38d75bad3c30b340b109f68226c4a5dfafd51363b205f6061f80f8091a4ae160
  Stored in directory: /home/cloudshell-user/.cache/pip/wheels/35/f8/42/fcfb5f01fc10ff69957a0964fe5a83920b5941bc5b57befbe2
Successfully built wrapt bitstruct
Installing collected packages: wrapt, tenacity, plotly, numpy, pandas, bitstruct, argparse-addons, textparser, diskcache, packaging, msgpack, python-can, cantools, botocore, s3transfer, boto3, cramjam, fsspec, fastparquet
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

awsebcli 3.20.6 requires botocore<1.29.100,>1.23.41, but you'll have botocore 1.21.65 which is incompatible.
aws-sam-translator 1.60.1 requires boto3==1.*,>=1.19.5, but you'll have boto3 1.18.60 which is incompatible.
aws-sam-cli 1.76.0 requires boto3==1.*,>=1.19.5, but you'll have boto3 1.18.60 which is incompatible.
Successfully installed argparse-addons-0.12.0 bitstruct-8.17.0 boto3-1.18.60 botocore-1.21.65 cantools-36.4.0 cramjam-2.6.2 diskcache-5.6.1 fastparquet-0.8.1 fsspec-2023.1.0 msgpack-1.0.5 numpy-1.21.6 packaging-23.1 pandas-1.3.5 plotly-5.3.1 python-can-4.2.1 s3transfer-0.5.2 tenacity-8.2.2 textparser-0.24.0 wrapt-1.10.0

[cloudshell-user@ip-xx-xx-xx-xxxx cloud]$ ./demo.sh --vehicle-name fwdemo
===================================
AWS IoT FleetWise Cloud Demo Script
===================================
Date: 2023-06-16T13:30:37+0000
Timestamp: 1686922236
Vehicle name: fwdemo
Fleet Size: 1
Checking AWS CLI version...
aws-cli/2.11.26
Getting AWS account ID...
111111111111
Getting account registration status...
NOT_REGISTERED
Registering account...
REGISTRATION_PENDING
Waiting for account to be registered...
Creating Timestream database...
arn:aws:timestream:us-east-1:111111111111:database/IoTFleetWiseDB-1686922236
Creating Timestream table...
Creating service role...
arn:aws:iam::111111111111:role/IoTFleetWiseServiceRole-us-east-1-1686922236
Waiting for role to be created...
Creating service role policy...
arn:aws:iam::111111111111:policy/IoTFleetWiseServiceRole-us-east-1-1686922236-policy
Waiting for policy to be created...
Attaching policy to service role...
Deleting vehicle fwdemo if it already exists...
Checking for existing signal catalog...
arn:aws:iotfleetwise:us-east-1:111111111111:signal-catalog/DefaultSignalCatalog
Updating Vehicle node in signal catalog...
arn:aws:iotfleetwise:us-east-1:111111111111:signal-catalog/DefaultSignalCatalog
Updating OBD signals in signal catalog...
arn:aws:iotfleetwise:us-east-1:111111111111:signal-catalog/DefaultSignalCatalog
Updating DBC signals in signal catalog...
arn:aws:iotfleetwise:us-east-1:111111111111:signal-catalog/DefaultSignalCatalog
Updating color attribute
arn:aws:iotfleetwise:us-east-1:111111111111:signal-catalog/DefaultSignalCatalog
Creating model manifest...
arn:aws:iotfleetwise:us-east-1:111111111111:model-manifest/fwdemo-1686922236-model-manifest
Updating attribute in model manifest...
arn:aws:iotfleetwise:us-east-1:111111111111:model-manifest/fwdemo-1686922236-model-manifest
Activating model manifest...
arn:aws:iotfleetwise:us-east-1:111111111111:model-manifest/fwdemo-1686922236-model-manifest
Creating decoder manifest with OBD signals...
arn:aws:iotfleetwise:us-east-1:111111111111:decoder-manifest/fwdemo-1686922236-decoder-manifest
Adding DBC signals to decoder manifest...
arn:aws:iotfleetwise:us-east-1:111111111111:decoder-manifest/fwdemo-1686922236-decoder-manifest
Activating decoder manifest...
arn:aws:iotfleetwise:us-east-1:111111111111:decoder-manifest/fwdemo-1686922236-decoder-manifest
Creating vehicle fwdemo...
arn:aws:iotfleetwise:us-east-1:111111111111:vehicle/fwdemo
Creating fleet...
arn:aws:iotfleetwise:us-east-1:111111111111:fleet/fwdemo-1686922236-fleet
Associating vehicle fwdemo...
Creating campaign from campaign-brake-event.json...
arn:aws:iotfleetwise:us-east-1:111111111111:campaign/fwdemo-1686922236-campaign
Waiting for campaign to become ready for approval...
arn:aws:iotfleetwise:us-east-1:111111111111:campaign/fwdemo-1686922236-campaign
Approving campaign...
arn:aws:iotfleetwise:us-east-1:111111111111:campaign/fwdemo-1686922236-campaign
Suspending campaign...
arn:aws:iotfleetwise:us-east-1:111111111111:campaign/fwdemo-1686922236-campaign
Resuming campaign...
arn:aws:iotfleetwise:us-east-1:111111111111:campaign/fwdemo-1686922236-campaign
Waiting until status of vehicle fwdemo is healthy...
Waiting 30 seconds for data to be collected...
The DB Name is IoTFleetWiseDB-1686922236
The DB Table is VehicleDataTable
Querying Timestream...

An error occurred (ValidationException) when calling the Query operation: line 1:76: Column 'vehicleName' does not exist

Environment (please complete the following information): AWS CloudShell

Additional context CFN Script was successfully deployed prior to running the demo

image

ruhland commented 1 year ago

Thanks for reporting the issue. Can you please enable logging in the AWS IoT Fleetwise Settings page. Can you please check if there are messages and if yes post them. Can you also please connect to the EC2 running the agent i-0d9b... and look at the edge log sudo journalctl -fu fwe@0 and paste the log of a few seconds. This will help us root cause and fix the issue.

mttwise commented 1 year ago

Thanks for helping out @ruhland

The fix was as follows:

  1. Check the logs from the simulator and noted it started at 20:49:03 UTC. After running 10 minutes, the simulator crashed out. 
    sudo journalctl -u cansim@0
-- Logs begin at Thu 2023-06-15 20:49:03 UTC, end at Fri 2023-06-16 14:40:59 UTC. --
Jun 15 20:54:16 ip-xxxxxxxx systemd[1]: Started CAN Simulator #0.
Jun 15 20:55:39 ip-xxxxxxxx bash[20838]: 2023-06-15 20:54:16.702244 Set BrakePedalPressure to 0
Jun 15 20:55:39 ip-xxxxxxxx bash[20838]: 2023-06-15 20:54:16.702282 Set ENGINE_SPEED to 1100
...
Jun 15 20:59:52 ip-xxxxxxxx bash[20838]: 2023-06-15 20:59:52.019100 Set BrakePedalPressure to 8000
Jun 15 20:59:52 ip-xxxxxxxx bash[20838]: 2023-06-15 20:59:52.019124 Set EngineTorque to 100
Jun 15 21:00:50 ip-xxxxxxxx systemd[1]: setup-socketcan.service: Found ordering cycle on cansim@0.service/stop
Jun 15 21:00:50 ip-xxxxxxxx systemd[1]: setup-socketcan.service: Found dependency on multi-user.target/stop
Jun 15 21:00:50 ip-xxxxxxxx systemd[1]: setup-socketcan.service: Found dependency on setup-socketcan.service/stop
Jun 15 21:00:50 ip-xxxxxxxx systemd[1]: setup-socketcan.service: Job cansim@0.service/stop deleted to break ordering cycle starting with setup-socketcan.service/stop
  1. Restart the simulator service using systemctl 
    sudo systemctl restart cansim@0

After running the service restart command, the simulator starting producing data and I was able to verify output using the candump command

candump vcan0

I was also able to query directly inside of Timestream and view data was there.

ruhland commented 1 year ago

Fix merged with https://github.com/aws/aws-iot-fleetwise-edge/pull/57 if problem persists please reopen