'ait-tlm-db-insert' Influxdb Test

Futabay commented 4 years ago

Precondition:

AIT is installed with virtual environment
Virtual environment is started by typing workon ait on your working terminals
Pcap file needs to be available. For this test, pcap file was created. Please see issues#244 about the pcap file creation.
Influxdb1.2.4 environment needs to be established. For this test, a Dockerfile was created. Please see issues#243 about Dockerfile creation.

Futabay commented 4 years ago

Test Steps:

Set config.yaml as follows:

database:
    dbname: ait

server:
    plugins: 
        - plugin:
            name: ait.gui.AITGUIPlugin
            inputs:
                - log_stream
                - telem_stream
            outputs:
                - command_stream

        - plugin:
            name: ait.core.server.plugins.data_archive.DataArchive
            inputs:
                - telem_stream
                - 8086
            datastore: ait.core.db.InfluxDBBackend

    inbound-streams:
        - stream:
            name: log_stream
            input: 
                - 2514

        - stream:
            name: telem_stream
            input: 
                - 3076

            handlers:
                - name: ait.core.server.handlers.CCSDSPacketHandler
                  packet_types:   {'01011100111' : 'CCSDS_HEADER'}

Open a terminal, start influxdb docker image and influxdb

Go to the Docker file directory and run docker container

$ docker container run -p 8086:8086 -h influxdb --name influxdb_ait -v <your/host/directory>:/var/lib/influxdb -d influxdb

OR if you have the container still in the process, simply start the container.

$ docker ps -a      #check if you have the container
$ docker start <container_name>

Get into influxdb container terminal

$ docker exec -it influxdb_ait /bin/bash

Start influxdb and check available databases


[root@influxdb /]# influx 
Connected to http://localhost:8086 version 1.2.4
InfluxDB shell version: 1.2.4 
> show databases
name: databases
name
----
_internal
ait


Note: For this test, `ait` database exists since the database was already created in [this test steps](https://github.com/NASA-AMMOS/AIT-Core/issues/243#issuecomment-589974082). You can also simply create a database by running `> CREATE DATABASE mydb` command ([https://docs.influxdata.com/influxdb/v1.7/introduction/getting-started/](https://docs.influxdata.com/influxdb/v1.7/introduction/getting-started/))

On the different terminal, start ait-server

(ait) $ ait-server
2020-02-22T21:27:36.378 | WARNING  | Config parameter command.history.filename specifies nonexistent path </some/path/to>/cmdhist.pcap
2020-02-22T21:27:36.380 | WARNING  | Config parameter dsn.cfdp.mib.path specifies nonexistent path </some/path/to>/AIT-Core/config/mib
2020-02-22T21:27:36.382 | WARNING  | Config parameter dsn.cfdp.datasink.outgoing.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/outgoing
2020-02-22T21:27:36.384 | WARNING  | Config parameter dsn.cfdp.datasink.incoming.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/incoming
2020-02-22T21:27:36.386 | WARNING  | Config parameter dsn.cfdp.datasink.tempfiles.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/tempfiles
2020-02-22T21:27:36.388 | WARNING  | Config parameter dsn.cfdp.datasink.pdusink.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/pdusink
2020-02-22T21:27:36.390 | WARNING  | Config parameter data.1553.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/1553
2020-02-22T21:27:36.392 | WARNING  | Config parameter data.bad.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/bad
2020-02-22T21:27:36.394 | WARNING  | Config parameter data.lehx.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/lehx
2020-02-22T21:27:36.396 | WARNING  | Config parameter data.planning.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/planning
2020-02-22T21:27:36.398 | WARNING  | Config parameter data.sdos.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/sdos
2020-02-22T21:27:36.400 | WARNING  | Config parameter data.uplink.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/uplink
2020-02-22T21:27:36.402 | WARNING  | Config parameter data.ats.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/ats
2020-02-22T21:27:37.209 | WARNING  | No handlers specified for stream log_stream
2020-02-22T21:27:37.212 | INFO     | Added inbound stream <PortInputStream address=:2514>
2020-02-22T21:27:37.492 | INFO     | Created handler CCSDSPacketHandler for stream telem_stream
2020-02-22T21:27:37.494 | INFO     | Added inbound stream <PortInputStream address=:3076>
2020-02-22T21:27:37.496 | WARNING  | No handlers specified for stream command_stream
2020-02-22T21:27:37.498 | INFO     | Added outbound stream <PortOutputStream name=command_stream>
2020-02-22T21:27:37.659 | WARNING  | [GUI Playback Configuration]Unable to locate DataArchive plugin configuration for historical data queries. Historical telemetry playback will be disabled in monitoring UI and server endpoints.
2020-02-22T21:27:37.663 | INFO     | Running AIT GUI v1.4.0
2020-02-22T21:27:37.667 | INFO     | Added plugin <Plugin name=AITGUIPlugin>
2020-02-22T21:27:37.670 | WARNING  | No plugin outputs specified for ait.core.server.plugins.data_archive.DataArchive
2020-02-22T21:27:37.808 | INFO     | Starting telemetry data archiving
2020-02-22T21:27:37.812 | INFO     | Added plugin <Plugin name=DataArchive>
2020-02-22T21:27:37.814 | INFO     | Starting <Broker at 0x108263ef0: _run> greenlet...
2020-02-22T21:27:37.816 | INFO     | Starting <Plugin name=AITGUIPlugin> greenlet...
2020-02-22T21:27:37.818 | INFO     | Starting <Plugin name=DataArchive> greenlet...
2020-02-22T21:27:37.820 | INFO     | Starting <PortOutputStream name=command_stream> greenlet...
2020-02-22T21:27:37.821 | INFO     | Starting <PortInputStream address=:2514> greenlet...
2020-02-22T21:27:37.824 | INFO     | Starting <PortInputStream address=:3076> greenlet...
2020-02-22T21:27:37.827 | INFO     | Subscribing <Plugin name=AITGUIPlugin> to topic log_stream
2020-02-22T21:27:37.829 | INFO     | Subscribing <Plugin name=AITGUIPlugin> to topic telem_stream
2020-02-22T21:27:37.831 | INFO     | Subscribing <PortOutputStream name=command_stream> to topic AITGUIPlugin
2020-02-22T21:27:37.833 | INFO     | Subscribing <Plugin name=DataArchive> to topic telem_stream
2020-02-22T21:27:37.835 | INFO     | Subscribing <Plugin name=DataArchive> to topic 8086
2020-02-22T21:27:37.836 | INFO     | Starting broker...

On another terminal, invoke ait-tlm-db-insert command

(ait) $ ait-tlm-db-insert --packet CCSDS_HEADER --backend influx <your/preferred/directory>/<some_name>.pcap
2020-02-22T21:30:45.560 | WARNING  | Config parameter command.history.filename specifies nonexistent path </some/path/to>/cmdhist.pcap
2020-02-22T21:30:45.562 | WARNING  | Config parameter dsn.cfdp.mib.path specifies nonexistent path </some/path/to>/AIT-Core/config/mib
2020-02-22T21:30:45.566 | WARNING  | Config parameter dsn.cfdp.datasink.outgoing.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/outgoing
2020-02-22T21:30:45.568 | WARNING  | Config parameter dsn.cfdp.datasink.incoming.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/incoming
2020-02-22T21:30:45.570 | WARNING  | Config parameter dsn.cfdp.datasink.tempfiles.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/tempfiles
2020-02-22T21:30:45.571 | WARNING  | Config parameter dsn.cfdp.datasink.pdusink.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/pdusink
2020-02-22T21:30:45.573 | WARNING  | Config parameter data.1553.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/1553
2020-02-22T21:30:45.575 | WARNING  | Config parameter data.bad.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/bad
2020-02-22T21:30:45.577 | WARNING  | Config parameter data.lehx.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/lehx
2020-02-22T21:30:45.579 | WARNING  | Config parameter data.planning.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/planning
2020-02-22T21:30:45.581 | WARNING  | Config parameter data.sdos.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/sdos
2020-02-22T21:30:45.583 | WARNING  | Config parameter data.uplink.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/uplink
2020-02-22T21:30:45.585 | WARNING  | Config parameter data.ats.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/ats
2020-02-22T21:30:45.802 | PROGRAM  | <your/directory/to>/.virtualenvs/ait/bin/ait-tlm-db-insert --packet CCSDS_HEADER --backend influx <your/preferred/directory>/<some_name>.pcap
2020-02-22T21:30:45.849 | INFO     | Processing <your/preferred/directory>/<some_name>.pcap
2020-02-22T21:30:45.866 | INFO     | Inserted 1 CCSDS_HEADER packets into database ait.
2020-02-22T21:30:45.868 | PROGRAM  | done.

Go back to Influxdb terminal (docker), verify that the data was inserted in the table


> use ait
Using database ait
> show measurements
name: measurements
name
----
CCSDS_HEADER

select * from CCSDS_HEADER name: CCSDS_HEADER time apid checkword_indicator data_packet element_id format_id frame_id packet_length packet_type secondary_header_flag sequence_count sequence_flags time_coarse time_fine time_id type version version_id zoe

1582428329000000000 743 1 1 1 1 1 11 1 0 0 1 1 1 1 0 0 1 1
6. Type `exit` to stop influxdb and type `exit` to stop influx docker container
exit [root@influxdb /]# exit exit
7. Stop docker container
$ docker stop influxdb_ait
8. Type `Ctrl+C` to stop `ait-server`

Note: When test is done, and you want to remove docker container & image from your host. Remove docker container  $ docker rm influxdb_ait  Remove docker image and dangling images  $ docker rmi influxdb  $ docker rmi $(docker images -f "dangling=true" -q)

Futabay commented 4 years ago

Found bugs: Please see Issues#244 for the information

MJJoyce commented 4 years ago

Looks like relevant issues were closed in #254

NASA-AMMOS / AIT-Core

'ait-tlm-db-insert' Influxdb Test #245