'ait-tlm-db-insert' SQLite3 Test

Futabay commented 4 years ago

Precondition:

AIT is installed with virtual environment
Virtual environment is started by typing workon ait on your working terminals
SQLite3 in installed on a testing machine.

Pcap file needs to be available. For this test, a pcap file was created as follows:

Create an empty .pcap file in your preferred directory
Open a terminal and start python

Create a pcap file with using ait.core.pcap

(ait) $ python
Python 3.7.6 (default, Dec 30 2019, 19:38:28) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ait.core.pcap as pcap
2020-02-22T19:24:52.037 | WARNING  | Config parameter command.history.filename specifies nonexistent path </some/path/to>/cmdhist.pcap
2020-02-22T19:24:52.040 | WARNING  | Config parameter dsn.cfdp.mib.path specifies nonexistent path </some/path/to>/AIT-Core/config/mib
2020-02-22T19:24:52.042 | WARNING  | Config parameter dsn.cfdp.datasink.outgoing.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/outgoing
2020-02-22T19:24:52.044 | WARNING  | Config parameter dsn.cfdp.datasink.incoming.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/incoming
2020-02-22T19:24:52.046 | WARNING  | Config parameter dsn.cfdp.datasink.tempfiles.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/tempfiles
2020-02-22T19:24:52.048 | WARNING  | Config parameter dsn.cfdp.datasink.pdusink.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/pdusink
2020-02-22T19:24:52.050 | WARNING  | Config parameter data.1553.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/1553
2020-02-22T19:24:52.052 | WARNING  | Config parameter data.bad.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/bad
2020-02-22T19:24:52.054 | WARNING  | Config parameter data.lehx.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/lehx
2020-02-22T19:24:52.056 | WARNING  | Config parameter data.planning.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/planning
2020-02-22T19:24:52.058 | WARNING  | Config parameter data.sdos.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/sdos
2020-02-22T19:24:52.060 | WARNING  | Config parameter data.uplink.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/uplink
2020-02-22T19:24:52.061 | WARNING  | Config parameter data.ats.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/ats
>>> foo = pcap.open('<your/preferred/directory>/<some_name>.pcap', 'w')
>>> data = bytearray(b'\x02\xE7\x40\x00\x00\x0B\x00\x00\x00\x01\x01\x71\x0C\x41\x00\x01')
>>> foo.write(data)
16

Note: Please see issues#240comment about packet data info.

Futabay commented 4 years ago

Test Steps:

Set config.yaml as follows:

database:
    dbname: </Pass/to/ait.db>

server:
    plugins: 
        - plugin:
            name: ait.gui.AITGUIPlugin
            inputs:
                - log_stream
                - telem_stream
            outputs:
                - command_stream

        - plugin:
            name: ait.core.server.plugins.data_archive.DataArchive
            inputs:
                - telem_stream
            datastore: ait.core.db.SQLiteBackend

    inbound-streams:
        - stream:
            name: log_stream
            input: 
                - 2514

        - stream:
            name: telem_stream
            input: 
                - 3076

            handlers:
                - name: ait.core.server.handlers.CCSDSPacketHandler
                  packet_types:   {'01011100111' : 'CCSDS_HEADER'}

Open a terminal, start/create sqlite3 database to the preferred location

$ sqlite3 ait.db

OR if you already have a database, you can reopen the database by running the following commands:

$ sqlite3
SQLite version 3.24.0 2018-06-04 14:10:15
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .open ait.db
sqlite> .databases
main: </diretory/of/ait.db>
sqlite>

On the different terminal, start ait-server

(ait) $ ait-server
2020-02-22T20:08:59.694 | WARNING  | Config parameter command.history.filename specifies nonexistent path </some/path/to>/cmdhist.pcap
2020-02-22T20:08:59.696 | WARNING  | Config parameter dsn.cfdp.mib.path specifies nonexistent path </some/path/to>/AIT-Core/config/mib
2020-02-22T20:08:59.697 | WARNING  | Config parameter dsn.cfdp.datasink.outgoing.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/outgoing
2020-02-22T20:08:59.699 | WARNING  | Config parameter dsn.cfdp.datasink.incoming.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/incoming
2020-02-22T20:08:59.700 | WARNING  | Config parameter dsn.cfdp.datasink.tempfiles.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/tempfiles
2020-02-22T20:08:59.701 | WARNING  | Config parameter dsn.cfdp.datasink.pdusink.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/pdusink
2020-02-22T20:08:59.703 | WARNING  | Config parameter data.1553.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/1553
2020-02-22T20:08:59.704 | WARNING  | Config parameter data.bad.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/bad
2020-02-22T20:08:59.705 | WARNING  | Config parameter data.lehx.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/lehx
2020-02-22T20:08:59.707 | WARNING  | Config parameter data.planning.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/planning
2020-02-22T20:08:59.708 | WARNING  | Config parameter data.sdos.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/sdos
2020-02-22T20:08:59.710 | WARNING  | Config parameter data.uplink.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/uplink
2020-02-22T20:08:59.711 | WARNING  | Config parameter data.ats.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/ats
2020-02-22T20:09:00.554 | WARNING  | No handlers specified for stream log_stream
2020-02-22T20:09:00.563 | INFO     | Added inbound stream <PortInputStream address=:2514>
2020-02-22T20:09:00.828 | INFO     | Created handler CCSDSPacketHandler for stream telem_stream
2020-02-22T20:09:00.830 | INFO     | Added inbound stream <PortInputStream address=:3076>
2020-02-22T20:09:00.832 | WARNING  | No handlers specified for stream command_stream
2020-02-22T20:09:00.834 | INFO     | Added outbound stream <PortOutputStream name=command_stream>
2020-02-22T20:09:01.012 | WARNING  | [GUI Playback Configuration]Unable to locate DataArchive plugin configuration for historical data queries. Historical telemetry playback will be disabled in monitoring UI and server endpoints.
2020-02-22T20:09:01.016 | INFO     | Running AIT GUI v1.4.0
2020-02-22T20:09:01.020 | INFO     | Added plugin <Plugin name=AITGUIPlugin>
2020-02-22T20:09:01.022 | WARNING  | No plugin outputs specified for ait.core.server.plugins.data_archive.DataArchive
2020-02-22T20:09:01.082 | INFO     | Starting telemetry data archiving
2020-02-22T20:09:01.084 | INFO     | Added plugin <Plugin name=DataArchive>
2020-02-22T20:09:01.085 | INFO     | Starting <Broker at 0x10bd04ef0: _run> greenlet...
2020-02-22T20:09:01.087 | INFO     | Starting <Plugin name=AITGUIPlugin> greenlet...
2020-02-22T20:09:01.089 | INFO     | Starting <Plugin name=DataArchive> greenlet...
2020-02-22T20:09:01.092 | INFO     | Starting <PortOutputStream name=command_stream> greenlet...
2020-02-22T20:09:01.094 | INFO     | Starting <PortInputStream address=:2514> greenlet...
2020-02-22T20:09:01.096 | INFO     | Starting <PortInputStream address=:3076> greenlet...
2020-02-22T20:09:01.140 | INFO     | Subscribing <Plugin name=AITGUIPlugin> to topic log_stream
2020-02-22T20:09:01.146 | INFO     | Subscribing <Plugin name=AITGUIPlugin> to topic telem_stream
2020-02-22T20:09:01.148 | INFO     | Subscribing <PortOutputStream name=command_stream> to topic AITGUIPlugin
2020-02-22T20:09:01.149 | INFO     | Subscribing <Plugin name=DataArchive> to topic telem_stream
2020-02-22T20:09:01.151 | INFO     | Starting broker...

On another terminal, invoke ait-tlm-db-insert command

(ait) $ ait-tlm-db-insert --packet CCSDS_HEADER --backend sqlite <your/preferred/directory>/<some_name>.pcap
2020-02-22T20:10:32.685 | WARNING  | Config parameter command.history.filename specifies nonexistent path </some/path/to>/cmdhist.pcap
2020-02-22T20:10:32.688 | WARNING  | Config parameter dsn.cfdp.mib.path specifies nonexistent path </some/path/to>/AIT-Core/config/mib
2020-02-22T20:10:32.689 | WARNING  | Config parameter dsn.cfdp.datasink.outgoing.path specifies nonexistent path </some/path/to>AIT-Core/ait/dsn/cfdp/datasink/outgoing
2020-02-22T20:10:32.691 | WARNING  | Config parameter dsn.cfdp.datasink.incoming.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/incoming
2020-02-22T20:10:32.693 | WARNING  | Config parameter dsn.cfdp.datasink.tempfiles.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/tempfiles
2020-02-22T20:10:32.695 | WARNING  | Config parameter dsn.cfdp.datasink.pdusink.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/pdusink
2020-02-22T20:10:32.697 | WARNING  | Config parameter data.1553.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/1553
2020-02-22T20:10:32.699 | WARNING  | Config parameter data.bad.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/bad
2020-02-22T20:10:32.701 | WARNING  | Config parameter data.lehx.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/downlink/lehx
2020-02-22T20:10:32.703 | WARNING  | Config parameter data.planning.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/planning
2020-02-22T20:10:32.705 | WARNING  | Config parameter data.sdos.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/sdos
2020-02-22T20:10:32.707 | WARNING  | Config parameter data.uplink.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/uplink
2020-02-22T20:10:32.709 | WARNING  | Config parameter data.ats.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-054/ats
2020-02-22T20:10:32.919 | PROGRAM  | <your/directory/to>/.virtualenvs/ait/bin/ait-tlm-db-insert --packet CCSDS_HEADER --backend sqlite <your/preferred/directory>/<some_name>.pcap
2020-02-22T20:10:32.926 | INFO     | Processing <your/preferred/directory>/<some_name>.pcap
2020-02-22T20:10:32.938 | INFO     | Inserted 1 CCSDS_HEADER packets into database  </diretory/of/ait.db>.
2020-02-22T20:10:32.940 | PROGRAM  | done.

Go back to SQLite3 terminal , verify that the data was inserted in the table

sqlite> .header on
sqlite> .mode column
sqlite> select * from CCSDS_HEADER;
time                      version     type        secondary_header_flag  apid        sequence_flags  sequence_count  packet_length  time_coarse  time_fine   time_id     checkword_indicator  zoe         packet_type  element_id  data_packet  version_id  format_id   frame_id  
------------------------  ----------  ----------  ---------------------  ----------  --------------  --------------  -------------  -----------  ----------  ----------  -------------------  ----------  -----------  ----------  -----------  ----------  ----------  ----------
2020-02-23T04:10:32.934Z  0           0           0                      743         1               0               11             1.0          1.0         1           1                    1           1            1           1            1           1           1         
sqlite>

Stop SQLite3
```
sqlite> .quit
```
Type Ctrl+C to stop ait-server

Futabay commented 4 years ago

Found bugs:

Python3 related tlmdict.keys() needs to be converted to a list. (ait_tlm_db_insert.py#35)
Other database.name needs to be updated to database.dbname. (ait_tlm_db_insert.py#L51, ait_tlm_db_insert.py#L54)

MJJoyce commented 4 years ago

Looks like the issues found were fixed in #254

NASA-AMMOS / AIT-Core

'ait-tlm-db-insert' SQLite3 Test #244