search

NASA-AMMOS / AIT-Core

MIT License
45 stars 27 forks source link

`ait.core.db` and `data_archive` plugin SQLite3 Test #240

Closed Futabay closed 4 years ago

Futabay commented 4 years ago

  1. Precondition:

    1. AIT is installed with virtual environment
    2. Virtual environment is started by typing workon ait on your working terminals
    3. SQLite3 in installed on a testing machine.

  2. Set config.yaml as follows:

    database:
    dbname: </Pass/to/ait.db>

server:
    plugins: 
        - plugin:
            name: ait.gui.AITGUIPlugin
            inputs:
                - log_stream
                - telem_stream
            outputs:
                - command_stream

        - plugin:
            name: ait.core.server.plugins.data_archive.DataArchive
            inputs:
                - telem_stream
            datastore: ait.core.db.SQLiteBackend

    inbound-streams:
        - stream:
            name: log_stream
            input: 
                - 2514

        - stream:
            name: telem_stream
            input: 
                - 3076
            handlers:
                - name: ait.core.server.handlers.CCSDSPacketHandler
                  packet_types:   {'01011100111' : 'CCSDS_HEADER'}

  3. Open a terminal, start/create sqlite3 database to the preferred location 

    $ sqlite3 ait.db

  4. On the different terminal, start ait-server

    (ait) $ ait-server
2020-02-21T08:38:10.663 | WARNING  | Config parameter command.history.filename specifies nonexistent path </some/path/to>/cmdhist.pcap
2020-02-21T08:38:10.666 | WARNING  | Config parameter dsn.cfdp.mib.path specifies nonexistent path </some/path/to>/AIT-Core/config/mib
2020-02-21T08:38:10.667 | WARNING  | Config parameter dsn.cfdp.datasink.outgoing.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/outgoing
2020-02-21T08:38:10.669 | WARNING  | Config parameter dsn.cfdp.datasink.incoming.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/incoming
2020-02-21T08:38:10.671 | WARNING  | Config parameter dsn.cfdp.datasink.tempfiles.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/tempfiles
2020-02-21T08:38:10.672 | WARNING  | Config parameter dsn.cfdp.datasink.pdusink.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/pdusink
2020-02-21T08:38:10.674 | WARNING  | Config parameter data.1553.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/downlink/1553
2020-02-21T08:38:10.676 | WARNING  | Config parameter data.bad.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/downlink/bad
2020-02-21T08:38:10.677 | WARNING  | Config parameter data.lehx.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/downlink/lehx
2020-02-21T08:38:10.679 | WARNING  | Config parameter data.planning.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/planning
2020-02-21T08:38:10.680 | WARNING  | Config parameter data.sdos.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/sdos
2020-02-21T08:38:10.682 | WARNING  | Config parameter data.uplink.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/uplink
2020-02-21T08:38:10.683 | WARNING  | Config parameter data.ats.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/ats
2020-02-21T08:38:10.958 | WARNING  | No handlers specified for stream log_stream
2020-02-21T08:38:10.966 | INFO     | Added inbound stream <PortInputStream address=:2514>
2020-02-21T08:38:11.162 | INFO     | Created handler CCSDSPacketHandler for stream telem_stream
2020-02-21T08:38:11.165 | INFO     | Added inbound stream <PortInputStream address=:3076>
2020-02-21T08:38:11.167 | WARNING  | No handlers specified for stream command_stream
2020-02-21T08:38:11.170 | INFO     | Added outbound stream <PortOutputStream name=command_stream>
2020-02-21T08:38:11.305 | WARNING  | [GUI Playback Configuration]Unable to locate DataArchive plugin configuration for historical data queries. Historical telemetry playback will be disabled in monitoring UI and server endpoints.
2020-02-21T08:38:11.309 | INFO     | Running AIT GUI v1.4.0
2020-02-21T08:38:11.314 | INFO     | Added plugin <Plugin name=AITGUIPlugin>
2020-02-21T08:38:11.316 | WARNING  | No plugin outputs specified for ait.core.server.plugins.data_archive.DataArchive
2020-02-21T08:38:11.346 | INFO     | Starting telemetry data archiving
2020-02-21T08:38:11.348 | INFO     | Added plugin <Plugin name=DataArchive>
2020-02-21T08:38:11.349 | INFO     | Starting <Broker at 0x108ea1ef0: _run> greenlet...
2020-02-21T08:38:11.351 | INFO     | Starting <Plugin name=AITGUIPlugin> greenlet...
2020-02-21T08:38:11.352 | INFO     | Starting <Plugin name=DataArchive> greenlet...
2020-02-21T08:38:11.354 | INFO     | Starting <PortOutputStream name=command_stream> greenlet...
2020-02-21T08:38:11.356 | INFO     | Starting <PortInputStream address=:2514> greenlet...
2020-02-21T08:38:11.357 | INFO     | Starting <PortInputStream address=:3076> greenlet...
2020-02-21T08:38:11.389 | INFO     | Subscribing <Plugin name=AITGUIPlugin> to topic log_stream
2020-02-21T08:38:11.393 | INFO     | Subscribing <Plugin name=AITGUIPlugin> to topic telem_stream
2020-02-21T08:38:11.395 | INFO     | Subscribing <PortOutputStream name=command_stream> to topic AITGUIPlugin
2020-02-21T08:38:11.397 | INFO     | Subscribing <Plugin name=DataArchive> to topic telem_stream
2020-02-21T08:38:11.399 | INFO     | Starting broker...

  5. On another terminal, start python and invoke db.py.connect() and .create() functions

    (ait) $ python
Python 3.7.6 (default, Dec 30 2019, 19:38:28) 
[Clang 11.0.0 (clang-1100.0.33.16)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ait.core.db as db
2020-02-21T08:40:15.519 | WARNING  | Config parameter command.history.filename specifies nonexistent path </some/path/to>/cmdhist.pcap
2020-02-21T08:40:15.522 | WARNING  | Config parameter dsn.cfdp.mib.path specifies nonexistent path </some/path/to>/AIT-Core/config/mib
2020-02-21T08:40:15.524 | WARNING  | Config parameter dsn.cfdp.datasink.outgoing.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/outgoing
2020-02-21T08:40:15.526 | WARNING  | Config parameter dsn.cfdp.datasink.incoming.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/incoming
2020-02-21T08:40:15.528 | WARNING  | Config parameter dsn.cfdp.datasink.tempfiles.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/tempfiles
2020-02-21T08:40:15.530 | WARNING  | Config parameter dsn.cfdp.datasink.pdusink.path specifies nonexistent path </some/path/to>/AIT-Core/ait/dsn/cfdp/datasink/pdusink
2020-02-21T08:40:15.533 | WARNING  | Config parameter data.1553.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/downlink/1553
2020-02-21T08:40:15.534 | WARNING  | Config parameter data.bad.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/downlink/bad
2020-02-21T08:40:15.536 | WARNING  | Config parameter data.lehx.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/downlink/lehx
2020-02-21T08:40:15.538 | WARNING  | Config parameter data.planning.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/planning
2020-02-21T08:40:15.540 | WARNING  | Config parameter data.sdos.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/sdos
2020-02-21T08:40:15.542 | WARNING  | Config parameter data.uplink.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/uplink
2020-02-21T08:40:15.544 | WARNING  | Config parameter data.ats.path specifies nonexistent path /gds/dev/data/LMC-062858/2020/2020-052/ats
>>> sqlbackend = db.SQLiteBackend()
>>> sqlbackend.connect()
>>> sqlbackend.create()
>>>

  6. On SQLite3 terminal , verify that the tables were created.

    
sqlite> .databases
main: </directory/of/ait.db>
sqlite> .tables
1553_HS_Packet      CCSDS_HEADER        Ethernet_HS_Packet
sqlite> .header on
sqlite> .mode column
sqlite>  pragma table_info('1553_HS_Packet');
cid         name        type        notnull     dflt_value                             pk        
----------  ----------  ----------  ----------  -------------------------------------  ----------
0           time        DATETIME    0           STRFTIME('%Y-%m-%dT%H:%M:%fZ', 'NOW')  0         
1           Voltage_A   INTEGER     0                                                  0         
2           Voltage_B   INTEGER     0                                                  0         
3           Voltage_C   INTEGER     0                                                  0         
4           Voltage_D   INTEGER     0                                                  0         
5           Current_A   REAL        0                                                  0         
sqlite> 
sqlite> pragma table_info('CCSDS_HEADER');
cid         name        type        notnull     dflt_value                             pk        
----------  ----------  ----------  ----------  -------------------------------------  ----------
0           time        DATETIME    0           STRFTIME('%Y-%m-%dT%H:%M:%fZ', 'NOW')  0         
1           version     INTEGER     0                                                  0         
2           type        INTEGER     0                                                  0         
3           secondary_  INTEGER     0                                                  0         
4           apid        INTEGER     0                                                  0         
5           sequence_f  INTEGER     0                                                  0         
6           sequence_c  INTEGER     0                                                  0         
7           packet_len  INTEGER     0                                                  0         
8           time_coars  REAL        0                                                  0         
9           time_fine   REAL        0                                                  0         
10          time_id     INTEGER     0                                                  0         
11          checkword_  INTEGER     0                                                  0         
12          zoe         INTEGER     0                                                  0         
13          packet_typ  INTEGER     0                                                  0         
14          element_id  INTEGER     0                                                  0         
15          data_packe  INTEGER     0                                                  0         
16          version_id  INTEGER     0                                                  0         
17          format_id   INTEGER     0                                                  0         
18          frame_id    INTEGER     0                                                  0         
sqlite> 
sqlite> pragma table_info('Ethernet_HS_Packet');
cid         name        type        notnull     dflt_value                             pk        
----------  ----------  ----------  ----------  -------------------------------------  ----------
0           time        DATETIME    0           STRFTIME('%Y-%m-%dT%H:%M:%fZ', 'NOW')  0         
1           sync_word   INTEGER     0                                                  0         
2           time_creat  REAL        0                                                  0         
3           product_ty  INTEGER     0                                                  0         
4           product_le  INTEGER     0                                                  0         
5           VoltageSam  REAL        0                                                  0         
6           Voltage_A   INTEGER     0                                                  0         
7           Voltage_B   INTEGER     0                                                  0         
8           Voltage_C   INTEGER     0                                                  0         
9           Voltage_D   INTEGER     0                                                  0         
10          footer      INTEGER     0                                                  0         
sqlite> 
6. On the python terminal, invoke `db.py` `.query()` function

sqlbackend.query('SELECT * FROM Ethernet_HS_Packet') <sqlite3.Cursor object at 0x106595730>

7. On the python terminal, invoke`db.py` `.insert()` function

import ait.core.tlm tlmdict = ait.core.tlm.getDefaultDict() ccsds_pkt = tlmdict['CCSDS_HEADER'] data = bytearray(b'\x02\xE7\x40\x00\x00\x0B\x00\x00\x00\x01\x01\x71\x0C\x41\x00\x01’) pkt = ait.core.tlm.Packet(ccsds_pkt, data=data) sqlbackend.insert(pkt)

8. On SQLite3 terminal , verify that the data was inserted.

sqlite> .header off sqlite> .mode list sqlite> select * from CCSDS_HEADER; 2020-02-21T16:48:21.172Z|0|0|0|743|1|0|11|1.0|1.0|1|1|1|1|1|1|1|1|1

7. Stop SQLite3

sqlite> .quit


8. On the python terminal, type `Ctrl`+D to exit
  1. Type Ctrl+C to stop ait-server
Futabay commented 4 years ago

Tested SQLite3 version

$ sqlite3 --version
3.24.0 2018-06-04 14:10:15 95fbac39baaab1c3a84fdfc82ccb7f42398b2e92f18a2a57bce1d4a713cbaapl
Futabay commented 4 years ago

Content of packet data created based on ccsds_header.yaml and used for this test is as follows:

data = bytearray(b'\x02\xE7\x40\x00\x00\x0B\x00\x00\x00\x01\x01\x71\x0C\x41\x00\x01’)
## CCSDS Packet ##
version:                        000
type:                           0
secondary header flag:          0
apid:                           01011100111 #743#
sequence flag:                  01
sequence count:                 00000000 000000
packet length:                  00000000 00001011
time_coars                      00000000 00000000 00000000 00000001
time_fine                       0000 0001
time_id                         01
checkword_indicator             1
zoe                             1
packet_type                     0001
<spare>                         0
element_id                      0001
data_packet                     1               
version_id                      0001 
format_id                       000001
<unknown>                       00000000
frame_id                        00000001