Help with hardcore output

[evilmonkey19]

Hello! I'm trying to parse some output data from a certain device and I'm trying to do a nested output. I have the following output from a network device:

  -----------------------------------------------------------------------------
  F/S/P                   : 0/1/1
  ONT-ID                  : 131
  Control flag            : active
  Run state               : online
  Config state            : normal
  Match state             : match
  DBA type                : SR
  ONT distance(m)         : 100
  ONT last distance(m)    : 100
  ONT battery state       : not support
  ONT power type          : -
  Memory occupation       : 23%
  CPU occupation          : 30%
  Temperature             : 70(C)
  Authentic type          : SN-auth
  SN                      : 123456789123AB12AB (HWTC-12AB12AB)
  Management mode         : OMCI
  Software work mode      : normal
  Isolation state         : normal
  ONT IP 0 address/mask   : 192.168.0.1/24
  ONT IP 1 address/mask   : 192.168.0.1/24
  Description             : ABCD
  Last down cause         : dying-gasp
  Last up time            : 09/02/2019 01:58:52+00:00
  Last down time          : 22/01/2019 01:57:55+00:00
  Last dying gasp time    : 22/01/2019 01:57:55+00:00
  Last restart reason     : -
  ONT online duration     : 22 day(s), 1 hour(s), 23 minute(s), 23 second(s) 
  ONT system up duration  : 22 day(s), 1 hour(s), 23 minute(s), 23 second(s) 
  Type C support          : Not support
  Interoperability-mode   : ITU-T
  Power reduction status  : -
  FEC upstream state      : use-profile-config
  VS-ID                   : 0
  VS name                 : admin-vs
  Global ONT-ID           : 131
  -----------------------------------------------------------------------------
  VoIP configure method   : Default
  -----------------------------------------------------------------------------
  Line profile ID      : 100
  Line profile name    : ftth
  -----------------------------------------------------------------------------
  FEC upstream switch :Disable 
  OMCC encrypt switch :On
  Qos mode            :PQ
  Mapping mode        :VLAN
  TR069 management    :Enable        
  TR069 IP index      :0
  ------------------------------------------------------------------------------
  Notes: * indicates Discrete TCONT(TCONT Unbound)
  ------------------------------------------------------------------------------
  <T-CONT   0>          DBA Profile-ID:1
  <T-CONT   1>          DBA Profile-ID:100
   <Gem Index 1>
   --------------------------------------------------------------------
   |Serv-Type:ETH |Encrypt:on  |Cascade:off |GEM-CAR:-            |
   |Upstream-priority-queue:0  |Downstream-priority-queue:-       |
   --------------------------------------------------------------------
    Mapping VLAN  Priority Port    Port  Bundle  Flow  Transparent
    index                  type    ID    ID      CAR   
   --------------------------------------------------------------------
    0       40    -        -       -     -       -     -          
    1       41    -        -       -     -       -     -          
   --------------------------------------------------------------------
  <T-CONT   2>          DBA Profile-ID:101
   <Gem Index 2>
   --------------------------------------------------------------------
   |Serv-Type:ETH |Encrypt:on  |Cascade:off |GEM-CAR:-            |
   |Upstream-priority-queue:0  |Downstream-priority-queue:-       |
   --------------------------------------------------------------------
    Mapping VLAN  Priority Port    Port  Bundle  Flow  Transparent
    index                  type    ID    ID      CAR   
   --------------------------------------------------------------------
    0       42    -        -       -     -       -     -          
   --------------------------------------------------------------------
  <T-CONT   3>          DBA Profile-ID:102
   <Gem Index 3>
   --------------------------------------------------------------------
   |Serv-Type:ETH |Encrypt:on  |Cascade:off |GEM-CAR:-            |
   |Upstream-priority-queue:0  |Downstream-priority-queue:-       |
   --------------------------------------------------------------------
    Mapping VLAN  Priority Port    Port  Bundle  Flow  Transparent
    index                  type    ID    ID      CAR   
   --------------------------------------------------------------------
    0       1     -        -       -     -       -     -          
    1       2     -        -       -     -       -     -          
    2       3     -        -       -     -       -     -          
    3       4     -        -       -     -       -     -          
   --------------------------------------------------------------------
  <T-CONT   4>          DBA Profile-ID:103
   <Gem Index 4>
   --------------------------------------------------------------------
   |Serv-Type:ETH |Encrypt:on  |Cascade:off |GEM-CAR:-            |
   |Upstream-priority-queue:0  |Downstream-priority-queue:-       |
   --------------------------------------------------------------------
    Mapping VLAN  Priority Port    Port  Bundle  Flow  Transparent
    index                  type    ID    ID      CAR   
   --------------------------------------------------------------------
    0       8     -        -       -     -       -     -          
   --------------------------------------------------------------------
  <T-CONT   5>          DBA Profile-ID:104
   <Gem Index 5>
   --------------------------------------------------------------------
   |Serv-Type:ETH |Encrypt:on  |Cascade:off |GEM-CAR:-            |
   |Upstream-priority-queue:0  |Downstream-priority-queue:-       |
   --------------------------------------------------------------------
  ------------------------------------------------------------------------------
  Notes: Run the display traffic table ip command to query 
         traffic table configuration
  -----------------------------------------------------------------------------
  Service profile ID   : 100
  Service profile name : ftth
[...]

I would like to have a json format output like so:

{
    "F/S/P": "0/1/1",
    "ONT-ID": "131",
    "T-CONTs": [
      {
        "T-CONT": "0",
        "DBA Profile-ID": "1",
      },
      {
        "T-CONT": "1",
        "DBA Profile-ID": "100",
        "Gem": {
          "Index": "1",
          "Serv-Type": "ETH"
        }
      },
      {
        "T-CONT": "2",
        "DBA Profile-ID": "101",
        "Gem": {
          "Index": "2",
          "Serv-Type": "ETH"
        }
      },
    ]
}

Do you any suggestions how can i achieve this in textfsm? My current template in textfsm is the following:

Value FSP (\d+\/\d+\/\d+)
Value ONT_ID (\d+)
Value CONTROL_FLAG (\w+)
Value CONFIG_STATE (\w+)
Value MATCH_STATE (\w+)
Value DBA_TYPE (\w+)
Value DISTANCE_M (\d+)
Value LAST_DISTANCE_M (\d+)
Value BATTERY_STATE (\w+\s*\w*)
Value POWER_TYPE (\w+)
Value MEMORY_OCCUPATION (\d+)
Value CPU_OCCUPATION (\d+)
Value TEMPERATURE (\d+\(\w\))
Value AUTHENTIC_TYPE (\S+)
Value SN (\w+\s*\(*\S*\))
Value MANAGEMENT_MODE (\w+)
Value SOFTWARE_WORK_MODE (\w+)
Value ISOLATION_STATE (\w+)
Value IP_0_ADDRESS_MASK (\S+)
Value IP_1_ADDRESS_MASK (\S+)
Value DESCRIPTION (\S+)
Value LAST_DOWN_CAUSE (\S+)
Value LAST_UP_TIME (\w+\/\w+\/\w+\s\w+:\w+:\w+\+\w+:\w+)
Value LAST_DOWN_TIME (\w+\/\w+\/\w+\s\w+:\w+:\w+\+\w+:\w+)
Value LAST_DYING_GASP_TIME (\w+\/\w+\/\w+\s\w+:\w+:\w+\+\w+:\w+)
Value LAST_RESTART_REASON (\w+)
Value TYPE_C_SUPPORT (\w+\s*\w*)
Value INTEROPERABILITY_MODE (\S+)
Value POWER_REDUCTION_STATUS (\w+)
Value FEC_UPSTREAM_STATE (\S+)
Value VS_ID (\d+)
Value VS_NAME (\S+)
Value GLOBAL_ONT_ID (\d+)
Value VOIP_CONFIGURE_METHOD (\w+)
Value LINE_PROFILE_ID (\d+)
Value LINE_PROFILE_NAME (\w+)
Value FEC_UPSTREAM_SWITCH (\w+)
Value OMCC_ENCRYPT_SWITCH (\w+)
Value QOS_MODE (\w+)
Value MAPPING_MODE (\w+)
Value TR069_MANAGEMENT (\w+)
Value TR069_IP_INDEX (\d+)
Value SERVICE_PROFILE_ID (\d+)
Value SERVICE_PROFILE_NAME (\w+)
Value TDM_PORT_TYPE (\w+)
Value TDM_SERVICE_TYPE (\w+)
Value MAC_LEARNING_FUNCTION_SWITCH (\w+)
Value ONT_TRANSPARENT_FUNCTION_SWITCH (\w+)
Value RING_CHECK_SWITCH (\w+)
Value RING_PORT_AUTO_SHUTDOWN (\w+)
Value RING_DETECT_FREQUENCY (\w+\s\(\w*\))
Value RING_RESUME_INTERVAL (\w+\s\(\w*\))
Value RING_DETECT_PERIOD (\w+\s\(\w*\))
Value MULTICAST_FORWARD_MODE (\w+)
Value MULTICAST_FORWARD_VLAN (\w+)
Value MULTICAST_MODE (\w+)
Value UPSTREAM_IGMP_PACKET_FORWARD_MODE (\w+)
Value UPSTREAM_IGMP_PACKET_FORWARD_VLAN (\w+)
Value UPSTREAM_IGMP_PACKET_PRIORITY (\w+)
Value NATIVE_VLAN_OPTION (\w+)
Value UPSTREAM_PQ_COLOR_POLICY (\w+)
Value DOWNSTREAM_PQ_COLOR_POLICY (\w+)
Value MONITOR_LINK (\w+)
Value MTU_BYTE (\w+)
Value ALARM_POLICY_PROFILE_ID (\d+)
Value ALARM_POLICY_PROFILE_NAME (\S+)
Value TR069_SERVER_PROFILE_ID (\d+)
Value TR069_SERVER_PROFILE_NAME (\S+)

Start
 ^\s+F/S/P\s+:\s+${FSP}
 ^\s+ONT-ID\s+:\s+${ONT_ID}
 ^\s+Control\s+flag\s+:\s+${CONTROL_FLAG}
 ^\s+Config\s+state\s+:\s+${CONFIG_STATE}
 ^\s+Match\s+state\s+:\s+${MATCH_STATE}
 ^\s+DBA\s+type\s+:\s+${DBA_TYPE}
 ^\s+ONT\s+distance\(m\)\s+:\s+${DISTANCE_M}
 ^\s+ONT\s+last\s+distance\(m\)\s+:\s+${LAST_DISTANCE_M}
 ^\s+ONT\s+battery\s+state\s+:\s+${BATTERY_STATE}
 ^\s+ONT\s+power\s+type\s+:\s+${POWER_TYPE}
 ^\s+Memory\s+occupation\s+:\s+${MEMORY_OCCUPATION}%
 ^\s+CPU\s+occupation\s+:\s+${CPU_OCCUPATION}%
 ^\s+Temperature\s+:\s+${TEMPERATURE}
 ^\s+Authentic\s+type\s+:\s+${AUTHENTIC_TYPE}
 ^\s+SN\s+:\s+${SN}
 ^\s+Management\s+mode\s+:\s+${MANAGEMENT_MODE}
 ^\s+Software\s+work\s+mode\s+:\s+${SOFTWARE_WORK_MODE}
 ^\s+Isolation\s+state\s+:\s+${ISOLATION_STATE}
 ^\s+ONT\s+IP\s+0\s+address\/mask\s+:\s+${IP_0_ADDRESS_MASK}
 ^\s+ONT\s+IP\s+1\s+address\/mask\s+:\s+${IP_1_ADDRESS_MASK}
 ^\s+Description\s+:\s+${DESCRIPTION}
 ^\s+Last\s+down\s+cause\s+:\s+${LAST_DOWN_CAUSE}
 ^\s+Last\s+up\s+time\s+:\s+${LAST_UP_TIME}
 ^\s+Last\s+down\s+time\s+:\s+${LAST_DOWN_TIME}
 ^\s+Last\s+dying\s+gasp\s+time\s+:\s+${LAST_DYING_GASP_TIME}
 ^\s+Last\s+restart\s+reason\s+:\s+${LAST_RESTART_REASON}
 ^\s+Type\s+C\s+support\s+:\s+${TYPE_C_SUPPORT}
 ^\s+Interoperability-mode\s+:\s+${INTEROPERABILITY_MODE}
 ^\s+Power\s+reduction\s+status\s+:\s+${POWER_REDUCTION_STATUS}
 ^\s+FEC\s+upstream\s+state\s+:\s+${FEC_UPSTREAM_STATE}
 ^\s+VS-ID\s+:\s+${VS_ID}
 ^\s+VS\s+name\s+:\s+${VS_NAME}
 ^\s+Global\s+ONT-ID\s+:\s+${GLOBAL_ONT_ID}
 ^\s+- -> Next
 ^\s+VoIP\s+configure\s+method\s+:\s+${VOIP_CONFIGURE_METHOD}
 ^\s+- -> Next
 ^\s+Line\s+profile\s+ID\s+:\s+${LINE_PROFILE_ID}
 ^\s+Line\s+profile\s+name\s+:\s+${LINE_PROFILE_NAME}
 ^\s+- -> Next
 ^\s+FEC\s+upstream\s+switch\s+:\s*${FEC_UPSTREAM_SWITCH}
 ^\s+OMCC\s+encrypt\s+switch\s+:\s*${OMCC_ENCRYPT_SWITCH}
 ^\s+Qos\s+mode\s+:\s*${QOS_MODE}
 ^\s+Mapping\s+mode\s+:\s*${MAPPING_MODE}
 ^\s+TR069\s+management\s+:\s*${TR069_MANAGEMENT}
 ^\s+TR069\s+IP\s+index\s+:\s*${TR069_IP_INDEX}
 ^\s+Service\s+profile\s+ID\s+:\s+${SERVICE_PROFILE_ID}
 ^\s+Service\s+profile\s+name\s+:\s+${SERVICE_PROFILE_NAME}
 ^\s+TDM\s+port\s+type\s+:\s+${TDM_PORT_TYPE}
 ^\s+TDM\s+service\s+type\s+:\s+${TDM_SERVICE_TYPE}
 ^\s+MAC\s+learning\s+function\s+switch\s+:\s+${MAC_LEARNING_FUNCTION_SWITCH}
 ^\s+ONT\s+transparent\s+function\s+switch\s+:\s+${ONT_TRANSPARENT_FUNCTION_SWITCH}
 ^\s+Ring\s+check\s+switch\s+:\s+${RING_CHECK_SWITCH}
 ^\s+Ring\s+port\s+auto-shutdown\s+:\s+${RING_PORT_AUTO_SHUTDOWN}
 ^\s+Ring\s+detect\s+frequency\s+:\s+${RING_RESUME_INTERVAL}
 ^\s+Ring\s+resume\s+interval\s+:\s+${RING_DETECT_PERIOD}
 ^\s+Multicast\s+forward\s+mode\s+:\s+${MULTICAST_FORWARD_MODE}
 ^\s+Multicast\s+forward\s+VLAN\s+:\s+${MULTICAST_FORWARD_VLAN}
 ^\s+Multicast\s+mode\s+:\s+${MULTICAST_MODE}
 ^\s+Upstream\s+IGMP\s+packet\s+forward\s+mode\s+:\s+${UPSTREAM_IGMP_PACKET_FORWARD_MODE}
 ^\s+Upstream\s+IGMP\s+packet\s+forward\s+VLAN\s+:\s+${UPSTREAM_IGMP_PACKET_FORWARD_VLAN}
 ^\s+Upstream\s+IGMP\s+packet\s+priority\s+:\s+${UPSTREAM_IGMP_PACKET_PRIORITY}
 ^\s+Native\s+VLAN\s+option\s+:\s+${NATIVE_VLAN_OPTION}
 ^\s+Upstream\s+PQ\s+color\s+policy\s+:\s+${UPSTREAM_PQ_COLOR_POLICY}
 ^\s+Downstream\s+PQ\s+color\s+policy\s+:\s+${DOWNSTREAM_PQ_COLOR_POLICY}
 ^\s+Monitor\s+link\s+:\s+${MONITOR_LINK}
 ^\s+MTU\(byte\)\s+:\s+${MTU_BYTE}
 ^\s+Alarm\s+policy\s+profile\s+ID\s+:\s+${ALARM_POLICY_PROFILE_ID}
 ^\s+Alarm\s+policy\s+profile\s+name\s+:\s+${ALARM_POLICY_PROFILE_NAME}
 ^\s+TR069\s+server\s+profile\s+ID\s+:\s+${TR069_SERVER_PROFILE_ID}
 ^\s+TR069\s+server\s+profile\s+name\s+:\s+${TR069_SERVER_PROFILE_NAME}

[harro]

Take a look at FILLDOWN

Since TextFSM produces a table as output (not JSON) then the first thing you need to decide - is what the output table needs to look like for you to be able to post-process it into JSON. Once you have your table format sorted then it should be clear which columns will need FILLDOWN to insure the innermost, child elements are complete when they are recorded to the table.

You probably want to make use of the state based parsing to make parsing the different sections of your config easier/clearer.

Lastly consider parsing this data in multiple passes. Using a different TextFSM template each time. The decision on if this is desirable or not, will be driven by whether you can readily represent the results in a single table, or several tables.

[evilmonkey19]

Thanks for the answer! I have already used passing multiple times over the output, which to me it produces the most maintainable template. Thanks for the suggestion :D