search

zowe / zebra

ZEBRA is an open-source incubator project for Zowe. It is a data parsing framework that allows quick and easy access to z/OS performance metrics.
https://zebra.talktothemainframe.com
Eclipse Public License 2.0
22 stars 13 forks source link

[BUG] Cannot read property 'mvsResource' of undefined at RMFIIIJSON #107

Closed liam-hogan closed 2 years ago

liam-hogan commented 2 years ago

Describe the bug Hey guys, I have been testing a pipeline deployment and I am seeing some recent issues with pulling RMF III reports. I am not sure what is causing this, and was hoping you guys had some info?

To Reproduce Steps to reproduce the behavior:

  1. Spin up Zebra stack using Docker
  2. Go to the Zebra web page
  3. Choose any report
  4. The report times out
  5. Zebra logs will report with an error

Expected behavior I should be able to view the JSON data for the supplied report, instead the webpage times out with no data. I was able to have this working about two weeks ago, but now this has stopped.

Error I get the following error when viewing the Docker log for zebra:

GET /v1/S04/rmf3/CPC - - ms - -
(node:24) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'mvsResource' of undefined
    at RMFIIIJSON (/zebra/app_server/v1_Controllers/RMF3Controller.js:218:29)
    at module.exports.RMFIII (/zebra/app_server/v1_Controllers/RMF3Controller.js:318:11)
    at Layer.handle [as handle_request] (/zebra/node_modules/express/lib/router/layer.js:95:5)
    at next (/zebra/node_modules/express/lib/router/route.js:137:13)
    at Route.dispatch (/zebra/node_modules/express/lib/router/route.js:112:3)
    at Layer.handle [as handle_request] (/zebra/node_modules/express/lib/router/layer.js:95:5)
    at /zebra/node_modules/express/lib/router/index.js:281:22
    at param (/zebra/node_modules/express/lib/router/index.js:354:14)
    at param (/zebra/node_modules/express/lib/router/index.js:365:14)
    at param (/zebra/node_modules/express/lib/router/index.js:365:14)
    at Function.process_params (/zebra/node_modules/express/lib/router/index.js:410:3)
    at next (/zebra/node_modules/express/lib/router/index.js:275:10)
    at Function.handle (/zebra/node_modules/express/lib/router/index.js:174:3)
    at router (/zebra/node_modules/express/lib/router/index.js:47:12)
    at Layer.handle [as handle_request] (/zebra/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/zebra/node_modules/express/lib/router/index.js:317:13)
(node:24) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 9)
(node:24) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'timestart' of undefined
    at /zebra/mongoV1.js:67:50
    at /zebra/mongoV1.js:87:21
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:189:7)

Configuration:

Server:

docker-compose version 1.25.0, build unknown
docker-py version: 4.1.0
CPython version: 3.8.10
OpenSSL version: OpenSSL 1.1.1f  31 Mar 2020

Zebra Config [Login info redacted]:

{
  "mongourl": "zebra-mongo",
  "dbinterval": "100",
  "dbname": "zebraDB",
  "appurl": "localhost",
  "appport": "3090",
  "mongoport": "27017",
  "ppminutesInterval": "30",
  "rmf3interval": "100",
  "zebra_httptype": "https",
  "useDbAuth": "true",
  "dbUser": "user",
  "dbPassword": "pass",
  "authSource": "admin",
  "use_cert": "false",
  "grafanaurl": "localhost",
  "grafanaport": "9000",
  "grafanahttptype": "http",
  "dds": {
    "S01": {
      "ddshhttptype": "http",
      "ddsbaseurl": "******",
      "ddsbaseport": "8803",
      "ddsauth": "true",
      "ddsuser": "******",
      "ddspwd": "******",
      "rmf3filename": "rmfm3.xml",
      "rmfppfilename": "rmfpp.xml",
      "mvsResource": "******",
      "PCI": 3340,
      "usePrometheus": "true",
      "useMongo": "true"
    }
  }
}

docker-compose.yaml

version: '3'
services:
  zebra:
    container_name: zebra
    build: ~/zebra/src
    restart: always
    ports:
      - '3090:3090'
    depends_on:
      - mongo
      - prometheus
      - grafana
  mongo:
    container_name: zebra-mongo
    image: mongo:5.0.3
    ports:
      - '27017:27017'
    volumes:
      - mongo-data:/data/db
  prometheus:
    container_name: zebra-prometheus
    image: prom/prometheus:v2.30.3
    ports:
      - '9090:9090'
    volumes:
      - prometheus-data:/prometheus/data
      - ~/zebra/src/config/prometheus:/etc/prometheus
  grafana:
    container_name: zebra-grafana
    image: grafana/grafana:8.2.2
    ports:
      - '3000:3000'
    depends_on:
      - prometheus
    volumes:
      - grafana-data:/var/lib/grafana
      - ~/zebra/src/config/grafana:/etc/grafana/provisioning/datasources
  json_exporter:
    image: prometheuscommunity/json-exporter
    container_name: json_exporter
    volumes:
      - ./json-exporter/config.yml:/config.yml:ro
    ports:
      - '7979:7979'
    depends_on:
      - zebra
volumes:
  mongo-data:
  prometheus-data:
  grafana-data:

Additional context I first thought it was the mongoURL, so I changed the URL to the context of the docker network. All of the hosts should be using the same underlying network. I have confirmed that the DDS URL, username and password work.

fernandozangari commented 2 years ago

From the browser if you type http or https://ip_of_mainframe:8803 do you see the RMF explorer app?

liam-hogan commented 2 years ago

From the browser if you type http or https://ip_of_mainframe:8803 do you see the RMF explorer app?

Hi, yes. I am assuming you are talking about the webpage that says "RMF Data Portal for Z/OS"? If so, then yes.

I am also able to confirm that the login I am using works and that I can see metrics.

behives commented 2 years ago

Hi @liam-hogan

a few things to verify first...

1) was previous session working also a docker environment? 2) can you make sure there is no other container(s) running/using same port? 3) is the connection to outside working well from container?

(you can use curl command too - and use proper options for your settings)

this will make sure there is valid connection from docker to DDS service.

liam-hogan commented 2 years ago

Hey @behives !

Hi @liam-hogan

a few things to verify first...

  1. was previous session working also a docker environment?

Yes

  1. can you make sure there is no other container(s) running/using same port?

Yes, here's the result from docker ps:

CONTAINER ID   IMAGE                               COMMAND                  CREATED          STATUS         PORTS                                           NAMES
ea1ae12c3207   prometheuscommunity/json-exporter   "/bin/json_exporter"     5 seconds ago    Up 2 seconds   0.0.0.0:7979->7979/tcp, :::7979->7979/tcp       json_exporter
064d59b47369   rmfpoc_zebra                        "docker-entrypoint.s…"   6 seconds ago    Up 5 seconds   0.0.0.0:3090->3090/tcp, :::3090->3090/tcp       zebra
4a9fb78425c3   grafana/grafana:8.2.2               "/run.sh"                7 seconds ago    Up 6 seconds   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp       zebra-grafana
714f756a3cea   prom/prometheus:v2.30.3             "/bin/prometheus --c…"   11 seconds ago   Up 7 seconds   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp       zebra-prometheus
ed179fe576ff   mongo:5.0.3                         "docker-entrypoint.s…"   11 seconds ago   Up 9 seconds   0.0.0.0:27017->27017/tcp, :::27017->27017/tcp   zebra-mongo
  1. is the connection to outside working well from container?

(you can use curl command too - and use proper options for your settings)

this will make sure there is valid connection from docker to DDS service.

docker exec -it zebra "bash"
root@064d59b47369:/zebra#  wget --no-check-certificate "http://[USERNAME]:[PASSWORD]@[IP]:8803/gpm/rmfm3.xml?report=CPC&resource=%22,RPRT,MVS_IMAGE%22"
--2022-09-27 19:03:20--  http://[USERNAME]:[PASSWORD]@[IP]:8803/gpm/rmfm3.xml?report=CPC&resource=%22,RPRT,MVS_IMAGE%22
Connecting to [IP]:8803... connected.
HTTP request sent, awaiting response... 401 Not authorized
Authentication selected: Basic realm="TSO userid on S00"
Connecting to [IP]:8803... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1324 (1.3K) [application/xml]
Saving to: 'rmfm3.xml?report=CPC&resource=%22,RPRT,MVS_IMAGE%22'

rmfm3.xml?report=CPC&resource=%22,RPRT,MVS_IMAGE%2 100%[================================================================================================================>]   1.29K  --.-KB/s    in 0s

2022-09-27 19:03:20 (75.4 MB/s) - 'rmfm3.xml?report=CPC&resource=%22,RPRT,MVS_IMAGE%22' saved [1324/1324]

root@064d59b47369:/zebra# ls
Auth.js     Zebra.yml       app_server  constants.js    metrics.js    mongoV1.js    nedbAdmin.js  package-lock.json  rmfm3.xml?report=CPC&resource=%22,RPRT,MVS_IMAGE%22
Dockerfile  Zebra_Swagger.json  bin     docker-compose.yml  metrics.json  my.db     node_modules  package.json       sslcert
Readme.md   app.js      config      manifest.yml    mongo.js      my.dbrefresh  nodemon.json  public
root@064d59b47369:/zebra# cat rmfm3.xml\?report\=CPC\&resource\=%22\,RPRT\,MVS_IMAGE%22
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="include/ddsml-m3.xsl"?>
<ddsml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="include/ddsml.xsd">
<server>
<name>RMF-DDS-Server</name>
<version>ZOSV2R5</version>
<functionality>3500</functionality>
<platform>z/OS</platform>
</server>
<report>
<metric id="CPC">
<description>CPC (Central Processor Complex)</description>
<format>report</format>
<numcols>2</numcols>
</metric>
<resource>
<reslabel>,RPRT,MVS_IMAGE</reslabel>
<restype>MVS_IMAGE</restype>
<reslabelurl>%2CRPRT%2CMVS_IMAGE</reslabelurl>
</resource>
<time-data>
<local-start>20220927150000</local-start>
<local-end>20220927150140</local-end>
<utc-start>20220927190000</utc-start>
<utc-end>20220927190140</utc-end>
<local-prev>20220927145910</local-prev>
<local-next>20220927150230</local-next>
<display-start locale="en-us">09/27/2022 15:00:00</display-start>
<display-end locale="en-us">09/27/2022 15:01:40</display-end>
<gatherer-interval unit="seconds">100</gatherer-interval>
<data-range unit="seconds">100</data-range>
</time-data>
<message id="GPM0731I">
<description>The specified system 'RPRT' is not a member of the sysplex</description>
<severity>3</severity>
</message>
</report>
</ddsml>

...I realize this is the wrong data, but it seems to work. Just to confirm, however:

docker exec -it zebra "bash"
root@baed0c580191:/zebra#  wget --no-check-certificate "http://[USERNAME]:[PASSWORD]@[IP]:8803/gpm/rmfm3.xml?report=CPC&resource=%22,S01,MVS_IMAGE%22"
--2022-09-27 19:14:19--  http://[USERNAME]:[PASSWORD]@[IP]:8803/gpm/rmfm3.xml?report=CPC&resource=%22,S01,MVS_IMAGE%22
Connecting to [IP]:8803... connected.
HTTP request sent, awaiting response... 401 Not authorized
Authentication selected: Basic realm="TSO userid on S00"
Connecting to [IP]:8803... connected.
HTTP request sent, awaiting response... 200 OK
Length: 32610 (32K) [application/xml]
Saving to: 'rmfm3.xml?report=CPC&resource=%22,S01,MVS_IMAGE%22'

rmfm3.xml?report=CPC&resource=%22,S01,MVS_IMAGE%22 100%[================================================================================================================>]  31.85K  --.-KB/s    in 0.001s

2022-09-27 19:14:19 (53.5 MB/s) - 'rmfm3.xml?report=CPC&resource=%22,S01,MVS_IMAGE%22' saved [32610/32610]

root@baed0c580191:/zebra# cat rmfm3.xml\?report\=CPC\&resource\=%22\,S01\,MVS_IMAGE%22
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="include/ddsml-m3.xsl"?>
<ddsml xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="include/ddsml.xsd">
<server>
<name>RMF-DDS-Server</name>
<version>ZOSV2R5</version>
<functionality>3500</functionality>
<platform>z/OS</platform>
</server>
<report>
<metric id="CPC">
<description>CPC (Central Processor Complex)</description>
<format>report</format>
<numcols>35</numcols>
</metric>
<resource>
<reslabel>,S01,MVS_IMAGE</reslabel>
<restype>MVS_IMAGE</restype>
<reslabelurl>%2CS01%2CMVS_IMAGE</reslabelurl>
</resource>
<time-data>
<local-start>20220927151140</local-start>
<local-end>20220927151320</local-end>
<utc-start>20220927191140</utc-start>
<utc-end>20220927191320</utc-end>
<local-prev>20220927151050</local-prev>
<local-next>20220927151410</local-next>
<display-start locale="en-us">09/27/2022 15:11:40</display-start>
<display-end locale="en-us">09/27/2022 15:13:20</display-end>
<gatherer-interval unit="seconds">100</gatherer-interval>
<data-range unit="seconds">100</data-range>
<num-samples>100</num-samples>
</time-data>
<caption><var><name>CPCHPNAM</name><value>T13AHP12</value></var><var><name>CPCHMOD</name><value>3931</value></var><var><name>CPCHMDL</name><value>7D2</value></var><var><name>CPCHBSTT</name><value>N</value></var><var><name>CPCHCMSU</name><value>18098</value></var><var><name>CPCHWF</name><value>****</value></var><var><name>CPCHLMSU</name><value>988</value></var><var><name>CPCHGNAM</name><value>N/A</value></var><var><name>CPCHIMSU</name><value>1097</value></var><var><name>CPCHCAP</name><value>0.0</value></var><var><name>CPCHLMAX</name><value>1011</value></var><var><name>CPCHGLIM</name><value>N/A</value></var><var><name>CPCHGL4H</name><value></value></var><var><name>CPCHMTMS</name><value>1</value></var><var><name>CPCHPRDS</name><value>100</value></var><var><name>CPCHAMSU</name><value>N</value></var><var><name>CPCHRMSU</name><value>14400</value></var><var><name>CPCHRGRP</name><value>N/A</value></var><var><name>CPCHGAUN</name><value>N/A</value></var><var><name>CPCHCPU</name><value>0000000000065C68</value></var><var><name>CPCHCPCN</name><value>LCST1</value></var><var><name>CPCHCPNO</name><value>132</value></var><var><name>CPCHIFAN</name><value>0</value></var><var><name>CPCHCBPN</name><value>N/A</value></var><var><name>CPCHICFN</name><value>4</value></var><var><name>CPCHIFLN</name><value>55</value></var><var><name>CPCHSUPN</name><value>3</value></var><var><name>CPCHPANO</name><value>83</value></var><var><name>CPCHWAIT</name><value>NO</value></var><var><name>CPCHPMSU</name><value>89</value></var><var><name>CPCHDEDC</name><value>0</value></var><var><name>CPCHDEDA</name><value>0</value></var><var><name>CPCHDEDO</name><value>N/A</value></var><var><name>CPCHDEDI</name><value>0</value></var><var><name>CPCHSHRC</name><value>132</value></var><var><name>CPCHSHRA</name><value>0</value></var><var><name>CPCHSHRO</name><value>N/A</value></var><var><name>CPCHSHRI</name><value>3</value></var><var><name>CPCHVCPU</name><value>NO</value></var><var><name>CPCHWMGT</name><value>NO</value></var><var><name>CPCHCCAI</name><value>100</value></var><var><name>CPCHCCCR</name><value>0</value></var><var><name>CPCHPRD</name><value>100</value></var><var><name>CPCHMCFS</name><value>1.00</value></var><var><name>CPCHMCF</name><value>1.00</value></var><var><name>CPCHCFS</name><value>1.00</value></var><var><name>CPCHCF</name><value>1.00</value></var><var><name>CPCHATDS</name><value>1.00</value></var><var><name>CPCHATD</name><value>1.00</value></var><var><name>CPCHMTM</name><value>1</value></var><var><name>CPCHMDLX</name><value></value></var><var><name>CPCHBSTC</name><value>None</value></var><var><name>CPCHCUTL</name><value>23.2</value></var><var><name>CPCHAUTL</name><value>0.0</value></var><var><name>CPCHOUTL</name><value>N/A</value></var><var><name>CPCHUUTL</name><value>0.3</value></var><var><name>CPCHLUTL</name><value>0.0</value></var><var><name>CPCHFUTL</name><value>0.4</value></var></caption>
<row refno="1"><col>*CP</col> <col/> <col/> <col/> <col>180</col> <col/> <col/> <col>0.4</col> <col>23.0</col> <col>23.4</col> <col>CS</col> <col>180</col> <col>0</col> <col>2101</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="2"><col>S13AOSA1</col> <col>0</col> <col>218</col> <col>N N N</col> <col>8.0</col> <col>19.9</col> <col>20.2</col> <col>0.0</col> <col>1.2</col> <col>1.2</col> <col>CP</col> <col>8</col> <col>0</col> <col>80</col> <col>62.8</col> <col>N/A</col> <col>N/A</col> <col>N/A</col> <col>SAK138D2</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>13</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="3"><col>S13AOSA2</col> <col>0</col> <col>214</col> <col>N N N</col> <col>8.0</col> <col>19.5</col> <col>19.9</col> <col>0.0</col> <col>1.2</col> <col>1.2</col> <col>CP</col> <col>8</col> <col>0</col> <col>10</col> <col>7.8</col> <col>N/A</col> <col>N/A</col> <col>N/A</col> <col>SAK138D2</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>65536</col> <col>29</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="4"><col>S13AOSA3</col> <col>0</col> <col>171</col> <col>N N N</col> <col>8.0</col> <col>15.6</col> <col>15.9</col> <col>0.0</col> <col>0.9</col> <col>1.0</col> <col>CP</col> <col>8</col> <col>0</col> <col>10</col> <col>7.8</col> <col>N/A</col> <col>N/A</col> <col>N/A</col> <col>SAK138D2</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>65536</col> <col>70</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="5"><col>T13AHP11</col> <col>0</col> <col>545</col> <col>N N N</col> <col>4.0</col> <col>99.4</col> <col>99.5</col> <col>0.0</col> <col>3.0</col> <col>3.0</col> <col>CP</col> <col>4</col> <col>0</col> <col>100</col> <col>0.0</col> <col>4</col> <col>0</col> <col>0</col> <col>S00</col> <col>ENGTEST1</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>3</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="6"><col>T13AHP12</col> <col>0</col> <col>975</col> <col>N N N</col> <col>8.0</col> <col>88.8</col> <col>89.2</col> <col>0.0</col> <col>5.4</col> <col>5.4</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S01</col> <col>ENGTEST1</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>4</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="7"><col>T13AHP13</col> <col>0</col> <col>959</col> <col>N N N</col> <col>8.0</col> <col>87.4</col> <col>87.8</col> <col>0.0</col> <col>5.3</col> <col>5.3</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S02</col> <col>ENGTEST1</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>5</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="8"><col>T13AHP14</col> <col>0</col> <col>939</col> <col>N N N</col> <col>8.0</col> <col>85.6</col> <col>85.9</col> <col>0.0</col> <col>5.2</col> <col>5.2</col> <col>CP</col> <col>8</col> <col>0</col> <col>60</col> <col>76.9</col> <col>3</col> <col>1</col> <col>4</col> <col>S03</col> <col>ENGTEST1</col> <col>60</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>6</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="9"><col>T13AHP21</col> <col>0</col> <col>3</col> <col>N N N</col> <col>8.0</col> <col>0.2</col> <col>0.3</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>CP</col> <col>8</col> <col>0</col> <col>250</col> <col>0.0</col> <col>8</col> <col>0</col> <col>0</col> <col>S00</col> <col>ENGTEST2</col> <col>250</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>19</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="10"><col>T13AHP22</col> <col>0</col> <col>8</col> <col>N N N</col> <col>8.0</col> <col>0.8</col> <col>0.8</col> <col>0.0</col> <col>0.0</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S01</col> <col>ENGTEST2</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>20</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="11"><col>T13AHP23</col> <col>0</col> <col>8</col> <col>N N N</col> <col>8.0</col> <col>0.7</col> <col>0.8</col> <col>0.0</col> <col>0.0</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S02</col> <col>ENGTEST2</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>21</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="12"><col>T13AHP24</col> <col>0</col> <col>9</col> <col>N N N</col> <col>8.0</col> <col>0.8</col> <col>0.9</col> <col>0.0</col> <col>0.0</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>60</col> <col>76.9</col> <col>3</col> <col>1</col> <col>4</col> <col>S03</col> <col>ENGTEST2</col> <col>60</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>22</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="13"><col>T13AHP31</col> <col>0</col> <col>13</col> <col>N N N</col> <col>8.0</col> <col>1.2</col> <col>1.3</col> <col>0.0</col> <col>0.1</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S00</col> <col>ENGTEST3</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>35</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="14"><col>T13AHP32</col> <col>0</col> <col>12</col> <col>N N N</col> <col>8.0</col> <col>1.1</col> <col>1.2</col> <col>0.0</col> <col>0.1</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S01</col> <col>ENGTEST3</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>36</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="15"><col>T13AHP33</col> <col>0</col> <col>12</col> <col>N N N</col> <col>8.0</col> <col>1.1</col> <col>1.2</col> <col>0.0</col> <col>0.1</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S02</col> <col>ENGTEST3</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>37</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="16"><col>T13AHP34</col> <col>0</col> <col>12</col> <col>N N N</col> <col>8.0</col> <col>1.1</col> <col>1.1</col> <col>0.0</col> <col>0.1</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>60</col> <col>76.9</col> <col>3</col> <col>1</col> <col>4</col> <col>S03</col> <col>ENGTEST3</col> <col>60</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>38</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="17"><col>T13AHP41</col> <col>0</col> <col>3</col> <col>N N N</col> <col>8.0</col> <col>0.2</col> <col>0.3</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S00</col> <col>ENGTEST4</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>51</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="18"><col>T13AHP42</col> <col>0</col> <col>8</col> <col>N N N</col> <col>8.0</col> <col>0.7</col> <col>0.8</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S01</col> <col>ENGTEST4</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>52</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="19"><col>T13AHP43</col> <col>0</col> <col>8</col> <col>N N N</col> <col>8.0</col> <col>0.7</col> <col>0.8</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S02</col> <col>ENGTEST4</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>53</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="20"><col>T13AHP44</col> <col>0</col> <col>7</col> <col>N N N</col> <col>8.0</col> <col>0.6</col> <col>0.7</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>CP</col> <col>8</col> <col>0</col> <col>60</col> <col>76.9</col> <col>3</col> <col>1</col> <col>4</col> <col>S03</col> <col>ENGTEST4</col> <col>60</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>98304</col> <col>54</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="21"><col>T13AHP51</col> <col>0</col> <col>8</col> <col>N N N</col> <col>8.0</col> <col>0.8</col> <col>0.8</col> <col>0.0</col> <col>0.0</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S00</col> <col>ENGTEST5</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>40960</col> <col>7</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="22"><col>T13AHP52</col> <col>0</col> <col>9</col> <col>N N N</col> <col>8.0</col> <col>0.8</col> <col>0.9</col> <col>0.0</col> <col>0.0</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S01</col> <col>ENGTEST5</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>40960</col> <col>23</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="23"><col>T13AHP53</col> <col>0</col> <col>9</col> <col>N N N</col> <col>8.0</col> <col>0.8</col> <col>0.9</col> <col>0.0</col> <col>0.0</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>100</col> <col>64.1</col> <col>5</col> <col>2</col> <col>1</col> <col>S02</col> <col>ENGTEST5</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>40960</col> <col>39</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="24"><col>T13AHP54</col> <col>0</col> <col>8</col> <col>N N N</col> <col>8.0</col> <col>0.8</col> <col>0.8</col> <col>0.0</col> <col>0.0</col> <col>0.1</col> <col>CP</col> <col>8</col> <col>0</col> <col>111</col> <col>97.3</col> <col>6</col> <col>1</col> <col>1</col> <col>S03</col> <col>ENGTEST5</col> <col>111</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>40960</col> <col>55</col> <col>NO</col> <col/> <col/> <col/> <col/> <col>N</col></row>
<row refno="25"><col>PHYSICAL</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>0.2</col> <col/> <col>0.2</col> <col>CY</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>0</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="26"><col></col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="27"><col>*IFL</col> <col/> <col/> <col/> <col>135</col> <col/> <col/> <col>0.1</col> <col>5.8</col> <col>5.8</col> <col>LS</col> <col>132</col> <col>3</col> <col>910</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="28"><col>L13AOSA1</col> <col/> <col/> <col>N N N</col> <col>8.0</col> <col>1.8</col> <col>1.8</col> <col>0.0</col> <col>0.3</col> <col>0.3</col> <col>LP</col> <col>8</col> <col>0</col> <col>100</col> <col>71.4</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="29"><col>L13AOSA2</col> <col/> <col/> <col>N N N</col> <col>8.0</col> <col>0.3</col> <col>0.4</col> <col>0.0</col> <col>0.0</col> <col>0.1</col> <col>LP</col> <col>8</col> <col>0</col> <col>100</col> <col>71.4</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="30"><col>T13ALNX1</col> <col/> <col/> <col>N N N</col> <col>8.0</col> <col>0.1</col> <col>0.1</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>LP</col> <col>8</col> <col>0</col> <col>100</col> <col>71.4</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="31"><col>T13ALNX2</col> <col/> <col/> <col>N N N</col> <col>1.0</col> <col>100</col> <col>100</col> <col>0.0</col> <col>1.8</col> <col>1.8</col> <col>LP</col> <col>0</col> <col>1</col> <col>0</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="32"><col>T13ALNX3</col> <col/> <col/> <col>N N N</col> <col>4.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>LP</col> <col>4</col> <col>0</col> <col>100</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="33"><col>T13ALNX4</col> <col/> <col/> <col>N N N</col> <col>16.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>LP</col> <col>16</col> <col>0</col> <col>10</col> <col>3.5</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="34"><col>T13ALNX5</col> <col/> <col/> <col>N N N</col> <col>24.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>LP</col> <col>24</col> <col>0</col> <col>100</col> <col>23.8</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="35"><col>T13ALNX6</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>100</col> <col>100</col> <col>0.0</col> <col>3.6</col> <col>3.6</col> <col>LP</col> <col>0</col> <col>2</col> <col>0</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="36"><col>T13ASE1</col> <col/> <col/> <col>N N N</col> <col>16.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>LP</col> <col>16</col> <col>0</col> <col>100</col> <col>35.7</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="37"><col>T13ASE2</col> <col/> <col/> <col>N N N</col> <col>16.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>LP</col> <col>16</col> <col>0</col> <col>100</col> <col>35.7</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="38"><col>T13ASE3</col> <col/> <col/> <col>N N N</col> <col>16.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>LP</col> <col>16</col> <col>0</col> <col>100</col> <col>35.7</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="39"><col>T13ASE4</col> <col/> <col/> <col>N N N</col> <col>16.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>LP</col> <col>16</col> <col>0</col> <col>100</col> <col>35.7</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col/> <col/></row>
<row refno="40"><col>PHYSICAL</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>0.0</col> <col/> <col>0.0</col> <col>LY</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="41"><col></col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="42"><col>*ICF</col> <col/> <col/> <col/> <col>2.0</col> <col/> <col/> <col>1.6</col> <col>50.0</col> <col>51.6</col> <col>FS</col> <col>0</col> <col>2</col> <col>0</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="43"><col>PL4CF13A</col> <col/> <col/> <col>N N N</col> <col>1.0</col> <col>100</col> <col>100</col> <col>0.0</col> <col>25.0</col> <col>25.0</col> <col>FP</col> <col>0</col> <col>1</col> <col>0</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="44"><col>PL5CF13A</col> <col/> <col/> <col>N N N</col> <col>1.0</col> <col>100</col> <col>100</col> <col>0.0</col> <col>25.0</col> <col>25.0</col> <col>FP</col> <col>0</col> <col>1</col> <col>0</col> <col>100</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="45"><col>PHYSICAL</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>1.6</col> <col/> <col>1.6</col> <col>FY</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="46"><col></col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="47"><col>*IIP</col> <col/> <col/> <col/> <col>40.0</col> <col/> <col/> <col>0.0</col> <col>0.3</col> <col>0.3</col> <col>US</col> <col>40</col> <col>0</col> <col>746</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<row refno="48"><col>T13AHP11</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="49"><col>T13AHP12</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>101</col> <col>40.6</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="50"><col>T13AHP13</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>101</col> <col>40.6</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="51"><col>T13AHP14</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>101</col> <col>40.6</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="52"><col>T13AHP21</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>101</col> <col>40.6</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="53"><col>T13AHP22</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>101</col> <col>40.6</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="54"><col>T13AHP23</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>101</col> <col>40.6</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="55"><col>T13AHP24</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="56"><col>T13AHP31</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="57"><col>T13AHP32</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="58"><col>T13AHP33</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="59"><col>T13AHP34</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="60"><col>T13AHP41</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="61"><col>T13AHP42</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="62"><col>T13AHP43</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="63"><col>T13AHP44</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="64"><col>T13AHP51</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="65"><col>T13AHP52</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="66"><col>T13AHP53</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="67"><col>T13AHP54</col> <col/> <col/> <col>N N N</col> <col>2.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>0.0</col> <col>UP</col> <col>2</col> <col>0</col> <col>10</col> <col>4.0</col> <col>0</col> <col>1</col> <col>1</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col>NO</col> <col/> <col/> <col/> <col>N</col> <col/></row>
<row refno="68"><col>PHYSICAL</col> <col/> <col/> <col/> <col/> <col/> <col/> <col>0.0</col> <col/> <col>0.0</col> <col>UY</col> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/> <col/></row>
<column-headers> <col type="T">CPCPPNAM</col> <col type="N">CPCPDMSU</col> <col type="N">CPCPAMSU</col> <col type="T">CPCPCAPD</col> <col type="N">CPCPLPNO</col> <col type="N">CPCPLEFU</col> <col type="N">CPCPLTOU</col> <col type="N">CPCPPLMU</col> <col type="N">CPCPPEFU</col> <col type="N">CPCPPTOU</col> <col type="T">CPCPIND</col> <col type="N">CPCPLPND</col> <col type="N">CPCPDEDP</col> <col type="N">CPCPWGHT</col> <col type="N">CPCPLPSH</col> <col type="T">CPCPVCMH</col> <col type="T">CPCPVCMM</col> <col type="T">CPCPVCML</col> <col type="T">CPCPOSNM</col> <col type="T">CPCPLPCN</col> <col type="N">CPCPLCIW</col> <col type="N">CPCPLCMW</col> <col type="N">CPCPLCXW</col> <col type="N">CPCPCGNM</col> <col type="N">CPCPCGLT</col> <col type="N">CPCPCGEM</col> <col type="N">CPCPCGEX</col> <col type="N">CPCPCSMB</col> <col type="N">CPCPUPID</col> <col type="T">CPCPCAPI</col> <col type="N">CPCPHWCC</col> <col type="N">CPCPHGNM</col> <col type="N">CPCPHWGC</col> <col type="T">CPCPBIIP</col> <col type="T">CPCPBSPD</col> </column-headers>
</report>
</ddsml>

Yes, it is talking to it.

behives commented 2 years ago

Thanks @liam-hogan

Can you check the NodeJS version on the container?

liam-hogan commented 2 years ago

@behives Yes, it is: v8.17.0

behives commented 2 years ago

Thanks @liam-hogan

Is the zebra container the old image or have you tried to rebuild the image from scratch recently? (in other word, how old is the Zebra image?)

What changed from two weeks ago from your process point of view?

Have you tried to run Zebra natively but not as docker? FYI - you won't need prometheus nor MongoDB to just simply connect Zebra to DDS...just clone main repo and connect with Zconfig.json you have.

liam-hogan commented 2 years ago

@behives For full context, I am building a script to auto-deploy zebra alongside Grafana/Prometheus/Mongo/json_exporter. Part of my testing was to make sure that if zebra isn't available, then it will pull in the repo.

For my test, I made sure that it would work if I needed to deploy with basically nothing downloaded and built yet.

I was incorrect with my date. This was working this past Thursday (9/23) as I can see from my private repo commits. After I removed and rebuilt, the issue appeared.

Here's the git status for zebra, you can see it's up to date and the only modded file is Prometheus (Zconfig is modified too, but gitignore doesn't show that):

git status
On branch main
Your branch is up to date with 'origin/main'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   src/config/prometheus/prometheus.yml

no changes added to commit (use "git add" and/or "git commit -a")

Also, I will try without Mongo and Prometheus to see if that changes anything.

behives commented 2 years ago

@liam-hogan thanks for the details.

I would first recommend to try it as native NodeJS app but not as Docker - often time the issue could be in communications/networking but it's hard to debug. Therefore, if you see the same issue with native node app we can debug it easily.

Please let me know after trying is as node app without docker if it has same issues....

thanks! Alex

behives commented 2 years ago

@liam-hogan the only guess I can think of now is , there could be residual container components didn't get updated while main zebra got updated - since the code was updated in repo about two weeks ago. Check if you can list details of the zebra container and try deleting everything and rebuild everything.

liam-hogan commented 2 years ago

Hey @behives , sorry for taking long to get back on this. I finally figured it out!

So, I did as suggested. When I installed node (just a standard install), I got the following version:

root@vm-671:~/RMFPoC# node -v
v10.19.0

Running a local install worked perfectly fine, so that definitely indicates an issue with the Docker install.

In the Dockerfile, the version is currently set to 8-slim. I updated the Docker FROM line as following:

FROM node:10-slim

This worked successfully and I am able to run the stack without an issue. However, I noticed some lines in the Docker build that probably indicates that this isn't the cleanest way to do this:

docker-compose up --build -d
Creating network "rmfpoc_default" with the default driver
Building zebra
Step 1/6 : FROM node:10-slim
10-slim: Pulling from library/node
62deabe7a6db: Pull complete
f698164f6049: Pull complete
bc29352cb629: Pull complete
85e84b4c858f: Pull complete
ac72e4359589: Pull complete
Digest: sha256:88932859e3d022d79161b99628c4c2c50e836437455e2d1b1a008d98367b10d6
Status: Downloaded newer image for node:10-slim
 ---> 6fbcbbb5c603
Step 2/6 : WORKDIR /zebra
 ---> Running in d50eaeb5eb7d
Removing intermediate container d50eaeb5eb7d
 ---> a512f56f99bc
Step 3/6 : COPY package*.json ./
 ---> 86b970122e30
Step 4/6 : RUN npm install
 ---> Running in 08a14f164dd1
npm WARN read-shrinkwrap This version of npm is compatible with lockfileVersion@1, but package-lock.json was generated for lockfileVersion@2. I'll try to do my best with it!
added 308 packages from 397 contributors and audited 311 packages in 5.086s

10 packages are looking for funding
  run `npm fund` for details

found 13 vulnerabilities (1 low, 1 moderate, 6 high, 5 critical)
  run `npm audit fix` to fix them, or `npm audit` for details
Removing intermediate container 08a14f164dd1
 ---> 02b003f5df2d
Step 5/6 : COPY . .
 ---> 6a791f317dea
Step 6/6 : CMD [ "npm", "start" ]
 ---> Running in fcff600f16be
Removing intermediate container fcff600f16be
 ---> 1e5d293fbb89
Successfully built 1e5d293fbb89
Successfully tagged rmfpoc_zebra:latest

Either way, it looks as though the node version may be the issue.

behives commented 2 years ago

Hi @liam-hogan ,

I'm glad to hear you found the issue. Zebra was tested to run okay with Node version 8.1x, due to our initial setup with DDS but it seems like it's also good with version 10.x. ( I think @jsanter27 fixed the issue with TLS issue sometime ago)

Thanks for the update and letting us know. I will close this issue now but if you see any other issues/bugs please open again.

Thanks, Alex