search

telefonicaid / fiware-sth-comet

A component of the FIWARE ecosystem in charge of managing historical and aggregated time series context information
https://fiware-sth-comet.rtfd.io
GNU Affero General Public License v3.0
27 stars 28 forks source link

Problem when querying Raw Data with STH-Comet - Returns empty #545

Open Rosiberto opened 4 years ago

Rosiberto commented 4 years ago

I have Orion, Cygnus and STH-Comet(installed and configured in formal mode). Each component is in a container docker. I implemented the infrastructure with docker-compose.yml.

The Cygnus container is configured as follows:

image: fiware/cygnus-ngsi:latest
    hostname: cygnus
    container_name: cygnus
    volumes:
      - /home/ubuntu/cygnus/multisink_agent.conf:/opt/fiware-cygnus/docker/cygnus-ngsi/multisink_agent.conf
    depends_on:
      - mongo
    networks:
      - default
    expose:
      - "5050"
      - "5080"
    ports:
      - "5050:5050"
      - "5080:5080" 
    environment:
      - CYGNUS_SERVICE_PORT=5050
      - CYGNUS_MONITORING_TYPE=http
      - CYGNUS_AGENT_NAME=cygnus-ngsi
      - CYGNUS_MONGO_SERVICE_PORT=5050
      - CYGNUS_MONGO_HOSTS=mongo:27017
      - CYGNUS_MONGO_USER=
      - CYGNUS_MONGO_PASS=
      - CYGNUS_MONGO_ENABLE_ENCODING=false
      - CYGNUS_MONGO_ENABLE_GROUPING=false
      - CYGNUS_MONGO_ENABLE_NAME_MAPPINGS=false
      - CYGNUS_MONGO_DATA_MODEL=dm-by-entity
      - CYGNUS_MONGO_ATTR_PERSISTENCE=column
      - CYGNUS_MONGO_DB_PREFIX=sth_
      - CYGNUS_MONGO_COLLECTION_PREFIX=sth_   
      - CYGNUS_MONGO_ENABLE_LOWERCASE=false
      - CYGNUS_MONGO_BATCH_TIMEOUT=30
      - CYGNUS_MONGO_BATCH_TTL=10
      - CYGNUS_MONGO_DATA_EXPIRATION=0
      - CYGNUS_MONGO_COLLECTIONS_SIZE=0
      - CYGNUS_MONGO_MAX_DOCUMENTS=0
      - CYGNUS_MONGO_BATCH_SIZE=1
      - CYGNUS_LOG_LEVEL=DEBUG
      - CYGNUS_SKIP_CONF_GENERATION=false 
      - CYGNUS_STH_ENABLE_ENCODING=false
      - CYGNUS_STH_ENABLE_GROUPING=false
      - CYGNUS_STH_ENABLE_NAME_MAPPINGS=false
      - CYGNUS_STH_DB_PREFIX=sth_
      - CYGNUS_STH_COLLECTION_PREFIX=sth_
      - CYGNUS_STH_DATA_MODEL=dm-by-entity
      - CYGNUS_STH_ENABLE_LOWERCASE=false
      - CYGNUS_STH_BATCH_TIMEOUT=30
      - CYGNUS_STH_BATCH_TTL=10
      - CYGNUS_STH_DATA_EXPIRATION=0
      - CYGNUS_STH_BATCH_SIZE=1

Obs: In the multisink_agent.conf file I changed the service and the servicepath:

cygnus-ngsi.sources.http-source-mongo.handler.default_service = tese

cygnus-ngsi.sources.http-source-mongo.handler.default_service_path = /iot

And the STH-Comet container looks like this:

image: fiware/sth-comet:latest
    hostname: sth
    container_name: sth
    depends_on:
      - cygnus
      - mongo
    networks:
      - default
    expose:
      - "8666"
    ports:
      - "8666:8666"
    environment:
      - STH_HOST=0.0.0.0
      - STH_PORT=8666
      - DB_URI=mongo:27017
      - DB_USERNAME=
      - DB_PASSWORD=
      - LOGOPS_LEVEL=DEBUG

In the STH-Comet config.js file I enabled CORS: config.cors = { enabled: 'true', ... }; and I changed the defaultService to tese and the defaultServicePath to /iot.

I use Cygnus to persist historical data. STH-Comet is used only to query raw and aggregated data.

Cygnus' signature on Orion did this:

"description": "A subscription All Entities",
  "subject": {
    "entities": [
      {
        "idPattern": ".*"
      }
    ],
    "condition": {
      "attrs": []
    }
  },
  "notification": {
    "http": {
      "url": "http://cygnus:5050/notify"
    },
    "attrs": [],
    "attrsFormat":"legacy"
  },
  "expires": "2040-01-01T14:00:00.00Z",
  "throttling": 5
}

The headers used for fiware-service and fiware-servicepath are:

Fiware-service: tese

Fiware-servicepath: /iot

The entities data are stored in orion-tese. I have the collection: entities

{
        "_id" : {
                "id" : "Tank1",
                "type" : "Tank",
                "servicePath" : "/iot"
        },
        "attrNames" : [
                "temperature"
        ],
        "attrs" : {
                "temperature" : {
                        "value" : 0.333,
                        "type" : "Float",
                        "mdNames" : [ ],
                        "creDate" : 1594334464,
                        "modDate" : 1594337770
                }
        },
        "creDate" : 1594334464,
        "modDate" : 1594337771,
        "lastCorrelator" : "f86d0d74-c23c-11ea-9c82-0242ac1c0005"
}

The raw and aggregated data are stored in sth_tese. I have the collections:

sth_/iot_Tank1_Tank.aggr and sth_/iot_Tank1_Tank

The sth_/iot_Tank1_Tank raw data is in mongoDB:

{
        "_id" : ObjectId("5f079d0369591c06b0fc981a"),
        "temperature" : 279,
        "recvTime" : ISODate("2020-07-09T22:41:05.670Z")
}
{
        "_id" : ObjectId("5f07a9eb69591c06b0fc981b"),
        "temperature" : 0.333,
        "recvTime" : ISODate("2020-07-09T23:36:11.160Z")
}

When I run: http://localhost:8666/STH/v1/contextEntities/type/Tank/id/Tank1/attributes/temperature?aggrMethod=sum&aggrPeriod=minute

or

http://localhost:8666/STH/v2/entities/Tank1/attrs/temperature?type=Tank&aggrMethod=sum&aggrPeriod=minute

I have the result: "sum": 279 and "sum": 0.333. I can recover ALL the aggregated data, max, min, sum, sum2.

The difficulty is with the STH-Comet when I try to retrieve the raw data, the return code is 200 and the value returns empty.

I've tried with APIs v1 and v2, to no avail.

request with v2:

http://sth:8666/STH/v2/entities/Tank1/attrs/temperature?type=Tank&lastN=10

Return

{
  "type": "StructuredValue",
  "value": []
}

request with v1:

http://sth:8666/STH/v1/contextEntities/type/Tank/id/Tank1/attributes/temperature?lastN=10

Return

{
  "contextResponses": [{
    "contextElement": {
      "attributes": [{
        "name": "temperature",
        "values": []
      }],
      "id": "Tank1",
      "isPattern": false,
      "type": "Tank"
    },
    "statusCode": {
      "code": "200",
      "reasonPhrase": "OK"
    }
  }]
}

The STH-Comet log shows that it is online and connects to the correct database:

time=2020-07-09T22:39:06.698Z | lvl=INFO | corr=n/a | trans=n/a | op=OPER_STH_DB_CONN_OPEN | from=n/a | srv=n/a | subsrv=n/a | comp=STH | msg=Establishing connection to the database at mongodb://@mongo:27017/sth_tese
time=2020-07-09T22:39:06.879Z | lvl=INFO | corr=n/a | trans=n/a | op=OPER_STH_DB_CONN_OPEN | from=n/a | srv=n/a | subsrv=n/a | comp=STH | msg=Connection successfully established to the database at mongodb://@mongo:27017/sth_tese
time=2020-07-09T22:39:07.218Z | lvl=INFO | corr=n/a | trans=n/a | op=OPER_STH_SERVER_START | from=n/a | srv=n/a | subsrv=n/a | comp=STH | msg=Server started at http://0.0.0.0:8666

The STH-Comet log with the api v2 request:

time=2020-07-09T23:46:47.400Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=GET /STH/v2/entities/Tank1/attrs/temperature?type=Tank&lastN=10
time=2020-07-09T23:46:47.404Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=Getting access to the raw data collection for retrieval...
time=2020-07-09T23:46:47.408Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=The raw data collection for retrieval exists
time=2020-07-09T23:46:47.412Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=No raw data available for the request: /STH/v2/entities/Tank1/attrs/temperature?type=Tank&lastN=10
time=2020-07-09T23:46:47.412Z | lvl=DEBUG | corr=998811d9-fac2-4701-b37c-bb9ae1b45b81 | trans=998811d9-fac2-4701-b37c-bb9ae1b45b81 | op=OPER_STH_GET | from=n/a | srv=tese | subsrv=/iot | comp=STH | msg=Responding with no points

According to the log, it establishes the connection to recover the raw data: msg=Getting access to the raw data collection for retrieval.... Confirms that the raw data exists: msg=The raw data collection for retrieval exists. But, it cannot recover this data and generates the message that the raw data is not available and does not return any points: msg=No raw data available for the request and msg=Responding with no points.

I already read the configuration part in the documentation. I've reinstalled everything, several times. I combed all settings and I can't find anything to justify this problem.

What could it be? Could someone with expertise in STH-Comet give any guidance?

Thanks!

fgalan commented 4 years ago

This is the same case described in SOF question https://stackoverflow.com/questions/62327070/consult-raw-data-with-sth-comet?noredirect=1#comment110984866_62327070 ?

Rosiberto commented 4 years ago

Yes, @fgalan. It is the same question

.

fgalan commented 4 years ago

Let's continue the debug in SOF. Please have a look to my last comment there (I have requested a full detail of the config.js being used).