search

noi-techpark / bdp-core

Open Data Hub / Timeseries Core
https://opendatahub.com
Other
9 stars 4 forks source link

As a maintainer of the Big Data Platform v2, I want to use all data collectors as in v1 #58

Closed Piiit closed 5 years ago

Piiit commented 5 years ago

Belongs to the EPIC #28 Datacollectors to mirror which are running on the v1 in production(where the endpoints are still working):

New TODO after testing:

ohnewein commented 5 years ago

@Piiit please test. Thanks!

Piiit commented 5 years ago

Tests

General issues

These issues are kept in another issue (see https://github.com/idm-suedtirol/bdp-commons/issues/78)

dc-parking-MeBo

Tested with https://github.com/idm-suedtirol/bdp-commons/blob/development/data-collectors/parking-MeBo/README.md

A type named occupacy exists, which is maybe wrong... needs to be checked.

There is a mismatch between the free and occupied data type, that must be checked. I think the migration script generates occupied records, but the data collector inserts free records. However, the webservices want to see both.

Result: We need to fix the following: See https://github.com/idm-suedtirol/bdp-commons/issues/77, and fix the "occupacy" type

dc-driwe, dc-nevicam, dc-alperia, dc-route220, dc-emobility-hydrogen

--> emobility-echarging & emobility-hydrogen

We have records pointing to the future: Compare created_on and timestamp... ??

select m.created_on, m.timestamp, double_value, t.cname, s.stationcode
from station s 
join measurement m on s.id = m.station_id
join type t on t.id = m.type_id
where stationtype ~* 'echarging'
and origin = 'ALPERIA'
and m.timestamp > now() - interval '100 days'
order by m.created_on desc
limit 10
created_on      timestamp       value   cname           stationcode
2019-02-19 08:55:08 2019-02-20 07:15:00 1   echarging-plug-status   ASM_00000194-2
2019-02-19 08:55:08 2019-02-20 07:15:00 1   echarging-plug-status   ASM_00000194-1
2019-02-19 08:55:08 2019-02-20 07:15:00 1   echarging-plug-status   ASM_00000192-2
2019-02-19 08:55:08 2019-02-20 07:15:00 1   echarging-plug-status   ASM_00000192-1

UDPATE: This is not an error, created_on shows when this (station,type) combination had its first measurement record created.

Tested with origin = ALPERIA, Nevicam, IIT (=hydrogen), DRIWE, route220

Info: emobility-hydrogen still points to a non-existing DTO inside application.properties app.station.DTO.class=it.bz.idm.bdp.dto.emobility.EchargingStationDto

Result: All have recent measurements (success); remove wrong DTO config (see above)

dc-carsharing-halapi

select m.created_on, m.timestamp, double_value, t.cname, s.stationcode, t.*, s.*
from station s 
join measurement m on s.id = m.station_id
join type t on t.id = m.type_id
where stationtype ~* 'car'
and m.timestamp > now() - interval '100 days'
order by m.timestamp desc
limit 1
created_on  timestamp   double_value    cname   stationcode
2019-02-20 08:40:24 2019-02-20 08:40:00 0.0000000000000000  availability    193011

Result: Record is old

dc-airquality-appabz

Login to the SFTP server and copy a new data file into uploads/.tests. Tested with 20192002_0930.dat...

select m.created_on, m.timestamp, double_value, t.cname, s.stationcode, t.*, s.*
from station s 
join measurement m on s.id = m.station_id
join type t on t.id = m.type_id
where stationtype ~* 'envir'
and origin = 'APPABZ'
and m.timestamp > now() - interval '100 days'
order by m.created_on desc
limit 1
created_on          timestamp           double_value         cname         stationcode
2018-06-27 12:00:49 2019-02-20 08:20:00 39.8700000000000000  NO2_mean_ppb  ML6

Result: OK, but Spring 5.0.10 and 5.0.12 used (bigger war file), remove one

dc-vehicletraffic-bluetooth

Inserted a record via...

curl -X POST \
  http://tomcat.testingmachine.eu/dc-vehicletraffic-bluetooth/json \
  -H 'Content-Type: application/json' \
  -d '[
    {
        "utcInMs": 1550657621000,
        "station_id": 23,
        "local_id": 3,
        "mac": "THIS IS A MAC ADDRESS",
        "gathered_on": "2019-20-02T12:13:14",
        "stationcode": "cfirmiano"
    }
]'
select m.created_on, m.timestamp, double_value, t.cname, s.stationcode, t.*, s.*
from station s 
join measurement m on s.id = m.station_id
join type t on t.id = m.type_id
where stationtype ~* 'bluet'
and m.timestamp > now() - interval '10 days'
order by m.created_on desc
limit 1
created_on          timestamp            string_value           cname               stationcode
2019-02-20 11:34:56 2020-08-02 14:13:14  THIS IS A MAC ADDRESS  vehicle detection   cfirmiano

Result: OK

meteorology

Result: This is a deprecated collector, nothing to test

dc-parking-tn

select m.created_on, m.timestamp, double_value, t.cname, s.stationcode
from station s 
join measurement m on s.id = m.station_id
join type t on t.id = m.type_id
where stationtype ~* 'park'
and m.timestamp > now() - interval '10 days'
order by m.timestamp desc
limit 1
created_on  timestamp   double_value    cname   stationcode
2019-02-20 01:05:04 2019-02-20 16:30:00 225.0000000000000000    parking-forecast-240    TRENTO:cteviabomportop6

A type named occupacy exists, which is maybe wrong... needs to be checked.

There is a mismatch between the free and occupied data type, that must be checked. I think the migration script generates occupied records, but the data collector inserts free records. However, the webservices want to see both.

Result: We need to fix the following: See idm-suedtirol/bdp-commons#77, and fix the "occupacy" type

Piiit commented 5 years ago

@bertolla Tested... most things are OK, but we have some issues to solve... see comment above. I put new todos into the first comment block.

Piiit commented 5 years ago

@bertolla @rcavaliere I think we are done with this, all remaining issue have either a separate user story or are low priority (minor issues) and can be fixed after the migration easier