As an Open Data Hub API consumer, I want to have all mobility APIs on the same domain with versioning

Piiit commented 5 years ago

Legend:

TODO

[x] Mobility: Create a mobility-webservice infrastructure clone to test a 1:1 copy of the production environment
[x] Mobility: Rename mobility-webservices NINJA to v2 and create redirects to the new OM, exception is just for the API calls itself, that is, OM/ninja/api/v2
[x] Mobility: Fix core.release pipeline to have ws-interface use always a stable old DTO library
[x] Mobility: Move old mobility web services into a subfolder called v1
[x] Mobility: Mark old web services as deprecated and give a link to the new v2
[x] Mobility: Add ROOT.war which shows all sub-services as JSON on OM

This is now a redirect to the general mobility swagger
[x] Mobility: Remove the INVALID warning in v1 and v2
[x] Mobility: Swagger text and links in v2 should look like in v1

TODOS FOR DEPLOYMENT ON PRODUCTION

[x] Create a swagger.opendatahub.bz.it endpoint
[x] Remove server dropdown from openapi3 config in ninja
[x] Make the openapi3 config server configurable for test/prod deployments
[x] ws-interface should not point to a wrong URL for the API V2 endpoint
[x] ws-interface remove swagger from each API in V1

Ignored, since we do not want to invest time into the V1 anymore
[x] Release the new production version of bpd-core (ws-interface, dc-interface, ...)
[x] Deploy all web services to prod-tomcat85-ws from each master branch
[x] Deploy the reader to prod-tomcat85-ws from its master branch (check new versions of dto/dal)
[x] httpd/conf.d/ws.conf --> Should we remove geodata.integreen-life.bz.it and /geoserver ???

Ignored, since we do not want to invest time into the V1 anymore
[x] Merge and release the "removeswagger" branch into development and then master
[x] Change all DNS entries in Route53 to the new addresses
Fix all applications that use the old V2 (which uses <host>/api/flat/... which is no longer available (it is <host>/flat/... now)
- [x] parking.bz.it
- [x] emobility web comp map + dashboard
- [x] generic map web component

Tourism part moved: See issue https://github.com/noi-techpark/odh-tourism/issues/152

bertolla commented 4 years ago

Both calls (v1,v2) work correctly if you just make API calls (http://api.opendatahub.bz.it/v1/environment/rest/get-station-details). The thing which causes problems is swagger (http://api.opendatahub.bz.it/v1/environment/swagger-ui.html) with it's strange redirecting. We probably need to fix this application wise. v2 example: http://api.opendatahub.bz.it/v2/flat/EChargingPlug/*?limit=-1&offset=0&shownull=false&distinct=true&where=sactive.eq.true&select=smetadata,pmetadata,mvalue,mvalidtime,sname In v2 swagger is not reachable because it would be earlier in the path and I think we also need to fix that application wise

RudiThoeni commented 4 years ago

we decided: api.opendatahub.bz.it mobility.api.opendatahub.bz.it tourism.api.opendatahub.bz.it

Piiit commented 4 years ago

@RudiThoeni @bertolla The mobility part has been done on testing. I moved the tourism part to a new issue, see link in first comment.

The test instance uses the production DB now, for a better performance comparison. The server configuration (most important httpd conf) can be found here https://github.com/noi-techpark/server-deployment/tree/master/prod-tomcat85-ws.

I built a test environment equal to the production one, with the following URLs:

ipchannels.integreen-life.testingmachine.eu
mobility.api.opendatahub.testingmachine.eu

So you can test redirects etc. Maybe with curl -I -L if you like... The redirect to the swagger from the root path of the above URLs does sometimes not work from Firefox, but always from curl or other browsers. I do not know why... on any firefox from any other machine it works perfectly too, but somehow my local one cached the redirect or something...

Please test also swagger stuff with old and new APIs. Is something missing?

Piiit commented 4 years ago

@stefanodavid Do you want to test this too? Maybe the new swagger stuff?

bertolla commented 4 years ago

@Piiit I tested v1 by using my old mocha test. Some calls did not work because of auth, but the call to parkingFrontEnd failed completely. See:

    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/bluetooth/rest/get-stations" (711ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/bluetooth/rest/get-station-details" (1822ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/bluetooth/rest/get-data-types" (210ms)
    ✓ Datatype request with parameter - bluetooth (341ms)
    ✓ Newest data request with parameter - bluetooth (611ms)
    ✓ Get random data of last 2 hours - bluetooth (587ms)
    ✓ Get random data of 2 hours on 30 November 2016 - bluetooth (494ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/carpooling/rest/get-stations" (613ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/carpooling/rest/get-station-details" (663ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/carpooling/rest/get-data-types" (186ms)
    ✓ Datatype request with parameter - carpooling (308ms)
    1) Newest data request with parameter - carpooling
    ✓ Get random data of last 2 hours - carpooling (335ms)
    ✓ Get random data of 2 hours on 30 November 2016 - carpooling (321ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/carsharing/rest/get-stations" (654ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/carsharing/rest/get-station-details" (409ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/carsharing/rest/get-data-types" (179ms)
    ✓ Datatype request with parameter - carsharing (319ms)
    ✓ Newest data request with parameter - carsharing (1119ms)
    ✓ Get random data of last 2 hours - carsharing (491ms)
    ✓ Get random data of 2 hours on 30 November 2016 - carsharing (494ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/roadweather/rest/get-stations" (647ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/roadweather/rest/get-station-details" (408ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/roadweather/rest/get-data-types" (176ms)
    ✓ Datatype request with parameter - roadweather (321ms)
    ✓ Newest data request with parameter - roadweather (527ms)
    ✓ Get random data of last 2 hours - roadweather (482ms)
    ✓ Get random data of 2 hours on 30 November 2016 - roadweather (580ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/emobility/rest/get-stations" (1184ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/emobility/rest/get-station-details" (1137ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/emobility/rest/get-data-types" (199ms)
    ✓ Datatype request with parameter - emobility (345ms)
    ✓ Newest data request with parameter - emobility (492ms)
    ✓ Get random data of last 2 hours - emobility (666ms)
    ✓ Get random data of 2 hours on 30 November 2016 - emobility (481ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/environment/rest/get-stations" (1153ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/environment/rest/get-station-details" (330ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/environment/rest/get-data-types" (409ms)
    ✓ Datatype request with parameter - environment (353ms)
    2) Newest data request with parameter - environment
    ✓ Get random data of last 2 hours - environment (576ms)
    ✓ Get random data of 2 hours on 30 November 2016 - environment (516ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/meteorology/rest/get-stations" (624ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/meteorology/rest/get-station-details" (767ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/meteorology/rest/get-data-types" (216ms)
    ✓ Datatype request with parameter - meteorology (318ms)
    ✓ Newest data request with parameter - meteorology (523ms)
    ✓ Get random data of last 2 hours - meteorology (534ms)
    ✓ Get random data of 2 hours on 30 November 2016 - meteorology (463ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/origin-destination/rest/get-stations" (715ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/origin-destination/rest/get-station-details" (256ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/origin-destination/rest/get-data-types" (171ms)
    ✓ Datatype request with parameter - origin-destination (318ms)
    ✓ Newest data request with parameter - origin-destination (492ms)
    ✓ Get random data of last 2 hours - origin-destination (536ms)
    ✓ Get random data of 2 hours on 30 November 2016 - origin-destination (492ms)
    3) Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/parkingFrontEnd/rest/get-stations"
    4) Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/parkingFrontEnd/rest/get-station-details"
    5) Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/parkingFrontEnd/rest/get-data-types"
    6) Datatype request with parameter - parkingFrontEnd
    7) Newest data request with parameter - parkingFrontEnd
    8) Get random data of last 2 hours - parkingFrontEnd
    9) Get random data of 2 hours on 30 November 2016 - parkingFrontEnd
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/street/rest/get-stations" (634ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/street/rest/get-station-details" (743ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/street/rest/get-data-types" (185ms)
    ✓ Datatype request with parameter - street (320ms)
    ✓ Newest data request with parameter - street (521ms)
    ✓ Get random data of last 2 hours - street (503ms)
    ✓ Get random data of 2 hours on 30 November 2016 - street (497ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/sensors/rest/get-stations" (601ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/sensors/rest/get-station-details" (1221ms)
    ✓ Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/sensors/rest/get-data-types" (224ms)
    ✓ Datatype request with parameter - sensors (319ms)
    10) Newest data request with parameter - sensors
    ✓ Get random data of last 2 hours - sensors (312ms)
    ✓ Get random data of 2 hours on 30 November 2016 - sensors (297ms)

  1) BDP WS v1:
       Newest data request with parameter - carpooling:
     TypeError: Cannot read property 'body' of undefined
      at /home/patrick/git/bdp/tests/webservices.js:50:34
      at _fulfilled (node_modules/q/q.js:854:54)
      at /home/patrick/git/bdp/tests/node_modules/q/q.js:883:30
      at Promise.promise.promiseDispatch (node_modules/q/q.js:816:13)
      at /home/patrick/git/bdp/tests/node_modules/q/q.js:570:49
      at runSingle (node_modules/q/q.js:137:13)
      at flush (node_modules/q/q.js:125:13)
      at processTicksAndRejections (internal/process/task_queues.js:75:11)

  2) BDP WS v1:
       Newest data request with parameter - environment:
     AssertionError: expected header content-type with value undefined to match application/json;charset=UTF-8

  3) BDP WS v1:
       Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/parkingFrontEnd/rest/get-stations":
     AssertionError: expected status code 404 to equal 200

  4) BDP WS v1:
       Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/parkingFrontEnd/rest/get-station-details":
     AssertionError: expected status code 404 to equal 200

  5) BDP WS v1:
       Service availability check : "https://mobility.api.opendatahub.testingmachine.eu/v1/parkingFrontEnd/rest/get-data-types":
     AssertionError: expected status code 404 to equal 200

  6) BDP WS v1:
       Datatype request with parameter - parkingFrontEnd:
     AssertionError: expected body to match JSON schema {
  "type": "array"
}.
-----
 error: Invalid type: string (expected array).
 data path: .
 schema path: /type.

  7) BDP WS v1:
       Newest data request with parameter - parkingFrontEnd:
     AssertionError: expected status code 404 to equal 200

  8) BDP WS v1:
       Get random data of last 2 hours - parkingFrontEnd:
     AssertionError: expected status code 404 to equal 200

  9) BDP WS v1:
       Get random data of 2 hours on 30 November 2016 - parkingFrontEnd:
     AssertionError: expected status code 404 to equal 200

  10) BDP WS v1:
       Newest data request with parameter - sensors:
     TypeError: Cannot read property 'body' of undefined

RudiThoeni commented 4 years ago

as i have tested it'S working nice work! but for me it's not easy to test because i don't know what endpoints needs authentication and which not... Example /v1/bikesharing/rest/get-stations is not returning data.... /v1/bikesharing/rest/bikes/get-stations returns data...

Piiit commented 4 years ago

@RudiThoeni Yes that is because bikes are open data and bikesharing itself is not AFAIK... Is that correct, @bertolla?

Rudi, if it returns 200 OK with an empty []. It is fine... if it returns another http code it is most probably a bug.

Piiit commented 4 years ago

@bertolla Yes that is clear, the old names do no longer exist when you use https://mobility.api.opendatahub.testingmachine.eu/v1/ as basepath... they are only reachable from https://ipchannels.integreen-life.opendatahub.testingmachine.eu/...

This is because we do want that people if they change to a new URL they should use proper names, and if they do not switch and keep the old ones they should still work. So we are backward compatible...

For example: These things should work for "parking"

For details see https://github.com/noi-techpark/server-deployment/blob/master/prod-tomcat85-ws/.ebextensions/httpd/conf.d/ws.conf

We have two sections here: 1) ServerName ipchannels.integreen-life.testingmachine.eu 2) ServerName mobility.api.opendatahub.testingmachine.eu

Another question that I have seen now. Can I remove the section of ServerName geodata.integreen-life.testingmachine.eu? Is the geoserver stuff still used on prod-tomcat-ws?

stefanodavid commented 4 years ago

@Piiit what am I supposed to test? I usually test only API calls when I put them into documentation...

Piiit commented 4 years ago

@stefanodavid I was thinking about the swagger headers + links, that is the text beneath the title, and in addition to signal this change to you to be sure that you do not miss anything

noi-techpark / bdp-core

As an Open Data Hub API consumer, I want to have all mobility APIs on the same domain with versioning #109