search

helldog136 / pystibmivb

Python library to query STIB/MIVB opendata
Creative Commons Attribution 4.0 International
3 stars 2 forks source link

get_passages(stop_name) returns only one stop #6

Closed danito closed 4 years ago

danito commented 4 years ago

Hi,

I've tried your example.py with service.get_passages("Saint-Denis") and it returns only data for stop_id: '5719' and it's missing info about stop_id 5152.

helldog136 commented 4 years ago

Hi, i'll take a look at it. Can you provide the full code of what you tested? It might be possible that, at the time of your call, the STIB API hasn't returned any waiting time for the stop_id '5152'. In this case it would be normal that you don't get any data for stop_id '5152'. The goal of this lib is to abstract the stop_id by just taking the stop_name.

danito commented 4 years ago

Hi, probably I didn't update your example.py after I updated pystibmivb. This time I updated to 1.2.0 and took the latest example.py file. Settings:

async def go(LOOP):
    stop_name = "Saint-Denis"
    lines_filter = [(54, "TRONE")]
    custom_session = aiohttp.ClientSession()
    APIClient = STIBAPIClient(LOOP, custom_session, CLIENT_ID, CLIENT_SECRET)
    service = STIBService(APIClient)
    print(await service.get_passages(stop_name))

(the rest I didn't modify) Result:

Error while parsing response from STIB. Raw response is: b'{\r\n  "points": [\r\n    {\r\n      "passingTimes": [\r\n        {\r\n          "destination": {\r\n            "fr": "DROGENBOS CHATEAU",\r\n            "nl": "DROGENBOS KASTEEL"\r\n          },\r\n          "expectedArrivalTime": "2020-06-02T22:39:00+02:00",\r\n          "lineId": "32"\r\n        },\r\n        {\r\n          "destination": {\r\n            "fr": "SPORTCENTRUM R.",\r\n            "nl": "SPORTCENTRUM R."\r\n          },\r\n          "expectedArrivalTime": "2020-06-02T22:31:00+02:00",\r\n          "lineId": "50"\r\n        },\r\n        {\r\n          "destination": {\r\n            "fr": "TRONE",\r\n            "nl": "TROON"\r\n          },\r\n          "expectedArrivalTime": "2020-06-02T22:30:00+02:00",\r\n          "lineId": "54"\r\n        },\r\n        {\r\n          "lineId": "82",\r\n          "message": {\r\n            "fr": "FIN DE SERVICE",\r\n            "nl": "EINDE DIENST"\r\n          }\r\n        },\r\n        {\r\n          "destination": {\r\n            "fr": "DIEWEG",\r\n            "nl": "DIEWEG"\r\n          },\r\n          "expectedArrivalTime": "2020-06-02T22:37:00+02:00",\r\n          "lineId": "97"\r\n        },\r\n        {\r\n          "destination": {\r\n            "fr": "DROGENBOS CHATEAU",\r\n            "nl": "DROGENBOS KASTEEL"\r\n          },\r\n          "expectedArrivalTime": "2020-06-02T22:55:00+02:00",\r\n          "lineId": "32"\r\n        },\r\n        {\r\n          "destination": {\r\n            "fr": "SPORTCENTRUM R.",\r\n            "nl": "SPORTCENTRUM R."\r\n          },\r\n          "expectedArrivalTime": "2020-06-02T22:47:00+02:00",\r\n          "lineId": "50"\r\n        },\r\n        {\r\n          "destination": {\r\n            "fr": "TRONE",\r\n            "nl": "TROON"\r\n          },\r\n          "expectedArrivalTime": "2020-06-02T22:50:00+02:00",\r\n          "lineId": "54"\r\n        },\r\n        {\r\n          "lineId": "82",\r\n          "message": {\r\n            "fr": "FIN DE SERVICE",\r\n            "nl": "EINDE DIENST"\r\n          }\r\n        },\r\n        {\r\n          "destination": {\r\n            "fr": "DIEWEG",\r\n            "nl": "DIEWEG"\r\n          },\r\n          "expectedArrivalTime": "2020-06-02T22:51:00+02:00",\r\n          "lineId": "97"\r\n        }\r\n      ],\r\n      "pointId": "5719"\r\n    }\r\n  ]\r\n}'
Traceback (most recent call last):
  File "example.py", line 36, in <module>
    LOOP.run_until_complete(go(LOOP))
  File "/usr/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete
    return future.result()
  File "example.py", line 20, in go
    print(await service.get_passages(stop_name))
  File "/home/dan/.local/lib/python3.6/site-packages/pystibmivb/service/STIBService.py", line 77, in get_passages
    raise ke
  File "/home/dan/.local/lib/python3.6/site-packages/pystibmivb/service/STIBService.py", line 68, in get_passages
    destination=json_passage["destination"][lang[LANG_STOP_NAME]],
KeyError: 'destination'
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f5d25267668>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f5d25014c78>, 4943579.630820976)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f5d252676a0>

When I use the example.py file without modifying anything it works fine:

[{'stop_id': '3755', 'line_id': '46', 'destination': 'WTC / GLIBERT', 'expected_arrival_time': '2020-06-02T22:43:00+02:00', 'line_color': '#DE3B21', 'line_text_color': '#FFFFFF', 'line_type': 'B', 'message': '', 'arriving_in': {'min': 8, 'sec': 53}}, {'stop_id': '3755', 'line_id': '46', 'destination': 'WTC / GLIBERT', 'expected_arrival_time': '2020-06-02T22:58:00+02:00', 'line_color': '#DE3B21', 'line_text_color': '#FFFFFF', 'line_type': 'B', 'message': '', 'arriving_in': {'min': 23, 'sec': 53}}]
{46}
{46}
[(46, 'GLIBERT')]
danito commented 4 years ago

My PR corrects the bug when one line is end of service with the default in nl. Now the result returns 


{'destination': {'fr': 'DIEWEG', 'nl': 'DIEWEG'}, 'expectedArrivalTime': '2020-06-02T23:35:00+02:00', 'lineId': '97'}
[{'stop_id': '5719', 'line_id': '32', 'destination': 'DROGENBOS CHATEAU', 'expected_arrival_time': '2020-06-02T23:40:00+02:00', 'line_color': '#F7E017', 'line_text_color': '#000000', 'line_type': 'T', 'message': '', 'arriving_in': {'min': 9, 'sec': 40}}, {'stop_id': '5719', 'line_id': '50', 'destination': 'SPORTCENTRUM R.', 'expected_arrival_time': '2020-06-02T23:45:00+02:00', 'line_color': '#B5BA05', 'line_text_color': '#000000', 'line_type': 'B', 'message': '', 'arriving_in': {'min': 14, 'sec': 40}}, {'stop_id': '5719', 'line_id': '54', 'destination': 'TRONE', 'expected_arrival_time': '2020-06-02T23:49:00+02:00', 'line_color': '#DE3B21', 'line_text_color': '#FFFFFF', 'line_type': 'B', 'message': 'LAATSTE VERTREK', 'arriving_in': {'min': 18, 'sec': 40}}, {'stop_id': '5719', 'line_id': '82', 'destination': '', 'expected_arrival_time': '2020-06-03T00:13:01', 'line_color': '#9EBFE3', 'line_text_color': '#000000', 'line_type': 'T', 'message': 'EINDE DIENST', 'arriving_in': {'min': 42, 'sec': 41}}, {'stop_id': '5719', 'line_id': '97', 'destination': 'DIEWEG', 'expected_arrival_time': '2020-06-02T23:35:00+02:00', 'line_color': '#991F36', 'line_text_color': '#FFFFFF', 'line_type': 'T', 'message': '', 'arriving_in': {'min': 4, 'sec': 40}}, {'stop_id': '5719', 'line_id': '32', 'destination': 'DROGENBOS CHATEAU', 'expected_arrival_time': '2020-06-02T23:58:00+02:00', 'line_color': '#F7E017', 'line_text_color': '#000000', 'line_type': 'T', 'message': '', 'arriving_in': {'min': 27, 'sec': 40}}, {'stop_id': '5719', 'line_id': '50', 'destination': 'SPORTCENTRUM R.', 'expected_arrival_time': '2020-06-03T00:05:00+02:00', 'line_color': '#B5BA05', 'line_text_color': '#000000', 'line_type': 'B', 'message': '', 'arriving_in': {'min': 34, 'sec': 40}}, {'stop_id': '5719', 'line_id': '54', 'destination': '', 'expected_arrival_time': '2020-06-03T00:13:01', 'line_color': '#DE3B21', 'line_text_color': '#FFFFFF', 'line_type': 'B', 'message': 'EINDE DIENST', 'arriving_in': {'min': 42, 'sec': 41}}, {'stop_id': '5719', 'line_id': '82', 'destination': '', 'expected_arrival_time': '2020-06-03T00:13:01', 'line_color': '#9EBFE3', 'line_text_color': '#000000', 'line_type': 'T', 'message': 'EINDE DIENST', 'arriving_in': {'min': 42, 'sec': 41}}, {'stop_id': '5719', 'line_id': '97', 'destination': 'DIEWEG', 'expected_arrival_time': '2020-06-02T23:51:00+02:00', 'line_color': '#991F36', 'line_text_color': '#FFFFFF', 'line_type': 'T', 'message': '', 'arriving_in': {'min': 20, 'sec': 40}}, {'stop_id': '5719', 'line_id': '32', 'destination': 'DROGENBOS CHATEAU', 'expected_arrival_time': '2020-06-02T23:40:00+02:00', 'line_color': '#F7E017', 'line_text_color': '#000000', 'line_type': 'T', 'message': '', 'arriving_in': {'min': 9, 'sec': 40}}, {'stop_id': '5719', 'line_id': '50', 'destination': 'SPORTCENTRUM R.', 'expected_arrival_time': '2020-06-02T23:45:00+02:00', 'line_color': '#B5BA05', 'line_text_color': '#000000', 'line_type': 'B', 'message': '', 'arriving_in': {'min': 14, 'sec': 40}}, {'stop_id': '5719', 'line_id': '54', 'destination': 'TRONE', 'expected_arrival_time': '2020-06-02T23:49:00+02:00', 'line_color': '#DE3B21', 'line_text_color': '#FFFFFF', 'line_type': 'B', 'message': 'LAATSTE VERTREK', 'arriving_in': {'min': 18, 'sec': 40}}, {'stop_id': '5719', 'line_id': '82', 'destination': '', 'expected_arrival_time': '2020-06-03T00:13:01', 'line_color': '#9EBFE3', 'line_text_color': '#000000', 'line_type': 'T', 'message': 'EINDE DIENST', 'arriving_in': {'min': 42, 'sec': 41}}, {'stop_id': '5719', 'line_id': '97', 'destination': 'DIEWEG', 'expected_arrival_time': '2020-06-02T23:35:00+02:00', 'line_color': '#991F36', 'line_text_color': '#FFFFFF', 'line_type': 'T', 'message': '', 'arriving_in': {'min': 4, 'sec': 40}}]
{32, 97, 82, 50, 212, 54}

That's what I use in example.py:

async def go(LOOP):
    stop_name = "Saint-Denis"
    custom_session = aiohttp.ClientSession()
    APIClient = STIBAPIClient(LOOP, custom_session, CLIENT_ID, CLIENT_SECRET)
    service = STIBService(APIClient)
    print(await service.get_passages(stop_name))

    shapefile_service = ShapefileService(APIClient)

    scherdemael = await shapefile_service.get_stop_infos(stop_name)
    print(scherdemael.get_lines())

    await custom_session.close()

As you can see, still no data for the second stop 5152 (and T97, B50 , B54 and T32 are still in service for both direction, only T82 is end of service).

danito commented 4 years ago

When I check ShapefileService.py and add L120: 

            print(self.stops_cache[stop_name])

Result is:

Saint-Denis({'lat': 50.81021150000001, 'lon': 4.31776625}): 
    ['5719H 32 1 DROGENBOS CHATEAU', 
     '5719H 82 1 DROGENBOS CHATEAU', 
     '5719H 97 1 DIEWEG', 
     '5719 50 1 LOT STATION', 
     '5152 54 1 FOREST (BERVOETS)', 
     '5719 212 1 CARREFOUR STALLE', 
     '5152G 32 2 DA VINCI', 
     '5152G 82 2 BERCHEM STATION', 
     '5152G 97 2 LOUISE', 
     '5152 50 2 GARE DU MIDI', 
     '5719 54 2 TRONE', 
     '5152 212 2 GARE CENTRALE']:
 ['32: T: #F7E017: #000000', '82: T: #9EBFE3: #000000', '97: T: #991F36: #FFFFFF', '50: B: #B5BA05: #000000', '54: B: #DE3B21: #FFFFFF', '212: B: #E87D0D: #E1337E']

Both stops are showing up... It's too late now to debug this.

danito commented 4 years ago

could'nt help myself. As default max_passages=15 and there are 12 different stops, maximum is reached before we get at stop 5152. When putting max_passages=115 I have all results for both stops (2x6 times).

helldog136 commented 4 years ago

I have contacted STIB guys to understand how to query multiple stop_ids in the same api call and i'm waiting for their answers. This could help fetching more results at once.

I don't really know what's best here... do i query STIB API for next passages and add at least one default (from time schedule) passage for each stop_id that returned no waiting time or do I kep it like this because all that matters is the effective next passages from the API...

The idea i had at first is that i don't really care to have the waiting times for all the possibles lines, i only care about the X next passages of a transportation medium (with an optional lines_filter) that will go through this stop...

danito commented 4 years ago

You can do submit multiple stop_ids to PassingTimeByPoints. It will ignore double stop_ids but there seem to be a limit, when passing all 12 ids, I got an error, but with the following it worked: https://opendata-api.stib-mivb.be/OperationMonitoring/4.0/PassingTimeByPoint/5719H, 5152G, 5719, 5152 Result:

{
  "points": [
    {
      "passingTimes": [
        {
          "destination": {
            "fr": "DROGENBOS CHATEAU",
            "nl": "DROGENBOS KASTEEL"
          },
          "expectedArrivalTime": "2020-06-03T23:17:00+02:00",
          "lineId": "32"
        },
        {
          "destination": {
            "fr": "SPORTCENTRUM R.",
            "nl": "SPORTCENTRUM R."
          },
          "expectedArrivalTime": "2020-06-03T23:09:00+02:00",
          "lineId": "50"
        },
        {
          "destination": {
            "fr": "TRONE",
            "nl": "TROON"
          },
          "expectedArrivalTime": "2020-06-03T23:10:00+02:00",
          "lineId": "54"
        },
        {
          "lineId": "82",
          "message": {
            "fr": "FIN DE SERVICE",
            "nl": "EINDE DIENST"
          }
        },
        {
          "destination": {
            "fr": "DIEWEG",
            "nl": "DIEWEG"
          },
          "expectedArrivalTime": "2020-06-03T23:06:00+02:00",
          "lineId": "97"
        },
        {
          "destination": {
            "fr": "DROGENBOS CHATEAU",
            "nl": "DROGENBOS KASTEEL"
          },
          "expectedArrivalTime": "2020-06-03T23:38:00+02:00",
          "lineId": "32"
        },
        {
          "destination": {
            "fr": "SPORTCENTRUM R.",
            "nl": "SPORTCENTRUM R."
          },
          "expectedArrivalTime": "2020-06-03T23:26:00+02:00",
          "lineId": "50"
        },
        {
          "destination": {
            "fr": "TRONE",
            "nl": "TROON"
          },
          "expectedArrivalTime": "2020-06-03T23:30:00+02:00",
          "lineId": "54"
        },
        {
          "lineId": "82",
          "message": {
            "fr": "FIN DE SERVICE",
            "nl": "EINDE DIENST"
          }
        },
        {
          "destination": {
            "fr": "DIEWEG",
            "nl": "DIEWEG"
          },
          "expectedArrivalTime": "2020-06-03T23:21:00+02:00",
          "lineId": "97"
        },
        {
          "destination": {
            "fr": "DROGENBOS CHATEAU",
            "nl": "DROGENBOS KASTEEL"
          },
          "expectedArrivalTime": "2020-06-03T23:17:00+02:00",
          "lineId": "32"
        },
        {
          "destination": {
            "fr": "SPORTCENTRUM R.",
            "nl": "SPORTCENTRUM R."
          },
          "expectedArrivalTime": "2020-06-03T23:09:00+02:00",
          "lineId": "50"
        },
        {
          "destination": {
            "fr": "TRONE",
            "nl": "TROON"
          },
          "expectedArrivalTime": "2020-06-03T23:10:00+02:00",
          "lineId": "54"
        },
        {
          "lineId": "82",
          "message": {
            "fr": "FIN DE SERVICE",
            "nl": "EINDE DIENST"
          }
        },
        {
          "destination": {
            "fr": "DIEWEG",
            "nl": "DIEWEG"
          },
          "expectedArrivalTime": "2020-06-03T23:06:00+02:00",
          "lineId": "97"
        },
        {
          "destination": {
            "fr": "DROGENBOS CHATEAU",
            "nl": "DROGENBOS KASTEEL"
          },
          "expectedArrivalTime": "2020-06-03T23:38:00+02:00",
          "lineId": "32"
        },
        {
          "destination": {
            "fr": "SPORTCENTRUM R.",
            "nl": "SPORTCENTRUM R."
          },
          "expectedArrivalTime": "2020-06-03T23:26:00+02:00",
          "lineId": "50"
        },
        {
          "destination": {
            "fr": "TRONE",
            "nl": "TROON"
          },
          "expectedArrivalTime": "2020-06-03T23:30:00+02:00",
          "lineId": "54"
        },
        {
          "lineId": "82",
          "message": {
            "fr": "FIN DE SERVICE",
            "nl": "EINDE DIENST"
          }
        },
        {
          "destination": {
            "fr": "DIEWEG",
            "nl": "DIEWEG"
          },
          "expectedArrivalTime": "2020-06-03T23:21:00+02:00",
          "lineId": "97"
        }
      ],
      "pointId": "5719"
    },
    {
      "passingTimes": [
        {
          "destination": {
            "fr": "DA VINCI",
            "nl": "DA VINCI"
          },
          "expectedArrivalTime": "2020-06-03T23:03:00+02:00",
          "lineId": "32"
        },
        {
          "destination": {
            "fr": "GARE DU MIDI",
            "nl": "ZUIDSTATION"
          },
          "expectedArrivalTime": "2020-06-03T23:22:00+02:00",
          "lineId": "50"
        },
        {
          "destination": {
            "fr": "FOREST (BERVOETS)",
            "nl": "VORST (BERVOETS)"
          },
          "lineId": "54",
          "message": {
            "fr": "LIGNE DÉVIÉE",
            "nl": "LIJN OMGELEID"
          }
        },
        {
          "lineId": "82",
          "message": {
            "fr": "FIN DE SERVICE",
            "nl": "EINDE DIENST"
          }
        },
        {
          "destination": {
            "fr": "LOUISE",
            "nl": "LOUIZA"
          },
          "expectedArrivalTime": "2020-06-03T23:05:00+02:00",
          "lineId": "97"
        },
        {
          "destination": {
            "fr": "DA VINCI",
            "nl": "DA VINCI"
          },
          "expectedArrivalTime": "2020-06-03T23:24:00+02:00",
          "lineId": "32"
        },
        {
          "destination": {
            "fr": "GARE DU MIDI",
            "nl": "ZUIDSTATION"
          },
          "expectedArrivalTime": "2020-06-03T23:42:00+02:00",
          "lineId": "50"
        },
        {
          "destination": {
            "fr": "FOREST (BERVOETS)",
            "nl": "VORST (BERVOETS)"
          },
          "lineId": "54",
          "message": {
            "fr": "LIGNE DÉVIÉE",
            "nl": "LIJN OMGELEID"
          }
        },
        {
          "lineId": "82",
          "message": {
            "fr": "FIN DE SERVICE",
            "nl": "EINDE DIENST"
          }
        },
        {
          "destination": {
            "fr": "LOUISE",
            "nl": "LOUIZA"
          },
          "expectedArrivalTime": "2020-06-03T23:16:00+02:00",
          "lineId": "97"
        },
        {
          "destination": {
            "fr": "DA VINCI",
            "nl": "DA VINCI"
          },
          "expectedArrivalTime": "2020-06-03T23:03:00+02:00",
          "lineId": "32"
        },
        {
          "destination": {
            "fr": "GARE DU MIDI",
            "nl": "ZUIDSTATION"
          },
          "expectedArrivalTime": "2020-06-03T23:22:00+02:00",
          "lineId": "50"
        },
        {
          "destination": {
            "fr": "FOREST (BERVOETS)",
            "nl": "VORST (BERVOETS)"
          },
          "lineId": "54",
          "message": {
            "fr": "LIGNE DÉVIÉE",
            "nl": "LIJN OMGELEID"
          }
        },
        {
          "lineId": "82",
          "message": {
            "fr": "FIN DE SERVICE",
            "nl": "EINDE DIENST"
          }
        },
        {
          "destination": {
            "fr": "LOUISE",
            "nl": "LOUIZA"
          },
          "expectedArrivalTime": "2020-06-03T23:05:00+02:00",
          "lineId": "97"
        },
        {
          "destination": {
            "fr": "DA VINCI",
            "nl": "DA VINCI"
          },
          "expectedArrivalTime": "2020-06-03T23:24:00+02:00",
          "lineId": "32"
        },
        {
          "destination": {
            "fr": "GARE DU MIDI",
            "nl": "ZUIDSTATION"
          },
          "expectedArrivalTime": "2020-06-03T23:42:00+02:00",
          "lineId": "50"
        },
        {
          "destination": {
            "fr": "FOREST (BERVOETS)",
            "nl": "VORST (BERVOETS)"
          },
          "lineId": "54",
          "message": {
            "fr": "LIGNE DÉVIÉE",
            "nl": "LIJN OMGELEID"
          }
        },
        {
          "lineId": "82",
          "message": {
            "fr": "FIN DE SERVICE",
            "nl": "EINDE DIENST"
          }
        },
        {
          "destination": {
            "fr": "LOUISE",
            "nl": "LOUIZA"
          },
          "expectedArrivalTime": "2020-06-03T23:16:00+02:00",
          "lineId": "97"
        }
      ],
      "pointId": "5152"
    }
  ]
}

So one should make sure that the list of stop_ids is unique. What's the aim for max_passages, why limit the result of passages, normally you have only 2 results per line?

helldog136 commented 4 years ago

The docuentation states that you can't put more than 10 IDs for the call.

I can't get the request with multiple Ids to work... How did you achieve that? can you give more information on what URL you contacted exactly? When I call /OperationMonitoring/4.0/PassingTimeByPoint/3755%2C3713 I get the following response: b'900906</ams:code>No matching resource found in the API for the given request</ams:message>Access failure for API: /OperationMonitoring/4.0, version: 4.0. Check the API documentation and add a proper REST resource path to the invocation URL</ams:description></ams:fault>'

To ensure uniqueness of the stop_ids you just need to put them in a set instead of a list.

my idea for max passages was to ensure not to have a ton of passages way in the future for the homeassistant sensor... This iis present since the first version of the lib and I think it mught be good to remove it as iit is not really the problem of the lib to limit the number of results. This s more a job for the user to decide which records to keep... It might go in a future version.

helldog136 commented 4 years ago

I increased the max_passages from 15 to 30. You can always set it to a higher number if needed.

I now call api with batches of 10 stop_ids... (1.3.0)

the max_passage logic will again evolve in the future because i'm not satisfied of it. I want this to specify the "x" next passages of the stop but it doesn't do that at the moment. Will also default it to None, None saying "no limit"