overpass.py - _waitForReady - Hardcoded Status String Positions

[e-bock]

Hi there,

first of all, great library, it's been really useful to me!

Now my issue, after experiementing with the public endpoints I've setup a personal overpass endpoint to escape the wait times and not over utilize public goods. I've followed along the instructions on: https://wiki.openstreetmap.org/wiki/Overpass_API/Installation Installing Release 0.7.58

Pointing the library to my endpoint via: Overpass(endpoint='<MyEndpoint>')

I ended up with the following error (happy to share a more detailed trace if needed): Exception: [overpass] could not fetch or interpret status of the endpoint

After some analysis it seems I'm failing in the libraries _waitForReady function because my status response has more rows and therefore the statusString[] indexes are off.

Status Response Connected as: 3232235754 Current time: 2022-06-01T15:19:28Z Announced endpoint: none Rate limit: 0 Currently running queries (pid, space limit, time limit, start time):

Can the function be changed to work line independent and detect required infos smarter? I'm just starting out so I don't want to mess with the libraries code.

Cheers!

[huyphan2612]

I also faced the same issue in this morning. However, it's quite weird is that this issue does not happen when I ran the overpass API at the local environment. It only happened when I ran overpass API on my ubuntu server. I did investigate the source code and found the status response is changed in the format, so this function is not working as expected. But I don't know why the code is running normally at my local environment, but not in the server.

[franz-benjamin]

Thanks for raising this issue. I will investigate this in one of the next days! I was not yet able to reproduce the issue, but I will try again.

[huyphan2612]

Thanks for raising this issue. I will investigate this in one of the next days! I was not yet able to reproduce the issue, but I will try again.

I tried to change the statusString index and it's working now. Please review my code. It's the first time I edited on github, pardon me if I did something wrong.

if statusString[2] == 'Rate limit: 0': --> if statusString[3] == 'Rate limit: 0': if statusString[3].endswith('slots available now.'): --> if statusString[4].endswith('slots available now.'):

[ZaraGi]

Hello, may I propose a more generic approach? With this solution the problem will recur whenever the order of the elements within the statusString list is changed, instead turning the list into a dictionary and checking the keys would be more robust.

Something like this:

tmp = statusString.copy()
statusString = {}
for element in tmp:
    if element == '':
        continue
    if element.endswith('slots available now.'):
        element = f'{element[-20:-1]}: {element[:-21]}'
    splits = element.split(': ')
    if len(splits) == 1:
        statusString[splits[0]] = ''
    else:
        statusString[splits[0]] = splits[1]

if ('Rate limit' in statusString.keys()) and  (statusString['Rate limit'] == ' 0'):
    return True

[rosepearson]

I'm glad to see there is an issue for this already! I too have only been using the library a short while - it's great! Thanks to the collaborators for putting it together!

Same deal - library was working great, but now I'm getting intermittent errors depending if my overpass queries are run locally or run by GitHub actions on an Ubuntu server. Looking forward to seeing this fixed :)

[simonmarti1992]

Hello, Thanks for raising this issue. I face the same issue while launching a query locally

[rosepearson]

Yup - exactly the same for me. I do get it occasionally when run locally - but generally it is fine. I assume this is something to do with the statusString returned to osm-python-tools becoming more permissive and osm-python-tools not yet being flexible enough to deal with variable length statusStrings. The suggested solution in the associated PR looks good to me!

[rosepearson]

I've had a bit of a look through my GH actions install details and it looks like my tests began failing when the s3transfer library updated from version 0.5.2 to version 0.6.0. Anyway I have pinned it to version 0.5.2 in my YML file for setting up the remote test environment. I'll report back there if that fixes the issue as a work around until osmpythontools is updated.

[rosepearson]

I had a look at pinning a few different libraries that differed in version between when my tests last passed and began failing due to the status string bug, and seemed like they plausibly could affect the form/contents of the status string. I tried pinning boto3, botocore, lxml and s3transfer. Sadly none of them fixed the issue.

[mocnik-science]

Thanks for your good observations and ideas, code examples, and your patience in this matter. I apologize for not having reacted before, other then saying that I will deal with it in some days. The reason is that I am doing some more background checks before releasing a solution in order to ensure that I do not introduce new issues (like testing with a buch of endpoints in this case). I did some last checks and have now uploaded a new version.

@e-bock, @ZaraGi, @rosepearson, @simonmarti1992, @ptphuy: I would be glad if you could check whether the version hosted here on GitHub solves the issue you are experiencing. Please note that the changes have not yet been released, so you have to check with the GitHub version of OSMPythonTools.

In case you confirm that the issues have been resolved, I will release the new version. Stay tuned.

[mocnik-science]

As a small addendum @e-bock: I hope it's ok if I wrote my own code instead of using yours. As @ZaraGi said, parsing the status string should be more robust. And I tried to actually do that.

[simonmarti1992]

Thank you @mocnik-science : problem solved > the version hosted on this GitHub works for me

[rosepearson]

Thanks @mocnik-science - my tests have finally run and the version hosted on this GitHub also works for me. Thanks for your thorough fixes/enhancements!

[huyphan2612]

The version hosted on this GitHub also works for me. Thanks @mocnik-science for your new update!

[mocnik-science]

@simonmarti1992 @rosepearson @ptphuy: Thanks for your testing and your replies! I have made some further tests in the meantime, and everything seems to work. I have released it as a new version 0.3.5 now.

[e-bock]

Sorry, took a while to get to test the new version but it works perfectly fine! Thanks a lot @mocnik-science for the quick fix!!