Gets to 96%, hits rate limit, never recovers

[nrgetik]

I've tried this with a few different locales, and it seems not to be working:

➜ snapmap-archiver -o . -l='decimal.lat,decimal.long' -r 10460 --write-json -t 7d
2024-05-20 10:30:45.564 | INFO     | snapmap_archiver.SnapmapArchiver:__init__:52 - Skipping Snaps older than [1715610645].
Location: (decimal.lat,decimal.long) |██████████████████████████████████████▎ | ▇▇▅ 96% [10000/10460] in 9s (~0s, 1091.9/s) 2024-05-20 10:30:55.006 | WARNING  | snapmap_archiver.SnapmapArchiver:_coordinate_query_failure:223 - You have been rate limited. Sleeping for [60] seconds...
Location: (decimal.lat,decimal.long) |██████████████████████████████████████▎ | ▄▂▂ 96% [10000/10460] in 1:24 (~4s, 119.9/s) 2024-05-20 10:32:09.328 | WARNING  | snapmap_archiver.SnapmapArchiver:_coordinate_query_failure:223 - You have been rate limited. Sleeping for [60] seconds...
Location: (decimal.lat,decimal.long) |██████████████████████████████████████▎ | ▂▄▆ 96% [10000/10460] in 1:46 (~5s, 94.1/s)

The average rate slowly approaches 0.0/s over the course of many more retries and eventually just hangs. Perhaps a bug was introduced in the refactor?

[king-millez]

Thanks for raising this. It might just need an exponential backoff. I'll investigate later today.

[king-millez]

Hi @nrgetik, I've had a go at this myself and it's working fine. Would you be ok with reaching out to me privately on Twitter (https://twitter.com/king_millez) and sending me the location(s) you're trying to download from? I can debug further from there...

[king-millez]

I've also had others reach out who are using the tool in densely populated areas (e.g., London) with no issues. This may have been a one-off?

[nrgetik]

@king-millez Unfortunately I don't have X Premium, so I can't DM you over there.

I tried this for a different, more densely populated location and saw the same behavior:

29.660662605458718,-82.31991878172093

I use NextDNS, but when I tail the logs while running the archiver, I see ms.sc-jpl.com allowed as normal.

Here's the traceback when I interrupt:

➜ snapmap-archiver -o . -l='lat, long' -r 10460 --write-json -t 7d
2024-05-28 08:31:09.507 | INFO     | snapmap_archiver.SnapmapArchiver:__init__:52 - Skipping Snaps older than [1716294669].
Location: (lat, long) |██████████████████████████████████████▎⚠︎| (!) 96% [10000/10460] in 3:08.9 (52.98/s)
Traceback (most recent call last):
  File "/Users/user/.pyenv/versions/3.12.2/bin/snapmap-archiver", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/snapmap_archiver/__main__.py", line 141, in main
    sm_archiver.query_coords(
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/snapmap_archiver/SnapmapArchiver.py", line 164, in query_coords
    api_response = requests.post(
                   ^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/urllib3/connection.py", line 653, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/ssl.py", line 1042, in _create
    self.do_handshake()
  File "/Users/user/.pyenv/versions/3.12.2/lib/python3.12/ssl.py", line 1320, in do_handshake
    self._sslobj.do_handshake()
KeyboardInterrupt

Perhaps I should try a different version of Python or requests ?

[nrgetik]

Got around to testing this some more today. Across platforms, versions, and different DNS configurations, it seems as though the value for the radius is the culprit. For example, 10000 works whereas 10460 does not.

[spectator8]

Hey guys, same for me, important is that is is not failing for each time when I use it. I downgraded app to version 2.2.0, but getting same troubles. Thanks for fix in advance.

[nrgetik]

Hey guys, same for me, important is that is is not failing for each time when I use it. I downgraded app to version 2.2.0, but getting same troubles. Thanks for fix in advance.

Yes, it's become intermittent for me as well now, regardless of radius value... there goes that theory

[king-millez]

I've noticed I put the wrong default radius value in the argparse config. I'm still yet to run into this issue myself. I'll just keep throwing the tool at densely populated areas until I hit something I guess 🤷‍♂️

I'll add some debug logs in the next release. If more people come across this issue, please add the --debug flag and post (or send me) the log outputs so we can get to the bottom of this :)

[king-millez]

See https://github.com/king-millez/snapmap-archiver/pull/22

[king-millez]

@nrgetik @spectator8 Alright everyone. Please upgrade with pip install snapmap-archiver==2.4.0 and use the --debug flag. While working on this update for ~2 hours I didn't encounter this issue once, so I'm interested in what is different about our setups.

[spectator8]

Many thanks mate @king-millez! I will make upgrade and start using debug flag. I will send you details in case of issues.

[nrgetik]

My most recent few runs with the new version in debug mode have all worked just fine. I'll keep trying for a failure. Thanks for the effort @king-millez

[spectator8]

Hello @king-millez , I was able to reproduce it. Params: -l="48.80632033548956, 19.637552330539375" --write-json -r 10000 -t 2d --debug

Here is full log: log.txt

Thanks!

[king-millez]

Ah interesting! This looks like a bug where snapmap-archiver always assumes snaps will be returned in a query. Debug mode ftw. I'll fix the root cause shortly :)

[king-millez]

Problem was this:

>>> [] is None
False
>>> not []
True

[king-millez]

This should now be fixed in version 2.4.1. Thanks for raising @nrgetik and thanks for the detective work @spectator8!