Use with multiple threads? (was: Record multiple HTTP requests in one function)

[NiCkWKT]

@vcr.use_cassette('responses/nearby_search.yaml')
def test_nearby_search():
    params = {
        "lat": 25.07663550005113,
        "lng": 121.37590495096043,
        "radius": 1000,
        "type": "restaurant",
        "language": "zh-TW",
        "include_static_map": False,
        "include_interactive_map": False
    }

    res = gmaps.nearby_search(**params)
    assert res["message"] == "20 result(s) yielded."

gmaps.nearby_search here contains nested HTTP request and nearby_search.yaml also records all the requests. But when I run pytest offline, the test failed.

Is there any other way to record multiple HTTP requests and replay them when testing offline? Many thanks!

[hartwork]

Hi @NiCkWKT,

Is there any other way to record multiple HTTP requests and replay them when testing offline?

Recording multiple HTTP request in a single cassette is expected to work out of the box. I verified locally using this code:

import vcr
import urllib.request

with vcr.use_cassette('fixtures/vcr_cassettes/synopsis.yaml'):
    for _ in range(2):
        response = urllib.request.urlopen('http://www.iana.org/domains/reserved').read()
        assert b'Example domains' in response

This will record two requests and their responses:

# grep request: fixtures/vcr_cassettes/synopsis.yaml | wc -l
2

But when I run pytest offline, the test failed.

What is the backtrace, Python version, VCR.py version and operating system in this context?

[NiCkWKT]

I am testing under a conda environment Python 3.9.16 vcrpy 4.3.1 OS: macOS 13.4 22F66 x86_64

There are 3 http requests when calling gmaps.nearby_search. For the first 2 requests, the info can be retrieved offline when I look at the log. But for the last one, here is the error log.

Relevant code:

  def get_place_details_n_photos_in_threads(self, thread_args: list, places: list, language: str):
      threads: List[threading.Thread] = []
      for i in range(len(thread_args)):
          place_id = thread_args[i][0]
          photo_reference = thread_args[i][1]
          t = threading.Thread(target=self.get_place_details_n_photos, args=(
              place_id, photo_reference, places, language, i))
          t.start()
          threads.append(t)
      for t in threads:
          t.join()

  def get_place_details_n_photos(self, place_id: str, photo_reference: str, result: List[dict], language: str, idx: int):
      # Google Place Details API by place_id for address, opening_hours, phone_number, google_map_url and website
      details = self.get_place_details(place_id, fields=[
                                       'formatted_address', 'international_phone_number', 'opening_hours', 'url', 'website', 'utc_offset'], language=language)
      phone_number = details.pop(
          'international_phone_number', '').replace(' ', '')
      details.update(
          {
              'address': details.pop('formatted_address', None),
              'google_map_url': details.pop('url', None),
              'phone_number': phone_number if phone_number else None
          }
      )
      result[idx].update(details)
      # Google Place Photos API by photo_reference for photo url
      photo_url = self.get_photo_url(photo_reference)
      result[idx].update({'photo': photo_url})

Is it because I do it in a multithreaded way? Is there any way to resolve this error? Thanks a lot

[hartwork]

Hi @NiCkWKT,

thanks for elaborating! My understanding is that:

• to VCR.py order of request matters, and
• use of multi-threading means that request order is by design not stable.

What I would personally do or consider is:

• separate threading from the actual work and test get_place_details_n_photos in isolation or
• make get_place_details_n_photos_in_threads work sequentially rather than with threads when under test
• mock away the network access when testing for the threading aspects of get_place_details_n_photos_in_threads e.g. using unittest.mock.patch

What do you think?

[NiCkWKT]

So VCR.py will playback HTTP requests sequentially. right? Probably I would go for removing threading under test Thanks for your guidance!!!