search

WordPress / openverse-api

The Openverse API allows programmatic access to search for CC-licensed and public domain digital media.
https://api.openverse.engineering/v1
MIT License
76 stars 50 forks source link

Freesound results are timing out #1041

Closed AetherUnbound closed 1 year ago

AetherUnbound commented 1 year ago

Description

In attempting to merge WordPress/openverse-api#1039, I observed that the tests were failing. The failing test showed:

media_path = 'audio', q = 'water running'

    def search_quotes_exact(media_path, q):
        """Only returns exact matches for the given query"""
        unquoted_response = requests.get(f"{API_URL}/v1/{media_path}?q={q}", verify=False)
        assert unquoted_response.status_code == 200
        unquoted_result_count = unquoted_response.json()["result_count"]
>       assert unquoted_result_count > 0
E       AssertionError

test/media_integration.py:53: AssertionError

I suspected that this might be an issue with dead link filtering, since these should be returning different results. When running the query for just "water" with logging level at DEBUG, I got the following logs:

Logs from a local query ``` openverse-api-web-1 | [2022-12-13 00:17:21,517 - root - 400][INFO] [2ffa389d464d43a29b8ceb1610f669fd] query={"query": {"bool": {"must": [{"bool": {"must": [{"bool": {"filter": [{"bool": {"must_not": [{"term": {"mature": true}}]}}, {"bool": {"must_not": [{"terms": {"provider": []}}]}}], "must": [{"simple_query_string": {"query": "water", "fields": ["tags.name", "title", "description"], "default_operator": "AND"}}]}}], "should": [{"simple_query_string": {"fields": ["title"], "query": "water", "boost": 10000}}]}}], "should": [{"rank_feature": {"field": "standardized_popularity", "boost": 10000}}]}}, "from": 0, "size": 40, "highlight": {"fields": {"tags.name": {}, "title": {}, "description": {}}, "order": "score"}}, es_took_ms=8 openverse-api-web-1 | [2022-12-13 00:17:21,518 - catalog.api.utils.validate_images.validate_images - 45][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] starting validation openverse-api-web-1 | [2022-12-13 00:17:21,519 - catalog.api.utils.validate_images.validate_images - 51][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] len(cached_statuses)=40 openverse-api-web-1 | [2022-12-13 00:17:21,519 - catalog.api.utils.validate_images.validate_images - 58][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] len(to_verify)=30 openverse-api-web-1 | [2022-12-13 00:17:21,539 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,539 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,539 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,540 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,540 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,540 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,540 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,540 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,540 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,540 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,542 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,543 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,543 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,544 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,544 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,544 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,544 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,544 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,544 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,544 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,544 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,545 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,545 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,545 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,545 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,545 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,545 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,546 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,546 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:21,546 - urllib3.connectionpool - 1003][DEBUG] [none] Starting new HTTPS connection (1): cdn.freesound.org:443 openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,840 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images._validation_failure - 150][WARNING] [2ffa389d464d43a29b8ceb1610f669fd] Failed to validate image! Reason: HTTPSConnectionPool(host='cdn.freesound.org', port=443): Read timed out. (read timeout=2) openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/508/508653_11085504-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/534/534225_11830391-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/534/534232_11830391-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/534/534228_11830391-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/540/540953_12022820-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/515/515776_11335579-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/515/515763_11335579-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/516/516032_7005313-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/540/540319_12044861-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,841 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/509/509493_11130756-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/520/520476_8092714-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/520/520696_5596396-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/521/521762_11129308-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/516/516846_11222545-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/509/509641_11129491-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/520/520913_5039423-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/516/516556_11222545-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/512/512862_5330055-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/513/513684_9492269-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/532/532340_11818895-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/512/512863_5330055-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/520/520692_5596396-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/512/512861_5330055-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,842 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/513/513493_5330055-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/529/529584_9747480-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/511/511261_5844915-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/520/520691_5596396-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/508/508683_10444078-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/513/513494_5330055-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 88][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] no response from provider key=valid:https://cdn.freesound.org/previews/516/516948_11375888-hq.mp3 openverse-api-web-1 | [2022-12-13 00:17:23,843 - catalog.api.utils.validate_images.validate_images - 93][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] caching status=-1 expiry=1800 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=230521bf-e1fe-4167-bd61-447f795178f6 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=8fd0798c-1a52-493f-b623-cf011b3827b7 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=7b15d685-b8be-453b-93ea-32b8f26054dc status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=fc3006b0-f9a6-42b3-a0ad-ece38ad2c232 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=841d66cb-9fb8-40e6-ab47-9c2209bb713d status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=95e45b7f-aa10-4909-b732-d394bd3cd69f status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=de82ce5b-5633-4452-8a95-9ae8cc2bf454 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=d4d8bad5-133f-4839-b351-85fbbedc9d9c status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=ae660e8a-1943-4453-985c-0c88c028f28f status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=a787d515-7d56-4f78-bc67-eb33c5a0968a status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=d4a3caff-2435-4f8a-83e0-fabeaaaaec94 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=502fa4c3-6679-4097-8510-07d20b11b9cd status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,844 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=3916a1ed-33c2-42a5-b227-9b96feb168c2 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=e4ef266a-44a8-4124-b045-2c23c233fbea status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=119fd02d-506b-49ac-b7c2-93009c4f92e8 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=08d052de-fd23-451f-9422-22f0588e64c2 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=c741bb8a-e990-455f-86fd-43e90adac2c5 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=3c86ec8f-04cc-4dad-bf1a-e8ea9a80b40c status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=2235d8b2-da16-4efa-ba1d-45520bb69fa9 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=3c340e2b-e3b1-4dba-859a-bd297bb72025 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=1dd1eee7-b27c-4bdf-938c-d1c1f0dea568 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=eb84972b-399c-42a9-bd7e-b68cff6678f1 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=b1e09098-ac5c-4449-a193-a337cb13bb5c status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=cb13915a-aa30-4f7d-8d70-7e46c1eaeb0e status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=e9ffef6f-91b3-4ca4-83e1-fe3f83c7db70 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=3db40fd2-eb1e-477f-b052-7154d23a15b9 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=ac1e2bc7-3caf-4168-9865-a324191e9f33 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=4895d099-5024-42f2-8b82-a885df80aedb status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=4c9f4e00-093b-4251-bd78-c9de4da2bd4c status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=5f6cc835-877a-4e43-9bc2-dc8048ce4683 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=ed422d4b-064f-4a3a-b936-38dcddbe4ce9 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=29665da0-4208-49bc-a655-b13a9e5e5256 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=485d6c5b-e048-47fc-a4c4-68e14fd4d68e status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=5b0183f8-171d-4a7d-9bab-afb037c80b26 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=e272e7ae-b9a2-432b-ab97-683546b1f69f status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,845 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=4fc3393d-86bf-4961-9ce3-faf3a15a67bb status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,846 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=0344350d-580f-40d7-b7e1-79f079fce2ab status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,846 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=64385997-4bcf-4bde-a576-bb2f95a2193c status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,846 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=fff6ec42-f174-42a5-a8c6-a2e6cf365d79 status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,846 - catalog.api.utils.validate_images.validate_images - 120][INFO] [2ffa389d464d43a29b8ceb1610f669fd] Deleting broken image from results id=7733b4b6-f611-4e33-931f-27b08ef033be status=-1 openverse-api-web-1 | [2022-12-13 00:17:23,846 - catalog.api.utils.validate_images.validate_images - 140][DEBUG] [2ffa389d464d43a29b8ceb1610f669fd] end validation end_time=1670890643.8466809 start_time=1670890641.5185585 delta=2.32812237739563 openverse-api-web-1 | [2022-12-13 00:17:23,851 - log_request_id.middleware - 47][INFO] [2ffa389d464d43a29b8ceb1610f669fd] method=GET path=/v1/audio/ status=200 openverse-api-web-1 | [13/Dec/2022 00:17:23] "GET /v1/audio/?q=water HTTP/1.1" 200 11176 ```

Sure enough, it looks like all of our local Freesound results are being filtered. If you go to one of the links (e.g. https://freesound.org/people/dryade07/sounds/520255/), it takes a while to load initially (certainly longer than 2 seconds) but does load.

I've marked this as critical since our tests are failing as a result of this. We may want to provide a workaround for tests before we try and address this at the API level.

Reproduction

  1. just up
  2. Visit http://localhost:50280/v1/audio/?source=freesound
  3. Observe the lack of results
  4. Visit http://localhost:50280/v1/audio/stats/
  5. Observe that there should be 828 Freesound results
sarayourfriend commented 1 year ago

If we merged WordPress/openverse-api#1027 we could use pook to mock all the requests. This kind of solution is recorded in WordPress/openverse#684.

Just to fix the specific failing test, we can skip dead link filtering for the exact match tests (probably the best "stop-gap").

sarayourfriend commented 1 year ago

Also, just to clarify because the issue description might imply otherwise, we check the url, not foreign_landing_url for "liveness validation": https://github.com/WordPress/openverse-api/blob/4edae23b8e12b8ceb19c75652b215621128fb4fa/api/catalog/api/controllers/search_controller.py#L173

I'm a little confused why production is working given we time-out the requests at 2 seconds, but the AWS datacenter the production Django is running in might be just a fast enough to get some of them back to produce a result page (assuming all of freesound results don't have liveness cached in Redis).