The Noun Project

[obulat]

This issue has been migrated from the CC Search Catalog repository

Author: dravadhis
Date: Fri Oct 23 2020
Labels: providers,🙅 status: discontinued

Provider API Endpoint / Documentation

http://api.thenounproject.com/

Provider description

The Noun Project API provides a collection of icons and photos.

Licenses Provided

CC BY 3.0 Public Domain Mark 1.0

Provider API Technical info

Rate Limits: 5000 requests/month Overall volume: 3 million (Rough Indication)

Checklist to complete before beginning development

No development should be done on a Provider API Script until the following info is gathered:

• [x] Verify there is a way to retrieve the entire relevant portion of the provider's collection in a systematic way via their API.
• [x] Verify the API provides license info (license type and version; license URL provides both, and is preferred)
• [x] Verify the API provides stable direct links to individual works.
• [x] Verify the API provides a stable landing page URL to individual works.
• [x] Note other info the API provides, such as thumbnails, dimensions, attribution info (required if non-CC0 licenses will be kept), title, description, other meta data, tags, etc.
• [ ] Attach example responses to API queries that have the relevant info.

General Recommendations for implementation

• The script should be in the src/cc_catalog_airflow/dags/provider_api_scripts/ directory.
• The script should have a test suite in the same directory.
• The script must use the ImageStore class (Import this from src/cc_catalog_airflow/dags/provider_api_scripts/common/storage/image.py).
• The script should use the DelayedRequester class (Import this from src/cc_catalog_airflow/dags/provider_api_scripts/common/requester.py).
• The script must not use anything from src/cc_catalog_airflow/dags/provider_api_scripts/modules/etlMods.py, since that module is deprecated.
• If the provider API has can be queried by 'upload date' or something similar, the script should take a --date parameter when run as a script, giving the date for which we should collect images. The form should be YYYY-MM-DD (so, the script can be run via python my_favorite_provider.py --date 2018-01-01).
• The script must provide a main function that takes the same parameters as from the CLI. In our example from above, we'd then have a main function my_favorite_provider.main(date). The main should do the same thing calling from the CLI would do.
• The script must conform to PEP8. Please use pycodestyle (available via pip install pycodestyle) to check for compliance.
• The script should use small, testable functions.
• The test suite for the script may break PEP8 rules regarding long lines where appropriate (e.g., long strings for testing).

Examples of other Provider API Scripts

For example Provider API Scripts and accompanying test suites, please see

• src/cc_catalog_airflow/dags/provider_api_scripts/flickr.py and
• src/cc_catalog_airflow/dags/provider_api_scripts/test_flickr.py, or
• src/cc_catalog_airflow/dags/provider_api_scripts/wikimedia_commons.py and
• src/cc_catalog_airflow/dags/provider_api_scripts/test_wikimedia_commons.py.

---

Original Comments:

Issue author dravadhis commented on Fri Oct 23 2020:

@mathemancer As mentioned in WordPress/openverse#1560 I will begin work on this. Thank You! source

Issue author dravadhis commented on Wed Oct 28 2020:

@mathemancer I am working on issue on issue WordPress/openverse#1559. The API requires OAuth 1.0 authorisation via an api_key and consumer_secret. To send an authorised request I need to use a library called requests-oauthlib (See here for reference). Should I add this library in the requirements file? Or is there any other way to go about this? source

mathemancer commented on Thu Oct 29 2020:

@dravadhis I think that should be fine. Just add it to both requirements files (make sure to freeze the version in prod requirements, but not dev). source

[stacimc]

The Noun Project API requires a search query and does not allow using wildcards (*) to paginate through all results. While that's the case, I'm not sure it'll be possible to add them as a provider in Openverse.

Another note from the API documentation:

Asset URLs are temporary and will expire within an hour after the call. Following that duration, a new call will be required to retrieve a new asset URL.

It looks like this refers to generated thumbnail urls, so we would not be able to use those. A sample icon within the API response looks like this:

{
      "attribution": "Cat by Raúl Inc from Noun Project",
      "collections": [],
      "creator": {
        "name": "Raúl Inc",
        "permalink": "/rlinze",
        "username": "rlinze"
      },
      "id": "49952",
      "license_description": "public-domain",
      "permalink": "/term/cat/49952",
      "tags": [
        "cat",
        "animal",
        "domesticated",
        "feline",
        "kitten",
        "kitty",
        "kitty-cat",
        "mammal",
        "pet"
      ],
      "term": "Cat",
      "thumbnail_url": "https://static.thenounproject.com/png/49952-200.png",
      "updated_at": "2019-04-22 19:22:17"
    },

https://thenounproject.com/term/cat/49952/ redirects to https://thenounproject.com/icon/cat-49952/