Add support for "natively installed" cloud credentials

Add support for using Application Default Credentials (GCP) and native IAM role (AWS) in lieu of requiring cloud storage credentials to be explicitly configured.

Testing (Local/Regression)

Revoke any local permissions the AWS, Google Cloud SDK, MinIO might have.
[Optional] Set up a standalone version of MinIO (if no server available).

Requirements

AWS
- Credentials that have access to the voxel51-test bucket.
- .ini file using those credentials with similar syntax to the following:
```
[default]
aws_access_key_id = ...
aws_secret_access_key = ...
aws_session_token = ...  ; if applicable
region = ...
```

GCP

Credentials that have access to the voxel51-test bucket.

JSON file with similar syntax to the following:

{
"type": "service_account",
    "project_id": "<project-id>",
    "private_key_id": "<private-key-id>",
    "private_key": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n",
    "client_email": "<account-name>@<project-id>.iam.gserviceaccount.com",
    "client_id": "<client-id>",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/..."
}

MinIO
- Credentials to an instance of MinIO
- A bucket in that instance, called voxel51-test, that has at least one file.
- .ini file using those credentials with similar syntax to the following:
```
[default]
access_key = ...
secret_access_key = ...
endpoint_url = ...
alias = ...  ; optional
region = ...  ; if applicable    
```

Run the following script:

# test.py

import os

import eta.core.storage as ecs

AWS_ACCESS_KEY_ID = ""
AWS_SECRET_ACCESS_KEY = ""
AWS_DEFAULT_REGION = ""
MINIO_ACCESS_KEY = ""
MINIO_SECRET_ACCESS_KEY = ""
MINIO_ENDPOINT_URL = ""

LOCAL_CREDS_PATH_AWS = ""  # /some/path/to/aws.ini
LOCAL_CREDS_PATH_GCP = ""  # /some/path/to/gcp.json
LOCAL_CREDS_PATH_MINIO = "" # /some/path/to/minio.ini

# Ensure any credential enivornment variables are unset
os.environ.pop("AWS_ACCESS_KEY_ID", None)
os.environ.pop("AWS_CONFIG_FILE", None)
os.environ.pop("AWS_DEFAULT_REGION", None)
os.environ.pop("AWS_SECRET_ACCESS_KEY", None)
os.environ.pop("AWS_SHARED_CREDENTIALS_FILE", None)
os.environ.pop("AWS_PROFILE", None)
os.environ.pop("GOOGLE_APPLICATION_CREDENTIALS", None)
os.environ.pop("MINIO_ACCESS_KEY", None)
os.environ.pop("MINIO_ALIAS", None)
os.environ.pop("MINIO_CONFIG_FILE", None)
os.environ.pop("MINIO_ENDPOINT_URL", None)
os.environ.pop("MINIO_REGION", None)
os.environ.pop("MINIO_SECRET_ACCESS_KEY", None)
os.environ.pop("MINIO_SHARED_CREDENTIALS_FILE", None)

# Ensure any active credentials are deactivated
if ecs.S3StorageClient.has_active_credentials():
    ecs.S3StorageClient.deactivate_credentials()

if ecs.GoogleCloudStorageClient.has_active_credentials():
    ecs.GoogleCloudStorageClient.deactivate_credentials()

if ecs.MinIOStorageClient.has_active_credentials():
    ecs.MinIOStorageClient.deactivate_credentials()

def get_aws_clients():
    clients = {}

    # === none ===
    try:
        ecs.S3StorageClient()
    except ecs.AWSCredentialsError:
        pass
    else:
        raise Exception(
            "Found S3 credentials. Please check the all defaults are disabled or removed."
        )

    # === explicit ===
    credentials, _ = ecs.S3StorageClient.load_credentials(
        credentials_path=LOCAL_CREDS_PATH_AWS
    )
    clients["explicit"] = ecs.S3StorageClient(credentials=credentials)
    assert isinstance(clients["explicit"], ecs.S3StorageClient)

    # === eta ===
    ecs.S3StorageClient.activate_credentials(LOCAL_CREDS_PATH_AWS)
    clients["eta"] = ecs.S3StorageClient()
    assert isinstance(clients["eta"], ecs.S3StorageClient)
    ecs.S3StorageClient.deactivate_credentials()

    # === boto3: individual env variables ===
    # Setting the following environment variables directly:
    # - AWS_ACCESS_KEY_ID
    # - AWS_SECRET_ACCESS_KEY
    # - AWS_DEFAULT_REGION
    os.environ["AWS_ACCESS_KEY_ID"] = AWS_ACCESS_KEY_ID
    os.environ["AWS_SECRET_ACCESS_KEY"] = AWS_SECRET_ACCESS_KEY
    os.environ["AWS_DEFAULT_REGION"] = AWS_DEFAULT_REGION

    clients["boto3:env"] = ecs.S3StorageClient()
    assert isinstance(clients["boto3:env"], ecs.S3StorageClient)

    del os.environ["AWS_ACCESS_KEY_ID"]
    del os.environ["AWS_SECRET_ACCESS_KEY"]
    del os.environ["AWS_DEFAULT_REGION"]

    # === boto3: shared credentials file ===
    # Setting the `AWS_SHARED_CREDENTIALS_FILE` environment variable to
    # point to a valid credentials `.ini` file
    os.environ["AWS_SHARED_CREDENTIALS_FILE"] = LOCAL_CREDS_PATH_AWS

    clients["boto3:creds-file"] = ecs.S3StorageClient()
    assert isinstance(clients["boto3:creds-file"], ecs.S3StorageClient)

    del os.environ["AWS_SHARED_CREDENTIALS_FILE"]

    # === boto3: aws config file ===
    # Setting the `AWS_CONFIG_FILE` environment variable to point to a
    # valid credentials `.ini` file

    os.environ["AWS_CONFIG_FILE"] = LOCAL_CREDS_PATH_AWS

    clients["boto3:config-file"] = ecs.S3StorageClient()
    assert isinstance(clients["boto3:config-file"], ecs.S3StorageClient)

    del os.environ["AWS_CONFIG_FILE"]
    return clients

def get_gcp_clients():
    clients = {}

    # === none ===
    try:
        ecs.GoogleCloudStorageClient()
    except ecs.GoogleCredentialsError:
        pass
    else:
        raise Exception(
            "Found GCP credentials. Please check the all defaults are disabled or removed."
        )

    # === explicit ===
    credentials, _ = ecs.GoogleCloudStorageClient.load_credentials(
        credentials_path=LOCAL_CREDS_PATH_GCP
    )
    clients["explicit"] = ecs.GoogleCloudStorageClient(credentials=credentials)
    assert isinstance(clients["explicit"], ecs.GoogleCloudStorageClient)

    # === eta ===
    ecs.GoogleCloudStorageClient.activate_credentials(LOCAL_CREDS_PATH_GCP)

    clients["eta"] = ecs.GoogleCloudStorageClient()
    assert isinstance(clients["eta"], ecs.GoogleCloudStorageClient)

    ecs.GoogleCloudStorageClient.deactivate_credentials()

    # === Application Default Credentials ===
    # https://google-auth.readthedocs.io/en/master/reference/google.auth.html

    os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = LOCAL_CREDS_PATH_GCP

    clients["ADC:env"] = ecs.GoogleCloudStorageClient()
    assert isinstance(clients["ADC:env"], ecs.GoogleCloudStorageClient)

    del os.environ["GOOGLE_APPLICATION_CREDENTIALS"]
    return clients

def get_minio_clients():
    clients = {}

    # === none ===
    try:
        ecs.MinIOStorageClient()
    except ecs.MinIOCredentialsError:
        pass
    else:
        raise Exception(
            "Found MinIO credentials. Please check the all defaults are disabled or removed."
        )

    # === explicit ===
    credentials, _ = ecs.MinIOStorageClient.load_credentials(
        credentials_path=LOCAL_CREDS_PATH_MINIO
    )
    clients["explicit"] = ecs.MinIOStorageClient(credentials=credentials)
    assert isinstance(clients["explicit"], ecs.MinIOStorageClient)

    # === eta ===
    ecs.MinIOStorageClient.activate_credentials(LOCAL_CREDS_PATH_MINIO)
    clients["eta"] = ecs.MinIOStorageClient()
    assert isinstance(clients["eta"], ecs.MinIOStorageClient)
    ecs.MinIOStorageClient.deactivate_credentials()

    # === individual env variables ===
    # Setting the following environment variables directly:
    # - MINIO_ACCESS_KEY
    # - MINIO_SECRET_ACCESS_KEY
    # - MINIO_ENDPOINT_URL
    os.environ["MINIO_ACCESS_KEY"] = MINIO_ACCESS_KEY
    os.environ["MINIO_SECRET_ACCESS_KEY"] = MINIO_SECRET_ACCESS_KEY
    os.environ["MINIO_ENDPOINT_URL"] = MINIO_ENDPOINT_URL

    clients["env"] = ecs.MinIOStorageClient()
    assert isinstance(clients["env"], ecs.MinIOStorageClient)

    del os.environ["MINIO_ACCESS_KEY"]
    del os.environ["MINIO_SECRET_ACCESS_KEY"]
    del os.environ["MINIO_ENDPOINT_URL"]

    # === shared credentials file ===
    # Setting the `MINIO_SHARED_CREDENTIALS_FILE` environment variable to
    # point to a valid credentials `.ini` file
    os.environ["MINIO_SHARED_CREDENTIALS_FILE"] = LOCAL_CREDS_PATH_MINIO

    clients["creds-file"] = ecs.MinIOStorageClient()
    assert isinstance(clients["creds-file"], ecs.MinIOStorageClient)

    del os.environ["MINIO_SHARED_CREDENTIALS_FILE"]

    # === config file ===
    # Setting the `MINIO_CONFIG_FILE` environment variable to point to a
    # valid credentials `.ini` file

    os.environ["MINIO_CONFIG_FILE"] = LOCAL_CREDS_PATH_MINIO

    clients["config-file"] = ecs.MinIOStorageClient()
    assert isinstance(clients["config-file"], ecs.MinIOStorageClient)

    del os.environ["MINIO_CONFIG_FILE"]

    return clients

providers = {
    "AWS": {
        "base_path": "s3://voxel51-test",
        "clients": get_aws_clients(),
    },
    "GCP": {
        "base_path": "gs://voxel51-test",
        "clients": get_gcp_clients(),
    },
    "MinIO": {
        "base_path": f"{MINIO_ENDPOINT_URL}/voxel51-test",
        "clients": get_minio_clients(),
    },
}

for provider_name, provider in providers.items():
    for client_key, client in provider["clients"].items():
        try:
            # ensure at one method call works and data is returned
            assert len(client.list_files_in_folder(provider["base_path"])) > 0
        except Exception as e:
            raise AssertionError(provider_name, client_key, e)

Testing (Cloud VM)

Following are the steps I performed for each cloud provider to test the code changes:

AWS

Create an EC2 instance (or use "IAM Test" ). Make sure it has an attached IAM role account that can access S3.
ssh onto the instance and ensure python, pip, git are installed.

Run the following in the terminal:

git clone https://github.com/voxel51/eta.git`
cd eta
git checkout develop
pip install ".[storage]"

Run the following script in python. It should print an error and exit.

# aws_test.py

import eta.core.storage as ecs

try:
    client = ecs.S3StorageClient()
    client.list_files_in_folder("s3://voxel51-test")
except ecs.AWSCredentialsError as e:
    print(e)
    exit()

Run the following in the terminal:

git checkout default-cloud-credentials
pip install ".[storage]"

Re-run aws_test.py. This time there should be no error and it should list the objects in the bucket.

GCP

Create an GCE instance (or use "default-service-account-test" ) in the "computer-vision-team" project. Make sure it has an attached service account that can access Storage.
ssh onto the instance and ensure python, pip, git are installed.

Run the following in the terminal:

git clone https://github.com/voxel51/eta.git`
cd eta
git checkout develop
pip install ".[storage]"

Run the following script in python. It should print an error and exit.

# gcp_test.py

import eta.core.storage as ecs

try:
    client = ecs.GoogleCloudStorageClient()
    client.list_files_in_folder("gs://voxel51-test")
except ecs.GoogleCredentialsError as e:
    print(e)
    exit()

Run the following in the terminal:

git checkout default-cloud-credentials
pip install ".[storage]"

Re-run gcp_test.py. This time there should be no error and it should list the objects in the bucket.

TODO

suppress logging from internal libraries

voxel51 / eta