marius-team / marius

Large scale graph learning on a single machine.
https://marius-project.org
Apache License 2.0
160 stars 45 forks source link

Integrated s3 client lib and its dependencies #119

Open basavaraj29 opened 2 years ago

basavaraj29 commented 2 years ago

the following libraries need to be installed in the base image

apt install zlib1g-dev
apt install zlib1g
apt-get install libssl-dev
apt install curl
apt-get install libcurl4-openssl-dev

creds are read from ~/.aws/credentials. format

[default]
aws_access_key_id=<...>
aws_secret_access_key=<...>

test code, will check in an example if the changes are good to go with.

#include <aws/s3/S3Client.h>
#include <aws/core/Aws.h>
#include <aws/s3/model/ListObjectsRequest.h>
#include <aws/s3/model/Object.h>

Aws::SDKOptions options;
Aws::InitAPI(options);
Aws::Client::ClientConfiguration client_config;
Aws::S3::S3Client s3_client(client_config);
Aws::S3::Model::ListObjectsRequest request;

const Aws::String bucket_name = "fb15k237";
request.WithBucket(bucket_name);
auto outcome = s3_client.ListObjects(request);

if (!outcome.IsSuccess()) {
    std::cerr << "Error: ListObjects: " <<
              outcome.GetError().GetMessage() << std::endl;
}
else {
    Aws::Vector<Aws::S3::Model::Object> objects =
            outcome.GetResult().GetContents();

    for (Aws::S3::Model::Object &object: objects) {
        std::cout << object.GetKey() << std::endl;
    }
}