New sink: Oracle Cloud Infrastructure 'oci_object_storage'

CarlDownsOracle commented 1 year ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Background

We have a customer that has standardized on Vector for their observability data collection needs.
Customer is moving AWS Kubernetes and Compute based workloads to Oracle Cloud Infrastructure (OCI).
Customer use case calls for archival of logs and metrics to OCI object storage.
Vector has several sink implementations, including aws_s3 sink.
https://vector.dev/docs/reference/configuration/sinks/aws_s3/

Attempted Solutions

Success with Vector

The aws_s3 sink works with OCI’s S3 compatible API using OCI auth credentials.

Customer Ask

Customer is asking if we can support an analog to the auth.assume_role feature in OCI because they are reticent to include any credentials in configuration files.
https://vector.dev/docs/reference/configuration/sinks/aws_s3/#auth.assume_role
OCI’s analog is something called an instance principal where Compute-based OCI API calls are authorized without passing in credentials.
https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm

Proposal

We are looking to see if there is an implementation path for an OCI-specific sink that can take advantage of the instance principal mechanism.

References

No response

Version

No response

CarlDownsOracle commented 1 year ago

Hi @jszwedko, just checking in to see what your thoughts are on this request.

jszwedko commented 1 year ago

Hi @CarlDownsOracle !

The request makes sense. I saw another recent issue, https://github.com/vectordotdev/vector/issues/16665 , for making it easier to use the aws_s3 sink with OCI object storage since it has an AWS S3 compatible endpoint, but I see your note here about wanting to support OCI's authentication mechanisms, for which a specialized sink makes sense.

We recently added a sink that leverages OpenDAL -- an abstraction over cloud storage engines. Adding OCI object storage support there could make it easier for Vector to support in the future (https://github.com/apache/incubator-opendal/issues/377 seems to be tracking that over there).

Thanks for filing!

CarlDownsOracle commented 1 year ago

Thanks Jesse !

I’ll look at the OpenDAL project … thanks for passing the ticket over as well!

Regards, Carl

[signature_1585353333] Carl Downs Principal Solution Architect Program Management Office

From: Jesse Szwedko @.> Date: Thursday, March 16, 2023 at 11:45 AM To: vectordotdev/vector @.> Cc: Carl Downs @.>, Mention @.> Subject: [External] : Re: [vectordotdev/vector] New sink: Oracle Cloud Infrastructure 'oci_object_storage' (Issue #16766)

Hi @CarlDownsOraclehttps://urldefense.com/v3/__https:/github.com/CarlDownsOracle__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_69wZwr2Q$ !

The request makes sense. I saw another recent issue, #16665https://urldefense.com/v3/__https:/github.com/vectordotdev/vector/issues/16665__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_68FM0lCw$ , for making it easier to use the aws_s3 sink with OCI object storage since it has an AWS S3 compatible endpoint, but I see your note here about wanting to support OCI's authentication mechanisms, for which a specialized sink makes sense.

We recently added a sink that leverages OpenDALhttps://urldefense.com/v3/__https:/github.com/apache/incubator-opendal__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_4rARa7dA$ -- an abstraction over cloud storage engines. Adding OCI object storage support there could make it easier for Vector to support in the future (apache/incubator-opendal#377https://urldefense.com/v3/__https:/github.com/apache/incubator-opendal/issues/377__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_6la4M6Kw$ seems to be tracking that over there).

Thanks for filing!

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/vectordotdev/vector/issues/16766*issuecomment-1472332866__;Iw!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_7SZ7jNmg$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AOTLLTSONFAXRAXM257KWDLW4M7T3ANCNFSM6AAAAAAVW5P6NY__;!!ACWV5N9M2RV99hQ!MW0-l0FH98jEDnnTAzNLT_B1R2DNX7QCGPGgPx8bbrmNaKsDgXOrMD8JOULXhBW3LtyIvufHBjk0z_t1G_4h8g1MVA$. You are receiving this because you were mentioned.Message ID: @.***>

Xuanwo commented 1 year ago

Challenge accepted! We (at OpenDAL) will soon implement support for OCI.

CarlDownsOracle commented 1 year ago

Thanks @Xuanwo ! Please reach out if you have any questions ... or we can assist in some way.

Xuanwo commented 1 year ago

Thanks @Xuanwo ! Please reach out if you have any questions ... or we can assist in some way.

I believe that the Always Free Tier will suffice for our development and integration testing needs. I'll let you know if we encounter any further issues. Thank you in advance.

CarlDownsOracle commented 1 year ago

Yes the Always Free Tier should be sufficient. You probably already have this:

https://docs.oracle.com/en-us/iaas/Content/Identity/Tasks/callingservicesfrominstances.htm

Thank you!

CarlDownsOracle commented 1 year ago

Hello @Xuanwo happy Friday. Just checking in to see how things are going.

Xuanwo commented 1 year ago

Hello @Xuanwo happy Friday. Just checking in to see how things are going.

Happy Friday to you too! This week, I am working on the implementation. However, there is a potential block as I am unable to register an OCI account which is necessary for integrating with real services. Despite submitting a ticket 10 days ago (ticket number 230321-001192, account oracle@xuanwo.io), I have not received any response yet. Would it be possible for you to assist me in resolving this issue?

CarlDownsOracle commented 1 year ago

Let me see if I can help. That ticket number ... may I ask in which system? Can't find it in My Oracle Support.

Xuanwo commented 1 year ago

Let me see if I can help. That ticket number ... may I ask in which system? Can't find it in My Oracle Support.

I think it called Oracle Customer Service, sent by oraclehelp_ww@oracle.com

CarlDownsOracle commented 1 year ago

Hi Xuanwo –

I’m unable to locate your request. If you want to DM or email me directly, I can help you.

Regards, Carl

[signature_117748667] Carl Downs Principal Solution Architect Program Management Office

From: Xuanwo @.> Date: Friday, March 31, 2023 at 12:47 PM To: vectordotdev/vector @.> Cc: Carl Downs @.>, Mention @.> Subject: [External] : Re: [vectordotdev/vector] New sink: Oracle Cloud Infrastructure 'oci_object_storage' (Issue #16766)

Let me see if I can help. That ticket number ... may I ask in which system? Can't find it in My Oracle Support.

I think it called Oracle Customer Service, sent by @.***

— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https:/github.com/vectordotdev/vector/issues/16766*issuecomment-1492364789__;Iw!!ACWV5N9M2RV99hQ!JlCeGCOTja4dQEa1UQVt9XhC4YWGoxT-Tt43MuNJUYZ027N4Tqt6MR2kFwuTw-djHOYYAiDkFiup8TvKUcGeCzjCzg$, or unsubscribehttps://urldefense.com/v3/__https:/github.com/notifications/unsubscribe-auth/AOTLLTVIZQEJG4U67ZQEXMDW64KARANCNFSM6AAAAAAVW5P6NY__;!!ACWV5N9M2RV99hQ!JlCeGCOTja4dQEa1UQVt9XhC4YWGoxT-Tt43MuNJUYZ027N4Tqt6MR2kFwuTw-djHOYYAiDkFiup8TvKUcH5OdTQrA$. You are receiving this because you were mentioned.Message ID: @.***>

Xuanwo commented 1 year ago

Hi, this issue is blocked by https://github.com/apache/incubator-opendal/issues/2563

All of our team members were unable to obtain an Oracle Cloud account, even after contacting the seller through formal channels.

vectordotdev / vector