search

JuliaCloud / AWS.jl

Julia interface to AWS
MIT License
159 stars 62 forks source link

Interest in wrapping the AWS Common Runtime #540

Closed Octogonapus closed 2 years ago

Octogonapus commented 2 years ago

Is there any interest in wrapping the AWS Common Runtime? Or wrapping something like the CRT for Python using PyCall.jl or PythonCall.jl? We have a need to use the AWS IoT SDK, so one of those projects would need to get wrapped at some level. Not sure if it fits in this package; maybe it would belong better as a new package in this organization?

ericphanson commented 2 years ago

If https://github.com/awslabs/aws-c-s3 can really get high throughput I think it would be really awesome to have. The others are less interesting to me personally but I’m sure some folks would be interested. I think the first step with making C-libraries accessible to Julia is to build them with Yggdrasil, and the second is to wrap them, I’ve never done that but I think Clang.jl is commonly used.

Probably AWS.jl isn’t the right repo for it, but JuliaCloud seems like a good place for such a repo.

Octogonapus commented 2 years ago

I have since created AWSCRT_jll.jl, LibAWSCRT.jl and AWSCRT.jl. In the future, I'll probably work on a higher level client for S3, but for now, my focus is on MQTT. This issue can probably be closed now.

Octogonapus commented 2 years ago

By the way, @ericphanson would you be interested in collaborating on the S3 client? We also have uses for very high S3 throughput, but they are in the future, so would you like me to contact you if we start working on one?

ericphanson commented 2 years ago

Yes, I would be interested! I’m not totally sure how much time I’ll have to help out but I’m definitely interested in improving high throughput s3 access from Julia.

jrevels commented 2 years ago

@Octogonapus I'll point out that @beacon-biosignals in general has a need for high-throughput reads and tabular data transforms in S3 - it's something we were planning on working on more in Q4 2022 / Q1 2023 :) would love to get a working group going at some point.

also, this is one of the primary topics we wanted to engage the community on in the JuliaCon 2022 BoF we're running on Data Engineering in Julia. Would love to discuss more there with you if you're be able to attend!