search

apache / opendal

Apache OpenDAL: access data freely.
https://opendal.apache.org
Apache License 2.0
3k stars 416 forks source link

feat: add a dns cache on resolving s3 #4068

Closed flaneur2020 closed 5 months ago

flaneur2020 commented 5 months ago

we runs an olap engine with opendal, while processing heavy bounch of analysis workloads, the highly concurrent s3 accesses will produce lots of dns resolve requests.

IMHO we can consider adding a small dns resolve cache on before requesting s3.

flaneur2020 commented 5 months ago

cc @everpcpc

Xuanwo commented 5 months ago

We already have DNS caching in HttpClient using trust-dns. The challenge is to implement global DNS caching across all OpenDAL operators or within your project.

Xuanwo commented 5 months ago

Ok, opendal already expose this to user, we don't need to touch opendal.

Xuanwo commented 5 months ago

This issue will be resolved by https://github.com/Xuanwo/reqwest-hickory-resolver