search

mmalecot / file-format

Crate for determining the file format of a given file or stream
Apache License 2.0
94 stars 15 forks source link

Add async support via Tokio #66

Open yasamoka opened 2 weeks ago

yasamoka commented 2 weeks ago

First of all, thank you for your great work on this crate!

This PR adds support for readers that implement Tokio's AsyncRead and AsyncSeek traits.

My use case is determining the file format of potentially large S3 objects without having to read entire objects into memory first. A reader that implements AsyncRead is provided courtesy of aws_smithy_types, but for AsyncSeek I had to implement my own wrapper on top. An example of such can be provided if needed.

Adding async support required moving common functionality away from the file format readers first, then introducing async implementations of those readers in parallel with the sync versions. The commit sequence shows this step-by-step.

Please let me know if there are any changes required.

mmalecot commented 2 weeks ago

Thanks for your PR! Can you please rebase on dev branch as I did changes today? I'm going to review your modifications, but the feature seems interesting :).

yasamoka commented 2 weeks ago

On it!

yasamoka commented 2 weeks ago

Done.

mmalecot commented 2 weeks ago

There’s quite a bit of code duplication between the sync and async reader implementations: there might be something we can do with macros.

I'll try to think about it on my own, but maybe we can take inspiration from other crates like surf or reqwest.

yasamoka commented 1 week ago

Hello, what was decided regarding this? I'd like to know. Thanks!

mmalecot commented 1 week ago

Hi, I closed the PR by deleting the dev branch, I've just restored it. Still need to see how to better manage readers without duplicating too much code.

yasamoka commented 1 week ago

Do let me know if I can help with something after you've thought about it.