typelevel / fs2

Compositional, streaming I/O library for Scala
https://fs2.io
Other
2.38k stars 603 forks source link

Replace fs2.hash with fs2.hashing.Hashing[F] #3454

Closed mpilquist closed 3 months ago

mpilquist commented 4 months ago

The fs2.hash object provides the ability to compute the hash of a stream of bytes (i.e. Stream[F, Byte]). The hash computation is modeled as a Pipe[F, Byte, Byte], where pulling on the output stream results in all the source bytes getting hashed and then a final chunk being emitted that contains the hash of the seen bytes.

This API is not expressive enough for common use cases though. Consider the case where a Stream[F, Byte] needs to be written to a file (or uploaded to an S3 bucket, or sent to a socket, etc) and the hash also needs to be stored. The current hashing API makes this awkward -- either requiring heavy duty machinery like broadcastThrough or requiring the stream to be processed twice.

This PR deprecates the fs2.hash object and replaces it with the new fs2.hashing package. The entry point to the new package is the Hashing[F] capability trait, allowing the creation of Hash[F] objects as well as providing various convenience methods.

The hashing package contains:

The Hashing[F] trait returns Hash[F] objects as resources (i.e. Resource[F, Hash[F]]) because (on some platforms) they have to be released after computations are complete.

With this new API, writing the contents of a source to a file and then subsequently writing a hash to a separate file, while processing the source just once, can be accomplished like so:

def writeWithHash[F[_]: Files: Hashing: MonadCancelThrow](path: Path): Pipe[F, Byte, Nothing] =
  source =>
     Stream.resource(Hashing[F].sha256).flatMap { h =>
       h.observe(source, Files[F].writeAll(path)).through(Files[F].writeAll(Path(s"$path.sha256")))
     }

The Hashing object also contains utility functions for hashing a pure stream and a chunk.

val h1 = Hashing.hashChunk(HashAlgorithm.SHA256, Chunk.array("The quick brown fox".getBytes))
val h2 = Hashing.hashPureStream(HashAlgorithm.SHA256, Stream.chunk(Chunk.array("The quick brown fox".getBytes)))
mpilquist commented 4 months ago

@armanbilge Thanks, addressed comments