AbsaOSS / fa-db

Functional Access to Database
Apache License 2.0
3 stars 0 forks source link

streaming functions for doobie and slick with tests #108

Open salamonpavel opened 11 months ago

salamonpavel commented 11 months ago

Added support for streaming for both Slick and Doobie. Since Slick works with Futures but for streaming it relies on IOs or other effect types there was a need to abstract the streaming operation in a new DBStreamingEngine.

There are following modules:

Closes #107

github-actions[bot] commented 11 months ago

JaCoCo code coverage report - scala 2.12.17

Total Project Coverage 58.19% :green_apple:
Module Coverage
fa-db:core Jacoco Report - scala:2.12.17 61.79% :x:
fa-db:streaming Jacoco Report - scala:2.12.17 0% :x:
Files |Module|File|Coverage [48.91%]|| |:-|:-|:-:|:-:| |fa-db:core Jacoco Report - scala:2.12.17|[NamingConvention.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2FNamingConvention.scala)|100%|:green_apple:| ||[StandardStatusHandling.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fstatus%2Fhandling%2Fimplementations%2FStandardStatusHandling.scala)|100%|:green_apple:| ||[DBSchema.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2FDBSchema.scala)|93.88%|:green_apple:| ||[ExplicitNamingRequired.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2FExplicitNamingRequired.scala)|87.5%|:green_apple:| ||[LettersCase.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2FLettersCase.scala)|85.29%|:green_apple:| ||[SnakeCaseNaming.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2Fimplementations%2FSnakeCaseNaming.scala)|84.29%|:green_apple:| ||[AsIsNaming.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2Fimplementations%2FAsIsNaming.scala)|40.63%|:x:| ||[DBEngine.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2FDBEngine.scala)|20%|:x:| ||[DBFunction.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2FDBFunction.scala)|6.83%|:x:| |fa-db:streaming Jacoco Report - scala:2.12.17|[DBStreamingFunction.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/streaming%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fstreaming%2FDBStreamingFunction.scala)|0%|:x:| ||[DBStreamingEngine.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/streaming%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fstreaming%2FDBStreamingEngine.scala)|0%|:x:|
github-actions[bot] commented 11 months ago

JaCoCo code coverage report - scala 2.13.12

Total Project Coverage 59.46% :green_apple:
Module Coverage
fa-db:core Jacoco Report - scala:2.13.12 63.49% :x:
fa-db:streaming Jacoco Report - scala:2.13.12 0% :x:
Files |Module|File|Coverage [49.53%]|| |:-|:-|:-:|:-:| |fa-db:core Jacoco Report - scala:2.13.12|[LettersCase.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2FLettersCase.scala)|100%|:green_apple:| ||[ExplicitNamingRequired.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2FExplicitNamingRequired.scala)|100%|:green_apple:| ||[NamingConvention.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2FNamingConvention.scala)|100%|:green_apple:| ||[StandardStatusHandling.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fstatus%2Fhandling%2Fimplementations%2FStandardStatusHandling.scala)|100%|:green_apple:| ||[DBSchema.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2FDBSchema.scala)|93.88%|:green_apple:| ||[SnakeCaseNaming.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2Fimplementations%2FSnakeCaseNaming.scala)|89.29%|:green_apple:| ||[AsIsNaming.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fnaming%2Fimplementations%2FAsIsNaming.scala)|72.22%|:x:| ||[DBEngine.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2FDBEngine.scala)|20%|:x:| ||[DBFunction.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/core%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2FDBFunction.scala)|7.11%|:x:| |fa-db:streaming Jacoco Report - scala:2.13.12|[DBStreamingFunction.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/streaming%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fstreaming%2FDBStreamingFunction.scala)|0%|:x:| ||[DBStreamingEngine.scala](https://github.com/AbsaOSS/fa-db/blob/a70daeef7199480ccd69d1ff6d556b4f754e1e07/streaming%2Fsrc%2Fmain%2Fscala%2Fza%2Fco%2Fabsa%2Ffadb%2Fstreaming%2FDBStreamingEngine.scala)|0%|:x:|
salamonpavel commented 11 months ago

Found a few tiny things and Jacoco coverage action is still failing (not on insufficient coverage though), so please take a look. Otherwise LGTM.

I pulled it, reviewed, compiled and ran tests.

Fixed this. The problem was in node version of Github's updated runner. Solved by an update of the jacoco github action version.

salamonpavel commented 11 months ago

@lsulak @jakipatryk Thanks for the initial review here and on the calls we had yesterday. Based on your comments I have made the streaming part for Slick as a separate package since additional dependencies are needed and we don't want to be importing transitive dependencies into our projects when streaming is not used. At the moment there is core module for non-streaming, streaming module for streaming, slick module for slick non-streaming, slick-streaming module for slick streaming and doobie module. Doobie module provides both non-streaming and streaming functions since Doobie already bring all the dependencis needed (works with fs2 streams under the hood) so there would be no benefit in splitting this into separate modules.