Open sevein opened 1 month ago
Attention: Patch coverage is 75.64103%
with 19 lines
in your changes missing coverage. Please review.
Project coverage is 54.64%. Comparing base (
c0afd76
) to head (25f1c7e
).
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
🚨 Try these New Features:
Sorry for the late update @sevein. I took a quick look a while ago, really nice and good learnings! I think @djjuhasz is also looking at it and will do the actual code review.
@djjuhasz yes, I've been looking at it and will provide feedback. I'm having a hard time wrapping my head around the iterator code flow.
I've been looking at it and will provide feedback. I'm having a hard time wrapping my head around the iterator code flow.
The syntax is a bit confusing :cry:... I found the docstring of the compiler's rangefunc package somewhat useful: https://github.com/golang/go/blob/master/src/cmd/compile/internal/rangefunc/rewrite.go.
There are a couple of areas where I'm unsure if I'm doing things correctly:
CreatePreservationTasks
accept an iterator seemed convenient because it allows the user to simply provide a function that yields objects, without having to worry about chunking at all. I'm thinking in savePreservationTasks
, you'd only need to write a function that walks the slice of a3m jobs and produces preservation tasks or use some kind of map & transform helper like this. It seems a better option if compared with accepting a slice I think?batch
accepts and returns an iterator which is a pattern I haven't seen a lot in other projects, e.g. pkg.go.dev/slices
has functions where they either accept or return an iterator but not both. But I've found at least one example: ChunkToSlices.I haven't found yet an implementation like this in other codebases, I should probably look around more. I assume other poeple are also trying to solve the problem of ublk inserts with iterators? A project that is slightly related is https://github.com/achille-roussel/sqlrange.
This PR explores a new pattern for handling bulk database operations using Go 1.23's range-over-function feature. It introduces a new
CreatePreservationTasks
method in the persistence package, which accepts a generator of preservation tasks (iter.Seq[*datatype.PreservationTask]
). The method utilizes thebatch
function to group tasks into manageable batches and leveragesCreateBulk
to perform efficient bulk inserts in a single INSERT operation.To view the main idea of this proposal in isolation, visit: https://go.dev/play/p/y7tYvD7NO28.
In future PRs, this new function could be used in areas like CreateAIPActivity and JobTracker to improve bulk task creation.