Refactorings to abstract input and output processing

clingen-data-model / clinvar-streams

1 stars 0 forks source link

Refactorings to abstract input and output processing #67

Closed theferrit32 closed 1 year ago

theferrit32 commented 2 years ago

read diff files lazily as a seq
read input messages as a lazy seq
generate output messages as a lazy seq
filter output messages as a lazy filter

There is a bug making it not identify duplicates when stream processing. These refactorings will make it easier to test and identify issues.

theferrit32 commented 1 year ago

The main files changed here is src/clinvar_raw/stream.clj and src/clinvar_raw/ingest.clj. Other files have minor changes, or changes accounting for the changes in stream.