monarch-initiative / koza

Data transformation framework for LinkML data models
https://koza.monarchinitiative.org/
BSD 3-Clause "New" or "Revised" License
48 stars 4 forks source link

Koza doesn't play nicely with generators #144

Open DnlRKorn opened 2 months ago

DnlRKorn commented 2 months ago

I tried to replace the following line of code

while (row := koza_app.get_row()) is not None:
xyz

with

koza_app = get_koza_app("go_annotation")
def row_gen():   while True:
       try:
           row = koza_app.get_row()
           yield row
       except StopIteration:
           return

for row in row_gen():
   xyz

But this caused some form of infinite loop to begin firing within the koza module

2024-08-16_14:09:34 | INFO | koza.io.reader.csv_reader | Finished processing 19 rows for go_annotation from file data/go/9606.go_annotations.gaf.gz 2024-08-16_14:09:34 | INFO | koza.io.reader.csv_reader | Finished processing 19 rows for go_annotation from file data/go/9606.go_annotations.gaf.gz 2024-08-16_14:09:34 | INFO | koza.io.reader.csv_reader | Finished processing 19 rows for go_annotation from file data/go/9606.go_annotations.gaf.gz 2024-08-16_14:09:34 | INFO | koza.io.reader.csv_reader | Finished processing 19 rows for go_annotation from file data/go/9606.go_annotations.gaf.gz 2024-08-16_14:09:34 | INFO | koza.io.reader.csv_reader | Finished processing 19 rows for go_annotation from file data/go/9606.go_annotations.gaf.gz

ptgolden commented 1 week ago

I'm fairly sure the issue here is that koza expects StopIteration to be raised to know when to stop running the transform script. Since that exception is never raised, the script continues to be run

ptgolden commented 1 week ago

i.e. it never breaks out of this loop:

https://github.com/monarch-initiative/koza/blob/fceafe5755e222dba14cecf8b7df09ca7dd16831/src/koza/app.py#L102-L119