AnswerDotAI / fastdata

Apache License 2.0
121 stars 7 forks source link

Fix generate to return results in submission order #8

Closed algal closed 4 days ago

algal commented 1 week ago

This updates generate to return generation results in the inputs order, rather than in the order generation jobs completed.

It still submits the jobs in inputs order, so the jobs will start in inputs order.

It relies on as_completed to determine, to update the progress bar whenever a job is completed.

This does not add any timeout or error handling logic, so we are still relying on the user to build that into job definitions -- e.g., to define jobs such that they will return None after a timeout period or in the event of an error.

I have lightly tested it, by executing 50 jobs with 5 workers, and inspecting results. Let's discuss @ncoop57

ncoop57 commented 1 week ago

Oh I really like this solution!!

ncoop57 commented 1 week ago

@algal Could you please add some tests into the nb that showcase it keeps thing in order? It can be a super small test.

algal commented 6 days ago

@ncoop57 I added a test which provides soft confirmation that the generate function now returns outputs in the order of inputs.