search

JuliaParallel / Dagger.jl

A framework for out-of-core and parallel execution
Other
610 stars 66 forks source link

Add support for limiting the evaluations of a streaming DAG #519

Closed JamesWrigley closed 1 month ago

JamesWrigley commented 1 month ago

This is useful for testing and benchmarking.

BTW I added the stream_max_evals option to SchedulerOptions and ThunkOptions, but I see that the other streaming options are sort of used adhoc and filtered out before being passed upwards. Is that just for quick prototyping? Or should the streaming options go somewhere else?

JamesWrigley commented 1 month ago

Also attempted to improve CI by installing the migration-helper branch of MemPool.jl in 4ff2fa1.

JamesWrigley commented 1 month ago

I figured out why my filtering was failing, I was doing it in the wrong place :stuck_out_tongue: There was some filtering in initialize_streaming!() that I think doesn't actually do anything because those options are in spec.options.propagate, not spec.options. I removed that filter and added it in the proper place in finalize_streaming!() in f634923.

(P.S. if you approve, I'll rebase this before merging to squash the fixup commits)

JamesWrigley commented 1 month ago

The CI seems to be a little overloaded so I'm gonna roguishly go ahead and merge this :pirate_flag: