search

FluxML / MacroTools.jl

MacroTools provides a library of tools for working with Julia code and expressions.
https://fluxml.ai/MacroTools.jl/stable/
Other
311 stars 79 forks source link

make sourcewalk process files async #115

Closed oxinabox closed 5 years ago

oxinabox commented 5 years ago

sourcewalk is too slow

MikeInnes commented 5 years ago

Is this safe? e.g. might we end up opening 1,000 file handles or something and erroring out? To some extent I'm ok with "try it and see" but it'd be nice to have a sense if you know.

Also, what kind of speedup does this give? Threading would be cool but I'd expect that most of the time is currently spent in the sourcewalk computation itself rather than file handling (but could be wrong of course).

oxinabox commented 5 years ago

I've followed similar patterns when downloading thousands of files and that went fine. This has to be less sensitive that that.

I've not done time trials, but it certainly feels more springy with this change than it did before.

MikeInnes commented 5 years ago

Sometimes this differs between OSs and such as well, though maybe there's already something handling this in the task system. Can't hurt to give it a shot.