pyathena provides a non-PEP async cursor that, if used correctly, could greatly accelerate runtime - it would be A Project to get it in, but would allow us to do things like queue several denormalized tables to run at once.
Some things to consider before going down this path:
How do we mark queries as parallelizable vs dependent?
How does this impact other databases?
How big of an overhaul to the existing codebase would this be?
pyathena provides a non-PEP async cursor that, if used correctly, could greatly accelerate runtime - it would be A Project to get it in, but would allow us to do things like queue several denormalized tables to run at once.
Some things to consider before going down this path: