When importing the large and wide PanStarrs detections table, STSci reports needing 41GiB of memory to complete the Finishing stage.
The Planning, Binning, and Finishing stages currently run inside the controller job, instead of inside dask workers. It can be trickier to add more memory to the controller (for various reasons).
Profile the Finishing stage to look for possible memory hogs and fix them.
Before submitting
Please check the following:
[ ] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
[ ] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
[ ] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Bug report
When importing the large and wide PanStarrs detections table, STSci reports needing 41GiB of memory to complete the Finishing stage.
The Planning, Binning, and Finishing stages currently run inside the controller job, instead of inside dask workers. It can be trickier to add more memory to the controller (for various reasons).
Profile the Finishing stage to look for possible memory hogs and fix them.
Before submitting Please check the following: