search

zhafen / linefinder

A tool for finding and classifying the worldlines of Lagrangian parcels of mass, in the context of hydrodynamic simulations of galaxy formation.
https://zhafen.github.io/linefinder
MIT License
0 stars 1 forks source link

ID Selection fails for galaxies #95

Open zhafen opened 6 years ago

zhafen commented 6 years ago

Originally reported by Zachary Hafen (Bitbucket: zhafen, GitHub: zhafen)

In particular, it breaks somewhere in the process of looping...

To do...

zhafen commented 6 years ago

Original comment by Zachary Hafen (Bitbucket: zhafen, GitHub: zhafen)

This issue won't go away for parallel ID selection unless I change how I do it. Why? Well, if it takes 30 seconds to do the union of two sets, and I do that 600 times, then that's nearly as long as the non-parallel calculation is. This will need to be changed at some point.

zhafen commented 6 years ago

Original comment by Zachary Hafen (Bitbucket: zhafen, GitHub: zhafen)

There's still that bad memory leak in the ID finding with regular parallel python, so I can't use that...

zhafen commented 6 years ago

Original comment by Zachary Hafen (Bitbucket: zhafen, GitHub: zhafen)

Saving as an array helped. But the Stampede file system is also being really slow right now.

zhafen commented 6 years ago

Original comment by Zachary Hafen (Bitbucket: zhafen, GitHub: zhafen)

Turns out a lot of this is because of Stampede, I think. The calculation itself got done pretty quickly. But storing the results can take a loooong time. What if I don't store it as a set, but as an array instead?

zhafen commented 6 years ago

Original comment by Zachary Hafen (Bitbucket: zhafen, GitHub: zhafen)

I think it's stuck on the unifying results step. When I do just snapshot 600, but two jug processes, it takes a looong time on that step, and jug status seems to read incorrectly that it's on another step.

zhafen commented 6 years ago

Original comment by Zachary Hafen (Bitbucket: zhafen, GitHub: zhafen)

It runs fine when doing just snapshot 600, running a single jug process and waiting for the results.