Closed antonst closed 8 years ago
strangely enough for smaller runs everything works fine. I would assume there are some nodes on gordon for which ssh keys are not properly configured.
Do you mean that the job isn't even starting with 1000 replicas, or it runs but failed halfway? If it failed halfway running, it could be that some replicas just didn't run for some reason. I saw this on Gordon too.
A quick check is to go to the job directory in the sandbox and do
tail -n 1 unit*/*mdout
A normally terminated Amber job should say something like:
| wallclock() was called 290025 times
If something else, or not even a mdout file there, then that replica failed.
Given that this is also a SDSC site, it might be the same SSH rate limiting as we see on Comet.
Do you mean that the job isn't even starting with 1000 replicas, or it runs but failed halfway?
No job starts and some of the CU's finish the first Amber run, while others don't even start.
Given that this is also a SDSC site, it might be the same SSH rate limiting as we see on Comet.
I would like to understand if this is specific to my account on Gordon. Mark, do you have the data for experiments with tasks other than bin/sleep or bin/date on Gordon which utilize more than 1000 cores concurrently?
Its a per-node cu limit, not a total concurrent cu limit afaik.
But I dont have such/any data on Gordon.
Its a per-node cu limit, not a total concurrent cu limit afaik.
So in theory if we use more cores per CU we can avoid this issue?
So in theory if we use more cores per CU we can avoid this issue?
Yes, at least on Comet I empirically verified that.
Great! Will give it a try.
with CU STDERR: