jlaura
commented
2 years ago Full job information:
export SACCT_FORMAT="JobID%20,JobName,User,Partition,NodeList,Elapsed,State,ExitCode,MaxRSS,AllocTRES%32"
(base) -bash-4.2$ sacct -j 45306855
JobID JobName User Partition NodeList Elapsed State ExitCode MaxRSS AllocTRES
-------------------- ---------- --------- ---------- --------------- ---------- ---------- -------- ---------- --------------------------------
45306855 lq30jig lweller longall mem1 00:40:44 FAILED 11:0 billing=1,cpu=1,mem=50G,node=1
45306855.batch batch mem1 00:40:44 FAILED 11:0 19785568K cpu=1,mem=50G,node=1
45306855.extern extern mem1 00:40:45 COMPLETED 0:0 0 billing=1,cpu=1,mem=50G,node=1If this runs in one environment and not another, we are going to need to know the differences between these environment. What OS are both of these machines running? Are the conda list outputs identical on both machines?
lwellerastro
jessemapel
ISIS version(s) affected: 5.0.2
Description
Jigsaw will segfault on astro's cluster when solving for camera accelerations and spacecraft position for my very large Kaguya TC south pole network. There are no error messages associated with the failure on the cluster and although it says a core was dumped, there is none. This happens whether I send a job, run an interactive job, or log on to a node directly and try and run it, or ask for an entire node and the memory available which is far more than the job needs.
It runs successfully on the astrovm's. Specifically, it runs on an astrovm built for me recently (igswzawgvsiplw), but it was built to match astrovm5.
This is not a memory issue as I watched the process run on my vm and it used ~20G of memory. From my job 45306855 that failed:
It looks like it fails right when it starts the first iteration.
I can send smaller quads to the cluster and solve for camera accelerations and spacecraft position without error, so this seems to be a size thing. I can also send the Kaguya TC polar network to the cluster and just solve for camera accelerations and that works as well (I think that uses about 17G of memory). The same is true for Titan - when I add spsolve=position it will not run on the cluster.
So the problem is specifically with large networks and solving for spacecraft position which is 100% necessary for both projects listed. This is likely related to https://github.com/USGS-Astrogeology/ISIS3/issues/3871
My Kaguya TC south pole network has 16238 images 548912 points 3899619 measures The points and measures exceed the LROC NAC south pole network.
How to reproduce
See data in /work/users/lweller/Isis3Tests/Jigsaw/NebSegFault/. See proc.txt for the command below.
See the very thin, not very informative stdout file SegFault_LOG_Jig from cluster run when/where it fails. Note, I added the output files and print.prt from the successful astrovm run for reference.
The takes a very long time to run, so I'd set maxits=1. When I first sent the job to the cluster I had it set to 5 iterations as below. I believe it took about 40+ minutes after sending to the cluster to start an iteration and die.
Possible Solution
Seems like nebula and it's nodes doesn't know something, or how to get access to something.
Additional context
Although I can run things my astrovm, I have limited memory there to run jigsaw, qnet and any other isis program at the same. Running Titan and Kaguya at the same time (which may happen since both take so long to run) limits what else I can do on the system. Additionally, there is only so much memory available on astrovm4 and astrovm5 for other project members to run their polar work when they get to it, in addition to the resources that are being used by the LROC NAC polar project.
We really need to be able to bundle these enormous networks on the cluster.
I was going to ask Janet to run her failing jigsaw on astrovm4 (has more memory than 5) to see if it is successful there.