Closed htruong0 closed 4 years ago
jcwhitehead
commented
4 years ago Hi Henry, thanks for opening an issue. We multithreaded submission in #29 to speed up large jobs, which led to some problems when submitting to a single CE with too many threads. Could you let me know if you're multithreading submission, and if so, on how many threads?
The conclusion reached in #29 was that it should be ok with split_dur_ce=True and arc_submit_threads=5.
htruong0
commented
4 years ago Hi James, I'm using the template header where arc_submit_threads=1 and split_dur_ce=True by default, so I don't think it's even multithreading the submission.
jcwhitehead
commented
4 years ago Interesting - think that's new if so. @marianheil have you noticed this?
I can do some testing later. It's possible that there's been some change to the CEs. I don't suppose you verified that adding the sleep you mentioned fixes this? If not I'll explore that when I have a look.
scarlehoff
commented
4 years ago If the threads are set to 1 it might be a problem on the ce side?
htruong0
commented
4 years ago No I haven't actually tried adding in the sleep but Adam suggested it to me as a fix when I asked him why my jobs failed.
marianheil
commented
4 years ago So far I only ever saw this with multi threaded submission, but I haven't used the grid recently. There was an update to arc a few weeks ago, maybe that changed something.
For testing we should not randomly switch between ce1 and ce2, but submit to one as quick as possible (in one thread). In general we should properly (not randomly) alternate between ce1 and ce2 if we are submitting too quick.
DWalker487
commented
4 years ago Not read this very closely, but I wonder if it couldn't also be r/w related to the arc job.dat file? I never managed to overload a ce with just one thread as far as I can remember.If it's on the gridui, and the filesystem is being hammered that might cause this kind of thing. Jobs would appear to fail when they were submitted faster than they could be written to the db. Putting the jobs file on the scratch disks helped a lot with this IME.
marianheil
commented
4 years ago Not read this very closely, but I wonder if it couldn't also be r/w related to the arc job.dat file?
No, this is the problem here. The database was correctly written, all local job informations are stored. The actual error @htruong0 got with arccat/pyHepGrid man runcard.py -p is Path "./simplerun.py" does not seem to exist on the grid node.
I got the same error a while back when using multi core submission. Adams diagnosed the error now and then:
It looks like you've managed to submit too quickly. If two requests for a job hit the CE at the same time, then ARC can't keep up with its arc job ids and they collide.
jcwhitehead
commented
4 years ago @marianheil We can slowly accumulate fixes to this in #80 .
marianheil
commented
4 years ago I was able to reproduce the error. Submission worked flawless for multiple thousand jobs (even with multicore submission) and then failed twice (once in 4/500 jobs and once in 4/100), both on ce1 with single core submission and no load balancing.
There is nothing obviously going wrong in these jobs, but all failed jobs where directly after each other. I will try slowing down the submission.
Full output:
marianheil
commented
4 years ago After further testing this seems like arc is just acting weird. I can also get the error message with successful jobs e.g.:
$ arcstat -j /scratch/mheil/tst_grid/jobs.dat gsiftp://ce1.dur.scotgrid.ac.uk:2811/jobs/q2K
KDmKwZ3wnXk5IKnL2N00mABFKDmABFKDmwaHKDmABFKDmrEEEzm
Job: gsiftp://ce1.dur.scotgrid.ac.uk:2811/jobs/q2KKDmKwZ3wnXk5IKnL2N00mABFKDmABFKDmwaHKDmABFKDmrEEEzm
Name: EXAMPLE_RUN_155
State: Finished
Exit Code: 0
Status of 1 jobs was queried, 1 jobs returned information
$ arccat -e -j /scratch/mheil/tst_grid/jobs.dat gsiftp://ce1.dur.scotgrid.ac.uk:2811/jobs/q2KKDmKwZ3wnXk5IKnL2N00mABFKDmABFKDmwaHKDmABFKDmrEEEzm
Path "./simplerun.py" does not seem to existAnd, even worse, I can get jobs that are marked as failed but produce output:
$ pyHepGrid man runcard_example.py -B -s -j16 -fl fail -ip --error
> Using header file pyHepGrid.headers.template_header.py
Sourcing runcard
Value set: runcard_example arcbase : ../../../../../../scratch/mheil/tst_grid/jobs.dat
Value set: runcard_example baseSeed : 2
Value set: runcard_example copy_log : True
Value set: runcard_example dbname : ../../../../../../scratch/mheil/tst_grid/database
WARNING: exampleDir defined in runcard_example.py but not pyHepGrid.headers.template_header.py.
> Be very careful if you're trying to override attributes that don't exist elsewhere.
> Or even if they do.
Value set: runcard_example exampleDir : .
Value set: runcard_example executable_exe : executable_example.sh
Value set: runcard_example executable_src_dir : .
WARNING: grid_executable defined in runcard_example.py but not pyHepGrid.headers.template_header.py.
> Be very careful if you're trying to override attributes that don't exist elsewhere.
> Or even if they do.
Value set: runcard_example grid_executable : example/executable.tar.gz
Value set: runcard_example grid_input_dir : example/input
Value set: runcard_example grid_output_dir : example/output
Value set: runcard_example jobName : EXAMPLE_RUN
Value set: runcard_example producRun : 500
Value set: runcard_example provided_warmup_dir : .
Value set: runcard_example runcardDir : .
Value set: runcard_example runfile : simplerun.py
Value set: runcard_example runmode : backend_example.ExampleProgram
> Overriding run mode to <class 'backend_example.ExampleProgram'>
> Arc Production
WARNING: Applying job status filter. Please ensure you have run stats directly before this command to update job statuses.
> Job status filter: fail
[16] dummy_folder : config Done: 496 Waiting: 0 Running: 0 Failed: 4 Missing: 0 Total: 500
> Retrieving information for job 16: dummy_folder (Production) [1/1]
Job: gsiftp://ce1.dur.scotgrid.ac.uk:2811/jobs/9OCMDmHwZ3wnXk5IKnL2N00mABFKDmABFKDmIGHKDmABFKDmYVN0Zm
Name: EXAMPLE_RUN_153
State: Failed
Job Error: LRMS error: (1) Job failed
Job: gsiftp://ce1.dur.scotgrid.ac.uk:2811/jobs/724MDmIwZ3wnXk5IKnL2N00mABFKDmABFKDmTSHKDmABFKDmHQMBRn
Name: EXAMPLE_RUN_154
State: Failed
Exit Code: 0
Job Error: LRMS error: (1) Job failed
Job: gsiftp://ce1.dur.scotgrid.ac.uk:2811/jobs/QVVLDmLwZ3wnXk5IKnL2N00mABFKDmABFKDm4oHKDmABFKDmlRh0Tn
Name: EXAMPLE_RUN_156
State: Failed
Exit Code: 0
Job Error: LRMS error: (1) Job failed
Job: gsiftp://ce1.dur.scotgrid.ac.uk:2811/jobs/8X6MDmMwZ3wnXk5IKnL2N00mABFKDmABFKDme4HKDmABFKDmnbVPWo
Name: EXAMPLE_RUN_157
State: Failed
Exit Code: 0
Job Error: LRMS error: (1) Job failed
Status of 4 jobs was queried, 4 jobs returned information
> Printing information for job 16: dummy_folder (Production) [1/1]
Path "./simplerun.py" does not seem to exist
Path "./simplerun.py" does not seem to exist
Path "./simplerun.py" does not seem to exist
Path "./simplerun.py" does not seem to exist
>
$ gfal-ls xroot://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/mheil/example/output/output-dummy_folder-config-154.tar.gz -l
-r-------- 1 196100011 196100011 626 Jun 11 11:48 xroot://se01.dur.scotgrid.ac.uk/dpm/dur.scotgrid.ac.uk/home/pheno/mheil/example/output/output-dummy_folder-config-154.tar.gz(note: the name always includes the seed)
Of the 500 jobs I submitted 5 did show Path "./simplerun.py" does not seem to exist, 4 where reported as failed, but only one (the first one that failed, seed 153) was not running/producing actual output.
Basically we can't trust the output from arc, and instead have to check the results explicitly (maybe something for #34). Considering that this only affects 1/500 jobs without an obvious solution (see discussion of #80), I suggest not spending more time on this and keeping it as a "known issue" until we actually know why this is happening.
jcwhitehead
commented
4 years ago I'm totally puzzled by that - whichever ARC file the Path X does not seem to exist message comes from, it should be followed immediately by exit 1 (although we're several versions of ARC behind master, so that might not be correct).
How is it still running?
https://github.com/scarlehoff/pyHepGrid/blob/f2d36426cc24b2210d626567c57a8cdd32cec232/src/pyHepGrid/src/runArcjob.py#L293-L296
A small percentage of the time jobs submitted to arc fail because they are submitted to the same CE at the same time. Potential fix would be to add a small sleep (~0.1s) between job submissions.