Closed karanjeets closed 7 years ago
Hi Karanjeet,
Can you give me some specifics on the system you are using?
Lucas A. Wilson, Ph.D. Director, Training & Professional Development Texas Advanced Computing Center The University of Texas at Austin +1 512 232 7351 lwilson@tacc.utexas.edumailto:lwilson@tacc.utexas.edu
From: Karanjeet Singh [mailto:notifications@github.com] Sent: Tuesday, August 30, 2016 1:12 AM To: TACC/launcher launcher@noreply.github.com Subject: [TACC/launcher] Error when running launcher (#7)
Can anyone please help here?
Below are the logs:
Tue Aug 30 00:09:08 CDT 2016
WARNING: LAUNCHER_WORKDIR variable not set. Using current directory.
Launcher: Setup complete.
------------- SUMMARY ---------------
Number of hosts: 1
Working directory: /data/projects/G-817549/aerosols/jobs
Processes per host: 2
Total processes: 2
Total jobs: 2
Scheduling method: dynamic
Launcher: Starting parallel tasks...
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
WARNING: No response from dynamic task server. Retrying...
WARNING: No response from dynamic task server. Retrying...
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
WARNING: No response from dynamic task server. Retrying...
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
WARNING: No response from dynamic task server. Retrying...
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
WARNING: No response from dynamic task server. Retrying...
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
WARNING: No response from dynamic task server. Retrying...
Ncat: Invalid -d delay "c251-127" (must be greater than 0). QUITTING.
WARNING: No response from dynamic task server. Retrying...
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/TACC/launcher/issues/7, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABLfulnImqihoB0Zp9KiVRBLNYB0ZJ25ks5qk8m0gaJpZM4JwLd-.
Thanks, @lwilson
I am running this on Wrangler. The SLURM script and job file are located at (0) & (1) respectively.
(0): /data/projects/G-817549/aerosols/jobs/launcher.slurm (1): /data/projects/G-817549/aerosols/jobs/check
Hi @karanjeets,
The -d flag was added for Lonestar5, and doesn't affect other systems which are using the BSD nc command. CentOS 7 (which is on Wrangler) uses the nmap/ncat implementation, which uses this flag for a different purpose. Upstream versions of nmap/ncat will include a --no-shutdown option which implements the same functionality, but Wrangler's version doesn't include this fix.
Long story short, I will build a small fix to detect whether the system has nc or ncat, and change the flags accordingly. For now, you can change the following lines in launcher
to have no -d flag:
71: export LAUNCHER_JID=`nc $LAUNCHER_DYN_SRV`
103: export LAUNCHER_JID=`nc $LAUNCHER_DYN_SRV`
Sorry about that. I will be sure to get this fix put in ASAP. I'll leave this issue open and will link to it when I push the fix.
-Luke
Hi @lwilson,
Thanks a lot for the fix. Minor Correction - The lines, you pointed out, are present in "launcher" script and not in "paramrun".
+1 to keep the issue open. Let me also change the subject of this issue to include the system name. It will help people to relate.
We are seeing this error in stampede-KNL. The fix works nicely.
I fixed my comment to reference launcher
and not paramrun
. I'm still working on a method for detecting which version is being used, and therefore which flags to include.
I have decided to remove the -d option for now since it is Cray specific.
Can anyone please help here?
Below are the logs: