search

chapel-lang / chapel

a Productive Parallel Programming Language
https://chapel-lang.org
Other
1.79k stars 420 forks source link

CHPL_COMM_DEBUG=1 w/ GASNet requires either CHPL_DEVELOPER=1 or OPTIMIZE=0 #7011

Closed PHHargrove closed 7 years ago

PHHargrove commented 7 years ago

I am trying to deploy some CI scripts to test Chapel's "master" with GASNet's "develop" on a daily basis. While I am having reasonably good luck with default builds, I am not having much luck with getting CHPL_COMM_DEBUG to work as I am expecting.

This is the second (less pressing for me) issue I've encountered.

I am building Chapel with the 1-line work-around I describe in #7010, and passing CHPL_COMM=gasnet CHPL_COMM_SUBSTRATE=udp CHPL_DEBUG=1 to gmake. By default I am seeing the following while compiling comm-chapel.c:

/tmp/BLD/dbg/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-everything/debug/include/udp-conduit/amudp_const.h:64:6: error: #error Tried to compile AMUDP client code with optimization enabled but also AMUDP_DEBUG (which seriously hurts performance). Disable C and C++ compiler optimization or reconfigure/rebuild without --enable-debug
     #error Tried to compile AMUDP client code with optimization enabled but also AMUDP_DEBUG (which seriously hurts performance). Disable C and C++ compiler optimization or reconfigure/rebuild without --enable-debug
      ^

I find that I can avoid this problem by adding CHPL_DEVELOPER=1 to the gmake command line.
Personally, I am OK with that one added setting for use within our CI scripting.
However, my reading of the Chapel documentation didn't sound as if CHPL_DEVELOPER=1 is an intended prerequisite for CHPL_COMM_DEBUG=1 (but I could have missed it).

Also, since I am uncertain what else might be happening as a side-effect of setting CHPL_DEVELOPER=1, I am wondering: is there some other approach I could/should be using to avoid this error? I see OPTIMIZE in the Makefiles, but that doesn't appear to be "user-facing" since it lacks a CHPL_ prefix.

bradcray commented 7 years ago

On your command line, CHPL_DEBUG is the setting that stands out as being unfamiliar to me. Does it work if you don't set that? And what caused you to set it?

You may have better luck setting DEBUG=1 which is what we do more often (and it looks like it results in CHPL_DEBUG being set as well, so may result in better compatibility.

You're right that the naming of OPTIMIZE (and here, DEBUG) is arguably at odds with other CHPL_-prefixed variables. If I had to rationalize the scheme all these years later, I might say that CHPL_* variables are things that we originally thought of as being set in the environment whereas DEBUG, OPTIMIZE, and WARNINGS are things we imagined being set directly on the make command-line (or, more often, indirectly by setting CHPL_DEVELOPER in the environment).

Fair (though belated) warning: Our Makefiles are somewhat spaghetti that rarely improves over time, in large part because it's so hard to test that one hasn't broken Makefiles upon changing them that there's a high "let sleeping dogs lie" factor in play.

bradcray commented 7 years ago

Ooh, but to our credit, we document these Makefile variables!

PHHargrove commented 7 years ago

@bradcray

Summary version:

Setting or not setting CHPL_DEBUG or DEBUG appears unrelated to the reported problem.
I probably should have omitted that from the report to reduce confusion.

Use of OPTIMIZE=0, however, works-around my problem (and I've updated the title accordingly).
That is enough for me, and I leave it up to you folks to decide if this requirement for CHPL_COMM_DEBUG to work is acceptable.

Full version:

On your command line, CHPL_DEBUG is the setting that stands out as being unfamiliar to me. Does it work if you don't set that? And what caused you to set it?

If I don't set CHLPL_DEBUG

I have retried with env CHPL_COMM=gasnet CHPL_COMM_SUBSTRATE=udp CHPL_COMM_DEBUG=1 gmake all to match your comment about environment vs command line, and to remove the use of CHPL_DEBUG.
The result is still the same error compiling comm-gasnet.c.

Use of DEBUG=1

I tried the command as above with the sole addition of DEBUG=1 to the end of the command line. This still yields the same error.

Use of OPTIMIZE=0

I tried again, with OPTIMIZE=0 both instead of DEBUG=1 and in addition to it.
Both combinations work for me.

Why I set CHPL_DEBUG in the first place

In my testing I encountered a SEGV and needed debuggable core files.
I never removed the setting after that.
The SEGV turned out to be a GASNet bug (fixed by Dan this morning) somewhat specific to Chapel's construction of multi-kilobyte command lines (environment serialization).
So this CI effort is paying off for both sides already.

Given the documentation (thanks for the pointer), I will use DEBUG the next time I need to debug something.

bradcray commented 7 years ago

Thanks for the further details Paul -- I'll try to reproduce now that I understand that it's not CHPL_DEBUG-specific.

bradcray commented 7 years ago

So far, I've been unable to reproduce this. @ronawho, have you tried?

@PHHargrove, where you wrote comm-chapel.c above, did you mean comm-gasnet.c? If so, what's your failing command line? Mine seems to be:

gcc -c  -MMD -MP -O3 -O3 --param max-inline-insns-single=35000 --param inline-unit-growth=10000 --param large-function-growth=200000 -Winline    -DCHPL_TASKS_MODEL_H=\"tasks-qthreads.h\" -DCHPL_THREADS_MODEL_H=\"threads-none.h\" -DCHPL_WIDE_POINTER_STRUCT -DCHPL_OPTIMIZE -DNDEBUG -DCHPL_JEMALLOC_PREFIX=chpl_je_ -DCHPL_HAS_GMP -D_GNU_SOURCE=1 -DGASNET_PAR -D_REENTRANT -I/data/cf/chapel/bradc/chapel/third-party/gasnet/install/linux64-gnu-none/seg-everything/nodbg/include -I/data/cf/chapel/bradc/chapel/third-party/gasnet/install/linux64-gnu-none/seg-everything/nodbg/include/udp-conduit -Wno-unused -Wno-unused-parameter -Wno-address -DGASNET_NEEDS_MAX_SEGSIZE -I/data/cf/chapel/bradc/chapel/third-party/qthread/install/linux64-gnu-none-flat/include -I. -I../../../include/localeModels/flat -I../../../include/localeModels -I../../../include/comm/gasnet -I../../../include/comm -I../../../include/tasks/qthreads -I../../../include/threads/none -I../../../include -I../../../include/qio -I../../../include/atomics/intrinsics -I../../../include/mem/jemalloc -I/data/cf/chapel/bradc/chapel/third-party/utf8-decoder -I../../../../build/runtime/linux64/gnu/arch-none/loc-flat/comm-gasnet/udp/everything/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-intrinsics/gmp/hwloc/re2/wide-struct/fs-none/include  -I/data/cf/chapel/bradc/chapel/third-party/jemalloc/install/linux64-gnu-none/include -I/data/cf/chapel/bradc/chapel/third-party/gmp/install/linux64-gnu-none/include -I/data/cf/chapel/bradc/chapel/third-party/hwloc/install/linux64-gnu-none-flat/include -o ../../../../build/runtime/linux64/gnu/arch-none/loc-flat/comm-gasnet/udp/everything/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-intrinsics/gmp/hwloc/re2/wide-struct/fs-none/src/comm/gasnet/comm-gasnet.o comm-gasnet.c

Thanks.

PHHargrove commented 7 years ago

@bradcray Yes, I definitely meant comm-gasnet.c.

Below is a failing command line. Basically the gasnet code is rejecting the combination of --enable-debug at configure time with the presence of -O (as indicated by gcc having defined __OPTIMIZE__).

source util/setchplenv.bash && \
  env CHPL_COMM=gasnet CHPL_COMM_SUBSTRATE=udp CHPL_COMM_DEBUG=1  \
    /usr/bin/gmake -j4 all
[...]
gcc -c  -MMD -MP -O3 -g3   -std=gnu99 -DCHPL_TASKS_MODEL_H=\"tasks-qthreads.h\" -DCHPL_THREADS_MODEL_H=\"threads-none.h\" -DCHPL_WIDE_POINTER_STRUCT -DCHPL_COMM_DEBUG -DCHPL_OPTIMIZE -DNDEBUG -DCHPL_JEMALLOC_PREFIX=chpl_je_ -DCHPL_HAS_GMP -D_GNU_SOURCE=1 -DGASNET_PAR -D_REENTRANT -I/tmp/BLD/dbg/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-everything/debug/include -I/tmp/BLD/dbg/chapel/third-party/gasnet/install/linux64-gnu-unknown/seg-everything/debug/include/udp-conduit -Wall -Wpointer-arith -Wnested-externs -Wwrite-strings -Wmissing-format-attribute -Wno-unused -Wno-unused-parameter -Wno-address -std=gnu99 -O0 -DGASNET_NEEDS_MAX_SEGSIZE -I/tmp/BLD/dbg/chapel/third-party/qthread/install/linux64-gnu-unknown-flat/include -I. -I../../../include/localeModels/flat -I../../../include/localeModels -I../../../include/comm/gasnet -I../../../include/comm -I../../../include/tasks/qthreads -I../../../include/threads/none -I../../../include -I../../../include/qio -I../../../include/atomics/intrinsics -I../../../include/mem/jemalloc -I/tmp/BLD/dbg/chapel/third-party/utf8-decoder -I../../../../build/runtime/linux64/gnu/arch-unknown/loc-flat/comm-gasnet/udp/everything/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-intrinsics/gmp/hwloc/re2/wide-struct/fs-none/include  -I/tmp/BLD/dbg/chapel/third-party/jemalloc/install/linux64-gnu-unknown/include -I/tmp/BLD/dbg/chapel/third-party/gmp/install/linux64-gnu-unknown/include -I/tmp/BLD/dbg/chapel/third-party/hwloc/install/linux64-gnu-unknown-flat/include -o ../../../../build/runtime/linux64/gnu/arch-unknown/loc-flat/comm-gasnet/udp/everything/tasks-qthreads/tmr-generic/unwind-none/mem-jemalloc/atomics-intrinsics/gmp/hwloc/re2/wide-struct/fs-none/src/comm/gasnet/comm-gasnet.o comm-gasnet.c
bradcray commented 7 years ago

OK, thanks -- I see what I missed and have been able to reproduce it now.

bradcray commented 7 years ago

Having thought about this a bit more...

Paul, with respect to your testing build, I think it'd be find either to take the approach of using CHPL_DEVELOPER=1 or using whatever combination of the main settings it implies by default that you'd find useful (OPTIMIZE=0, DEBUG=1, WARNINGS=1 — as I think you know by now, these respectively disable optimizations, enable debugging, and enable warnings that we use to keep our code clean-ish, respectively). Other things that CHPL_DEVELOPER does at make-time are fairly innocuous, such as generating [emacs | vim] tags files by default. Having CHPL_DEVELOPER set at chpl invocation time has a lot of other impacts on things, but I don't think you're considering that.

With respect to addressing the possibility of someone tripping over this error in the future, it seems like our main options are:

I'd be OK with any of these approaches and am curious what others think (particularly @ronawho).

PHHargrove commented 7 years ago

@bradcray

Paul, with respect to your testing build,...

Thanks for the recommendation(s).
Based on the description you gave, I'll plan to use CHPL_DEVELOPER so we can know quickly if changes we've made in GASNet produce warnings that collide with your -Werror.

ronawho commented 7 years ago

We actually already force -O0 for CHPL_COMM_DEBUG in our gnu/clang/intel makefiles. Where we were going wrong is that we were mistakenly setting -DNDEBUG since #1099

I'm just going to add -UNDEBUG do the same line that's throwing -O0 with https://github.com/chapel-lang/chapel/pull/7678

Side note -- the comment above the #error we're hitting cracked me up: 

/* idiot proofing */
#if defined(AMUDP_DEBUG) && (defined(__OPTIMIZE__) || defined(NDEBUG))
    #error Tried to compile AMUDP client code with optimization enabled but also AMUDP_DEBUG...
#endif