search

NOAA-EMC / NCEPLIBS-bufr

The NCEPLIBS-bufr library contains routines and utilites for working with the WMO BUFR format.
Other
44 stars 19 forks source link

memory problem in iupm.f #319

Closed edwardhartnett closed 1 year ago

edwardhartnett commented 1 year ago

When I run intest1.F90 with address santizer I get errors, for all three versions of the test:


ed@koko:~/NCEPLIBS-bufr/b/test$ ./intest1_4 
 Testing reading IN_1, CRBMG with OPENBF IO = SEC3
=================================================================
==53343==ERROR: AddressSanitizer: global-buffer-overflow on address 0x56204eed8c20 at pc 0x56204ee32e9b bp 0x7ffea323a0c0 sp 0x7ffea323a0b0
READ of size 8 at 0x56204eed8c20 thread T0
    #0 0x56204ee32e9a in iupm_ /home/ed/NCEPLIBS-bufr/src/iupm.f:53
    #1 0x56204ee11819 in wrdlen_ /home/ed/NCEPLIBS-bufr/src/wrdlen.F:110
    #2 0x56204ee17504 in cobfl_ /home/ed/NCEPLIBS-bufr/src/cobfl.c:140
    #3 0x56204ededa5c in intest1 /home/ed/NCEPLIBS-bufr/test/intest1.F90:33
    #4 0x56204edee9b5 in main /home/ed/NCEPLIBS-bufr/test/intest1.F90:105
    #5 0x7faed8366082 in __libc_start_main ../csu/libc-start.c:308
    #6 0x56204eded5cd in _start (/home/ed/NCEPLIBS-bufr/b/test/intest1_4+0x205cd)

0x56204eed8c21 is located 0 bytes to the right of global variable '*.LC2' defined in '/home/ed/NCEPLIBS-bufr/src/wrdlen.F' (0x56204eed8c20) of size 1
SUMMARY: AddressSanitizer: global-buffer-overflow /home/ed/NCEPLIBS-bufr/src/iupm.f:53 in iupm_
Shadow bytes around the buggy address:
  0x0ac489dd3130: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ac489dd3140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ac489dd3150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ac489dd3160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ac489dd3170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ac489dd3180: 00 00 00 00[01]f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x0ac489dd3190: 00 00 00 00 02 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x0ac489dd31a0: 04 f9 f9 f9 f9 f9 f9 f9 06 f9 f9 f9 f9 f9 f9 f9
  0x0ac489dd31b0: 00 00 00 00 00 00 04 f9 f9 f9 f9 f9 06 f9 f9 f9
  0x0ac489dd31c0: f9 f9 f9 f9 00 00 00 00 00 00 03 f9 f9 f9 f9 f9
  0x0ac489dd31d0: 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 07 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==53343==ABORTING
ed@koko:~/NCEPLIBS-bufr/b/test$ ./intest1_8
 Testing reading IN_1, CRBMG with OPENBF IO = SEC3
=================================================================
==53406==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55868f846c00 at pc 0x55868f7a0d7a bp 0x7ffd3d1788e0 sp 0x7ffd3d1788d0
READ of size 8 at 0x55868f846c00 thread T0
    #0 0x55868f7a0d79 in iupm_ /home/ed/NCEPLIBS-bufr/src/iupm.f:53
    #1 0x55868f77f6f8 in wrdlen_ /home/ed/NCEPLIBS-bufr/src/wrdlen.F:110
    #2 0x55868f7853e3 in cobfl_ /home/ed/NCEPLIBS-bufr/src/cobfl.c:140
    #3 0x55868f75b9f4 in intest1 /home/ed/NCEPLIBS-bufr/test/intest1.F90:33
    #4 0x55868f75c894 in main /home/ed/NCEPLIBS-bufr/test/intest1.F90:105
    #5 0x7f9d54b0c082 in __libc_start_main ../csu/libc-start.c:308
    #6 0x55868f75b5ad in _start (/home/ed/NCEPLIBS-bufr/b/test/intest1_8+0x205ad)

0x55868f846c01 is located 0 bytes to the right of global variable '*.LC2' defined in '/home/ed/NCEPLIBS-bufr/src/wrdlen.F' (0x55868f846c00) of size 1
SUMMARY: AddressSanitizer: global-buffer-overflow /home/ed/NCEPLIBS-bufr/src/iupm.f:53 in iupm_
Shadow bytes around the buggy address:
  0x0ab151f00d30: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab151f00d40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab151f00d50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab151f00d60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab151f00d70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ab151f00d80:[01]f9 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 00 00
  0x0ab151f00d90: 02 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 04 f9 f9 f9
  0x0ab151f00da0: f9 f9 f9 f9 06 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x0ab151f00db0: 00 00 04 f9 f9 f9 f9 f9 06 f9 f9 f9 f9 f9 f9 f9
  0x0ab151f00dc0: 00 00 00 00 00 00 03 f9 f9 f9 f9 f9 00 00 00 00
  0x0ab151f00dd0: 00 f9 f9 f9 f9 f9 f9 f9 07 f9 f9 f9 f9 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==53406==ABORTING
ed@koko:~/NCEPLIBS-bufr/b/test$ ./intest1_d
 Testing reading IN_1, CRBMG with OPENBF IO = SEC3
=================================================================
==53408==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55a683682c20 at pc 0x55a6835dce9b bp 0x7ffe73da4cf0 sp 0x7ffe73da4ce0
READ of size 8 at 0x55a683682c20 thread T0
    #0 0x55a6835dce9a in iupm_ /home/ed/NCEPLIBS-bufr/src/iupm.f:53
    #1 0x55a6835bb819 in wrdlen_ /home/ed/NCEPLIBS-bufr/src/wrdlen.F:110
    #2 0x55a6835c1504 in cobfl_ /home/ed/NCEPLIBS-bufr/src/cobfl.c:140
    #3 0x55a683597a5c in intest1 /home/ed/NCEPLIBS-bufr/test/intest1.F90:33
    #4 0x55a6835989b5 in main /home/ed/NCEPLIBS-bufr/test/intest1.F90:105
    #5 0x7f4d14865082 in __libc_start_main ../csu/libc-start.c:308
    #6 0x55a6835975cd in _start (/home/ed/NCEPLIBS-bufr/b/test/intest1_d+0x205cd)

0x55a683682c21 is located 0 bytes to the right of global variable '*.LC2' defined in '/home/ed/NCEPLIBS-bufr/src/wrdlen.F' (0x55a683682c20) of size 1
SUMMARY: AddressSanitizer: global-buffer-overflow /home/ed/NCEPLIBS-bufr/src/iupm.f:53 in iupm_
Shadow bytes around the buggy address:
  0x0ab5506c8530: f9 f9 f9 f9 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab5506c8540: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab5506c8550: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab5506c8560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ab5506c8570: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0ab5506c8580: 00 00 00 00[01]f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x0ab5506c8590: 00 00 00 00 02 f9 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x0ab5506c85a0: 04 f9 f9 f9 f9 f9 f9 f9 06 f9 f9 f9 f9 f9 f9 f9
  0x0ab5506c85b0: 00 00 00 00 00 00 04 f9 f9 f9 f9 f9 06 f9 f9 f9
  0x0ab5506c85c0: f9 f9 f9 f9 00 00 00 00 00 00 03 f9 f9 f9 f9 f9
  0x0ab5506c85d0: 00 00 00 00 00 f9 f9 f9 f9 f9 f9 f9 07 f9 f9 f9
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==53408==ABORTING
ed@koko:~/NCEPLIBS-bufr/b/test$

So, what is happening here?

The problem code is here:

      RECURSIVE FUNCTION IUPM(CBAY,NBITS) RESULT(IRET)

      USE MODV_IM8B

      COMMON /HRDWRD/ NBYTW,NBITW,IORD(8)

      CHARACTER*128 BORT_STR
      CHARACTER*8   CBAY
      CHARACTER*8   CINT
      DIMENSION     INT(2)
      EQUIVALENCE   (CINT,INT)

C----------------------------------------------------------------------
C----------------------------------------------------------------------

C     Check for I8 integers.

      IF(IM8B) THEN
         IM8B=.FALSE.

         CALL X84(NBITS,MY_NBITS,1)
         IRET = IUPM(CBAY,MY_NBITS)

         IM8B=.TRUE.
         RETURN
      ENDIF

      IF(NBITS.GT.NBITW) GOTO 900

So looks like this is being caused by the common block.

How come I can't find any place where NBITW is set? 

ed@koko:~/NCEPLIBS-bufr/src$ find . -name '*.f'|xargs grep 'NBITW='
ed@koko:~/NCEPLIBS-bufr/src$ find . -name '*.f'|xargs grep 'NBITW ='
jbathegit commented 1 year ago

It's set within wrdlen.F, so if you had used '*.[fF]' instead of '*.f' then it would have showed up.

jack-woollen commented 1 year ago

I think its complaining about the call to iupm in wrdlen IA = IUPM('A',8) Arg1 is defined as char*8 in iupm.

jack-woollen commented 1 year ago

@edwardhartnett @jbathegit see previous comment

jbathegit commented 1 year ago

I'm taking a look now. I'm able to run under GNU with the address sanitizer turned (on per Ed's instructions in #244), so I can reproduce the error.

jack-woollen commented 1 year ago

@jbathegit By using IA = IUPM('A ',8) in wrdlen fixes all but 19 of the problems. However, most of the remaining problems are also in iupm.

jbathegit commented 1 year ago

I'm confused. IA = IUPM('A ',8) is what the wrdlen code contains already, so how does using that "fix" anything?

FWIW, I just defined a new character*8 variable ca8, set ca8(1:1) to 'A', then passed ca8(1:1) into iupm and that fixed the problem. But I do see other address sanitizer errors now popping up in test_3_IN, test_5_IN, test_IN_7, test_OUT_4, test_OUT_5, test_debufr_1, test_debufr_2, test_sinv, and test_readmp. So will need to look at those one at a time...

jack-woollen commented 1 year ago

github truncated what i sent before. it was A followed by 7 blanks. that also works.

jack-woollen commented 1 year ago

the remaining problems are gnarlier.

jbathegit commented 1 year ago

Yeah, I'm looking at the test_3_IN one now, which is crapping out for some reason at the SUBSET = TAG(INODE(LUN)) line in readns...

jack-woollen commented 1 year ago

inode(lun)=0 !

jbathegit commented 1 year ago

yeah, I saw that too, and I think it's because there's never been a prior call to readmg, and therefore never a prior call to cktaba. The test_IN_3 code calls openbf and then does an immediate call to ireadns.

The idea behind ireadns was that it automatically combines the functionality of readmg and readsb, but if readmg hasn't already been called somewhere then we end up with inode(lun) = 0 on that line. So seems like a logic bug.

jack-woollen commented 1 year ago

probably want to set it to blank

jbathegit commented 1 year ago

yeah, I guess, but only if inode(lun) = 0. I don't have a better idea off the top of my head. I'll give that a try.

jbathegit commented 1 year ago

OK, it got past that point, but test_3_IN is still failing now at line 89 of icbfms. Another iupm issue, so I'll try fixing that the same way I fixed wrdlen previously.

jack-woollen commented 1 year ago

bin there done that. next one's a head scratcher.

jbathegit commented 1 year ago

Going forward, how about we divide and conquer here, and also let's switch this conversation over to #244 since we've apparently resolved this particular memory problem in iupm?

jack-woollen commented 1 year ago

ok i'll work from the bottom up