search

NOAA-EMC / NCEPLIBS-bufr

The NCEPLIBS-bufr library contains routines and utilites for working with the WMO BUFR format.
Other
44 stars 19 forks source link

adding test for ufbcup #461

Closed edwardhartnett closed 1 year ago

edwardhartnett commented 1 year ago

Part of #445 Fixes #344

Adding test for ufbcup().

edwardhartnett commented 1 year ago

@jbathegit this code:

  ! Open input file.
  open(unit = 11, file = 'testfiles/IN_9', form ='unformatted')
  call openbf(11, 'IN', 11)

  ! Open output file.
  open(unit = 12, file = 'test_ufbcup_out', form ='unformatted')
  call openbf(12, 'IN', 12)

  call readmg(11, subset, jdate, iret)
  print *, subset, jdate, iret
  call ufbcup(11, 12)

Results in this error message:

 ***********BUFR ARCHIVE LIBRARY ABORT**************
 BUFRLIB: UFBCUP - LOCATION OF INTERNAL TABLE FOR INPUT BUFR FILE DOES NOT AGREE WITH EXPECTED LOCATION IN INTERNAL SUBSET ARRAY
 ***********BUFR ARCHIVE LIBRARY ABORT**************

What am I doing wrong? How do I get the right location of internal table? What is this error trying to tell me?

edwardhartnett commented 1 year ago

@jbathegit if you can help me understand what I am doing wrong in this test I can fix it...

jbathegit commented 1 year ago

Please remember you can also reach out to @jack-woollen on these types of questions, in case I'm tied up with other stuff. In fact, for this particular routine, it might be better for him to chime in anyway. ;-)

That said, and off the top of my head, I don't see anything wrong with what you're doing in your openbf call, since 'testfiles/IN_9' does contain an embedded DX BUFR table. However, in addition to calling readmg (to read in a BUFR message), I believe you also need to call readsb (to read in a data subset) before you can call ufbcup.

edwardhartnett commented 1 year ago

OK, getting further!

Now I am hitting this error:

 ***********BUFR ARCHIVE LIBRARY ABORT**************
 BUFRLIB: UFBCUP - A MESSAGE MUST BE OPEN IN OUTPUT BUFR FILE, NONE ARE
 ***********BUFR ARCHIVE LIBRARY ABORT**************

How do I open a message in the output file? THat's my next problem...

edwardhartnett commented 1 year ago

OK, now my code seems to succeed! But how do I check that it did?

  ! Open input file.
  open(unit = 11, file = 'testfiles/IN_9', form ='unformatted')
  call openbf(11, 'IN', 11)

  ! Open output file.
  open(unit = 12, file = 'test_ufbcup_out', form ='unformatted')
  call openbf(12, 'OUT', 11)

  ! Read a message.
  call readmg(11, subset, jdate, iret)
  if (iret .ne. 0 .or. subset .ne. 'ADPSFC' .or. jdate .ne. 23022519) stop 10

  ! Load a subset of data.
  call readsb(11, iret)
  if (iret .ne. 0) stop 20

  ! Create a new message in the output file.
  call openmg(12, subset, jdate)

  ! Call ufbcup.
  call ufbcup(11, 12)

Are there some subroutines I can call after ufbcup() that would confirm that it worked?

jack-woollen commented 1 year ago

ufdump

edwardhartnett commented 1 year ago

When I try ufdump() I get:

***********BUFR ARCHIVE LIBRARY ABORT**************
 BUFRLIB: UFDUMP - INPUT BUFR FILE IS OPEN FOR OUTPUT, IT MUST BE OPEN FOR INPUT
 ***********BUFR ARCHIVE LIBRARY ABORT**************
edwardhartnett commented 1 year ago

OK, now I'm doing this:

  ! Now reopen the file as input.
  open(unit = 12, file = 'test_ufbcup_out', form ='unformatted')
  call openbf(12, 'IN', 12)

  ! Read a message in the file.
  call readmg(12, subset, jdate, iret)
  if (iret .ne. 0 .or. subset .ne. 'ADPSFC' .or. jdate .ne. 23022519) stop 10

  ! Dump the data subset.
  call ufdump(12,0)

  ! Close the file again.
  call closbf(12)

And I get this output:

MESSAGE TYPE ADPSFC    

001192  SID                         CWGN  ( 8)CCITT IA5                 TABLE B ENTRY - STATION IDENTIFICATION          
006002  XOB                       262.43  DEG E                         TABLE B ENTRY - LONGITUDE                       
005002  YOB                        49.03  DEG N                         TABLE B ENTRY - LATITUDE                        
004192  DHR                         0.00  HOURS                         TABLE B ENTRY - OBSERVATION TIME MINUS CYCLE TIM
010194  ELV                        251.0  METER                         TABLE B ENTRY - STATION ELEVATION               
001193  TYP                        284.0  CODE TABLE                    TABLE B ENTRY - OI/SSI REPORT TYPE              
055006  T29                        512.0  CODE TABLE                    TABLE B ENTRY - NMC OFFICE NOTE 29 REPORT TYPE  
002001  ITP                      MISSING  CODE TABLE                    TABLE B ENTRY - INSTRUMENT TYPE                 
           {PLEVL}     2 REPLICATIONS
    ++++++  PLEVL  REPLICATION #     1  ++++++
001194  CAT                          0.0  CODE TABLE                    TABLE B ENTRY - NMC OFFICE NOTE 29 CATEGORY     
007192  PRC                        980.6  MB                            TABLE B ENTRY - PRESSURE COORDINATE             
010198  PQM                          3.0  CODE TABLE                    TABLE B ENTRY - PRESSURE (QUALITY) MARKER       
013193  QQM                          3.0  CODE TABLE                    TABLE B ENTRY - SPECIFIC HUMIDITY (QUALITY) MARK
012195  TQM                          2.0  CODE TABLE                    TABLE B ENTRY - REPORTED TEMPERATURE (QUALITY) M
010196  ZQM                          3.0  CODE TABLE                    TABLE B ENTRY - HEIGHT (QUALITY) MARKER         
011192  WQM                          2.0  CODE TABLE                    TABLE B ENTRY - WIND (QUALITY) MARKER           
 (<enter> for MORE, q <enter> to QUIT)

           [OBLVL]     1 REPLICATIONS
001198  SRC                          PRP  ( 3)CCITT IA5                 TABLE B ENTRY - FILE NAME OF DATA SOURCE        
001200  FHR                      MISSING  HOUR                          TABLE B ENTRY - FORECAST LENGTH                 
           <PEVN>     1 REPLICATIONS
010197  POB                        980.6  MB                            TABLE B ENTRY - (SURFACE) PRESSURE OBSERVATION  
010192  PMO                       1014.0  MB                            TABLE B ENTRY - MEAN SEA-LEVEL PRESSURE OBSERVAT
           <QEVN>     1 REPLICATIONS
013192  QOB                        664.0  MG/KG                         TABLE B ENTRY - SPECIFIC HUMIDITY OBSERVATION   
           <TEVN>     1 REPLICATIONS
012192  TOB                        -16.0  DEG C                         TABLE B ENTRY - REPORTED TEMPERATURE OBSERVATION
           <ZEVN>     1 REPLICATIONS
010195  ZOB                        251.0  METER                         TABLE B ENTRY - HEIGHT OBSERVATION              
           <WEVN>     1 REPLICATIONS
011003  UOB                          4.8  M/S                           TABLE B ENTRY - U-COMPONENT WIND OBSERVATION    
 (<enter> for MORE, q <enter> to QUIT)

011004  VOB                         -1.7  M/S                           TABLE B ENTRY - V-COMPONENT WIND OBSERVATION    
           <CEVN>     0 REPLICATIONS
           <SEVN>     1 REPLICATIONS
020001  HOVI                     MISSING  METER                         TABLE B ENTRY - HORIZONTAL VISIBILITY           
012111  MXTM                     MISSING  KELVIN                        MAXIMUM TEMPERATURE                             
012112  MITM                     MISSING  KELVIN                        MINIMUM TEMPERATURE                             
012244  TDO                      MISSING  DEG C                         DEWPOINT TEMPERATURE OBSERVATION (NOT ASSIMILATE
020010  TOCC                     MISSING  PERCENT                       CLOUD COVER (TOTAL)                             
011041  MXGS                     MISSING  M/S                           MAXIMUM WIND SPEED (GUSTS)                      
012245  THI                      MISSING  DEG F                         HEAT INDEX                                      
012246  TCH                      MISSING  DEG F                         WIND CHILL                                      
050197  CDBZ                     MISSING  METER                                                                         
050217  CEIL                     MISSING  METER                         CEILING                                         
           <SBT>     0 REPLICATIONS
           <AODSEQ>     0 REPLICATIONS
    ++++++  PLEVL  REPLICATION #     2  ++++++
001194  CAT                          0.0  CODE TABLE                    TABLE B ENTRY - NMC OFFICE NOTE 29 CATEGORY     
007192  PRC                        980.6  MB                            TABLE B ENTRY - PRESSURE COORDINATE             
 (<enter> for MORE, q <enter> to QUIT)

010198  PQM                          3.0  CODE TABLE                    TABLE B ENTRY - PRESSURE (QUALITY) MARKER       
013193  QQM                          3.0  CODE TABLE                    TABLE B ENTRY - SPECIFIC HUMIDITY (QUALITY) MARK
012195  TQM                          2.0  CODE TABLE                    TABLE B ENTRY - REPORTED TEMPERATURE (QUALITY) M
010196  ZQM                          3.0  CODE TABLE                    TABLE B ENTRY - HEIGHT (QUALITY) MARKER         
011192  WQM                          2.0  CODE TABLE                    TABLE B ENTRY - WIND (QUALITY) MARKER           
           [OBLVL]     1 REPLICATIONS
001198  SRC                          PRP  ( 3)CCITT IA5                 TABLE B ENTRY - FILE NAME OF DATA SOURCE        
001200  FHR                      MISSING  HOUR                          TABLE B ENTRY - FORECAST LENGTH                 
           <PEVN>     1 REPLICATIONS
010197  POB                        980.6  MB                            TABLE B ENTRY - (SURFACE) PRESSURE OBSERVATION  
010192  PMO                       1014.0  MB                            TABLE B ENTRY - MEAN SEA-LEVEL PRESSURE OBSERVAT
           <QEVN>     1 REPLICATIONS
013192  QOB                        664.0  MG/KG                         TABLE B ENTRY - SPECIFIC HUMIDITY OBSERVATION   
           <TEVN>     1 REPLICATIONS
012192  TOB                        -16.0  DEG C                         TABLE B ENTRY - REPORTED TEMPERATURE OBSERVATION
           <ZEVN>     1 REPLICATIONS
 (<enter> for MORE, q <enter> to QUIT)

010195  ZOB                        251.0  METER                         TABLE B ENTRY - HEIGHT OBSERVATION              
           <WEVN>     1 REPLICATIONS
011003  UOB                          4.8  M/S                           TABLE B ENTRY - U-COMPONENT WIND OBSERVATION    
011004  VOB                         -1.7  M/S                           TABLE B ENTRY - V-COMPONENT WIND OBSERVATION    
           <CEVN>     0 REPLICATIONS
           <SEVN>     1 REPLICATIONS
020001  HOVI                     MISSING  METER                         TABLE B ENTRY - HORIZONTAL VISIBILITY           
012111  MXTM                     MISSING  KELVIN                        MAXIMUM TEMPERATURE                             
012112  MITM                     MISSING  KELVIN                        MINIMUM TEMPERATURE                             
012244  TDO                        -22.0  DEG C                         DEWPOINT TEMPERATURE OBSERVATION (NOT ASSIMILATE
020010  TOCC                     MISSING  PERCENT                       CLOUD COVER (TOTAL)                             
011041  MXGS                     MISSING  M/S                           MAXIMUM WIND SPEED (GUSTS)                      
012245  THI                      MISSING  DEG F                         HEAT INDEX                                      
012246  TCH                      MISSING  DEG F                         WIND CHILL                                      
050197  CDBZ                     MISSING  METER                                                                         
050217  CEIL                     MISSING  METER                         CEILING                                         
           <SBT>     0 REPLICATIONS
 (<enter> for MORE, q <enter> to QUIT)

           <AODSEQ>     0 REPLICATIONS

 >>> END OF SUBSET <<<
edwardhartnett commented 1 year ago

Problem is, that's hard to use in a test. I need some subroutines that give me some of the values as parameters, so I can easily check values...

edwardhartnett commented 1 year ago

Now I have this code for reading the file created by ufbcup():

  ! Now reopen the file as input.
  open(unit = 12, file = 'test_ufbcup_out', form ='unformatted')
  call openbf(12, 'IN', 12)

  ! Read a message in the file.
  call readmg(12, subset, jdate, iret)
  if (iret .ne. 0 .or. subset .ne. 'ADPSFC' .or. jdate .ne. 23022519) stop 10

  ! Load a subset of data.
  call readsb(12, iret)
  if (iret .ne. 0) stop 30

This fails here in readsb():


C  SEE IF THERE IS ANOTHER SUBSET IN THE MESSAGE
C  ---------------------------------------------

      IF(NSUB(LUN).EQ.MSUB(LUN)) THEN
         IRET = -1
         GOTO 100

@jbathegit or @jack-woollen when I create a file with ufbcup() shouldn't I be able to do a readmg() and then a readsb() on that file?

What am I doing wrong?

jack-woollen commented 1 year ago

This is actually quite a complicated routine in terms of what you might use it for. It is meant to copy data from one BUFR file into another. The rub is they don't have to have the same table definitions. You need to enter with a message open in each file, one read one write, but not necessarily the same message type or element definitions. A subset must have been read in the input, so there is data in the input subset buffer. You may or may not have data already in the output subset buffer. It will scan the tags (mnemonics) in each subset and when it finds a matching tag it transfers the value from the input subset buffer, into the same tag in the output subset buffer. You might want to do this when an updated definition for some type of data has been made by adding new elements on top of an older table format, and you have data stored in the older table format you want to copy data directly into the new format. Even the tags in the two files could have different definitions (number of bits, range, etc). Its a niche process without wide application, but it will do things impossible any other way. And it makes a huge number of those impossible things possible. The current code tests the simplest case which is direct copy between identical message types and element definitions. That is a simple and direct test where the results can be completely verified by comparing ufdump outputs of subsets from each file. That checking can be done in a similar way the ufbrw test is checked, internally in the test code. Tests which use different tables sre more complicated to stage, and maybe more difficult to verify. My sense is the simple test covers the basics and in some ways is a superset of any other of the possible tests. I do think the documentation needs a little more information.

jack-woollen commented 1 year ago

@edwardhartnett In your test you need to call writsb after ufbcup before closing the file.

edwardhartnett commented 1 year ago

OK, I am having loads of fun with this test! I feel like I'm starting to get the hang of this library, at least a little bit.

Here's my next question. I tried to get the number of messages in the file, and I got an unreasonably large number. @jack-woollen and @jbathegit shouldn't calling ufbint() with 'IREC' get the number of messages in the file?

Here's the relevant code:

  ! Get IREC, the number of BUFR messages, according to ufbint()
  ! documentation. But I get a very large number of messages!
  call ufbint(12, hdr, 1, 1, iret, 'IREC')
  print *, hdr(1,1)
  if (hdr(1,1) .ne. 100000000000.00000_8) stop 30

It's telling me there are 100000000000 messages in the file. That can't be right!

jack-woollen commented 1 year ago

if you call openbf(0,'QUIET',0) it will do nothing but alloc the arrays

edwardhartnett commented 1 year ago

How do I get the SID?

I have this code:

  ! Get SID, the station ID.
  call ufbint(12, hdr, 1, 1, iret, 'SID')
  print *, hdr(1,1)

This results in:

 Testing ufbcup.
   6.0134710428730278E-154
 SUCCESS
jack-woollen commented 1 year ago

The SID returns a char(8). You can print hdr(1) with A format or equivalence hdr(1) to a char(8) variable. On most systems you could just define hdr as char(8), but maybe not all.

edwardhartnett commented 1 year ago

@jack-woollen can you answer my question above about the IREC value of 100000000000?

edwardhartnett commented 1 year ago

OK, I'm having a lot of fun with this test and could keep adding to it, but let's get what is here committed and doing some good, and I can circle around to it again, or continue to explore these functions in new tests.