search

NOAA-EMC / bufr-query

Apache License 2.0
1 stars 2 forks source link

Print Queries not showing all subsets for some merged files. #11

Closed rmclaren closed 1 month ago

rmclaren commented 2 months ago

[Ronald.Mclaren@hfe07 build]$ ./bin/show_queries.x /scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/task/T2B/bufr2ioda5/SNOMAD/gdas.20240708/gdas.t12z.snomad.tm00.bufr_d Available subset variants: NC000015 NC255161 Total number of subset variants found: 2

NC000015 Dimensioning Sub-paths: 1d *

Queries: 1d int NC000015/WGOSIDS WIGOS IDENTIFIER SERIES 1d int NC000015/WGOSISID WIGOS ISSUER OF IDENTIFIER 1d int NC000015/WGOSISNM WIGOS ISSUE NUMBER 1d string NC000015/WGOSLID WIGOS LOCAL IDENTIFIER (CHARACTER) 1d int NC000015/STID State identifier 1d int NC000015/NSID National station number 1d string NC000015/LSTN Long station or site name 1d int NC000015/TOST TYPE OF STATION 1d int NC000015/YEAR YEAR 1d int NC000015/MNTH MONTH 1d int NC000015/DAYS DAY 1d int NC000015/HOUR HOUR 1d int NC000015/MINU MINUTES 1d float NC000015/CLATH LATITUDE (HIGH ACCURACY) 1d float NC000015/CLONH LONGITUDE (HIGH ACCURACY) 1d float NC000015/HSMSL HEIGHT OF STATION GROUND ABOVE MSL 1d float NC000015/HSALG[1] HEIGHT OF SENSOR ABOVE LOCAL GROUND 1d float NC000015/TMDB TEMPERATURE/DRY BULB TEMPERATURE 1d float NC000015/HSALG[2] HEIGHT OF SENSOR ABOVE LOCAL GROUND 1d int NC000015/MODM METHOD OF SNOW DEPTH MEASUREMENT 1d int NC000015/SOGR STATE OF THE GROUND 1d float NC000015/TOSD TOTAL SNOW DEPTH 1d int NC000015/SDEN Snow density (liquid water content) 1d int NC000015/MSWEM Method of snow water equivalent measurement 1d int NC000015/SWEMS Snow water equivalent 1d string NC000015/SEQNUM CHANNEL SEQUENCE NUMBER 1d string NC000015/BUHD BULLETIN BEING MONITORED (TTAAii) 1d string NC000015/BORG BULLETIN BEING MONITORED (CCCC) 1d string NC000015/BULTIM BULLETIN BEING MONITORED (YYGGgg) 1d string NC000015/BBB BULLETIN BEING MONITORED (BBB) 1d int NC000015/RCTS RECEIPT TIME SIGNIFICANCE 1d int NC000015/RCYR YEAR - TIME OF RECEIPT 1d int NC000015/RCMO MONTH - TIME OF RECEIPT 1d int NC000015/RCDY DAY - TIME OF RECEIPT 1d int NC000015/RCHR HOUR - TIME OF RECEIPT 1d int NC000015/RCMI MINUTE - TIME OF RECEIPT 1d int NC000015/CORN CORRECTED REPORT INDICATOR

NC255161 Dimensioning Sub-paths:

Queries: [hfe07:2498671:0:2498671] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x8) ==== backtrace (tid:2498671) ==== 0 /apps/spack/linux-rocky8-x86_64/gcc-9.2.0/ucx-1.14.1-5npvdbhw4fuc7pfjfdgptrhacdofefiu/lib/libucs.so.0(ucs_handle_error+0x294) [0x7f4c71377b04] 1 /apps/spack/linux-rocky8-x86_64/gcc-9.2.0/ucx-1.14.1-5npvdbhw4fuc7pfjfdgptrhacdofefiu/lib/libucs.so.0(+0x2ecbf) [0x7f4c71377cbf] 2 /apps/spack/linux-rocky8-x86_64/gcc-9.2.0/ucx-1.14.1-5npvdbhw4fuc7pfjfdgptrhacdofefiu/lib/libucs.so.0(+0x2efc4) [0x7f4c71377fc4] 3 /lib64/libpthread.so.0(+0x12cf0) [0x7f4c74f94cf0] 4 /apps/spack/linux-rocky8-x86_64/gcc-8.5.0/gcc-9.2.0-ku6r4f5qa5obpfnqpa6pezhogxq6sp7h/lib64/libstdc++.so.6(_ZNKSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEE6lengthEv+0) [0x7f4c745a30a0] 5 ./bin/show_queries.x() [0x428fc4] 6 ./bin/show_queries.x() [0x428280] 7 ./bin/show_queries.x() [0x423a4c] 8 /lib64/libc.so.6(__libc_start_main+0xe5) [0x7f4c73b3ed85] 9 ./bin/show_queries.x() [0x4233ae]

Segmentation fault (core dumped)

rmclaren commented 1 month ago

Fixed in latest additions to feature/in_parallel branch.

jiaruidong2017 commented 1 month ago

Thanks @rmclaren for your efforts. Would you please let me know how to build the bufr-query package to get the show_queries.x for a test?

jiaruidong2017 commented 1 month ago

@rmclaren I built it successfully. Thanks.

jiaruidong2017 commented 1 month ago

show_queries.x works perfect. Can the bufr2ioda.x convert the merged bufr file (gdas.t12z.snomad.tm00.bufr_d) into the ioda format?

rmclaren commented 1 month ago

@jiaruidong2017 In the context of bufr-query you would use bufr2netcdf.x which is equivalent to bufr2ioda.x (works the same way, slightly different YAML file). In the future IODA will be able to read BUFR directly as this library is becoming a backend of this library.

rmclaren commented 1 month ago

@jiaruidong2017 I guess I should have said that you can read BUFR directly with IODA with the develop branch of IODA (right now). I'm working on parallelizing this (via MPI)...

jiaruidong2017 commented 1 month ago

Thanks @rmclaren for your updates. Do you have a built bufr2netcdf.x available for me to use?

rmclaren commented 1 month ago

Are you using the latest version of the GDAS App by any chance (develop)?

jiaruidong2017 commented 1 month ago

Okay, I will try to build the GDASApp for the bufr2netcdf.x. Thanks.

rmclaren commented 1 month ago

It shoould end up in the build/bin or <install>/bin directory along with show_queries.x.

jiaruidong2017 commented 1 month ago

I just built the gdasapp package, and checked both show_queries.x and bufr2netcdf.x. But, they all failed as below:

/scratch2/NCEPDEV/stmp1/Jiarui.Dong/gdasapp/gdasapp.9/build/bin/show_queries.x input_synop.bufr_d


Available subset variants:
NC000015
NC255161
Total number of subset variants found: 2

NC000015 Dimensioning Sub-paths: 1d *

Queries: 1d int NC000015/WGOSIDS WIGOS IDENTIFIER SERIES 1d int NC000015/WGOSISID WIGOS ISSUER OF IDENTIFIER 1d int NC000015/WGOSISNM WIGOS ISSUE NUMBER 1d string NC000015/WGOSLID WIGOS LOCAL IDENTIFIER (CHARACTER) 1d int NC000015/STID State identifier 1d int NC000015/NSID National station number 1d string NC000015/LSTN Long station or site name 1d int NC000015/TOST TYPE OF STATION 1d int NC000015/YEAR YEAR 1d int NC000015/MNTH MONTH 1d int NC000015/DAYS DAY 1d int NC000015/HOUR HOUR 1d int NC000015/MINU MINUTES 1d float NC000015/CLATH LATITUDE (HIGH ACCURACY) 1d float NC000015/CLONH LONGITUDE (HIGH ACCURACY) 1d float NC000015/HSMSL HEIGHT OF STATION GROUND ABOVE MSL 1d float NC000015/HSALG[1] HEIGHT OF SENSOR ABOVE LOCAL GROUND 1d float NC000015/TMDB TEMPERATURE/DRY BULB TEMPERATURE 1d float NC000015/HSALG[2] HEIGHT OF SENSOR ABOVE LOCAL GROUND 1d int NC000015/MODM METHOD OF SNOW DEPTH MEASUREMENT 1d int NC000015/SOGR STATE OF THE GROUND 1d float NC000015/TOSD TOTAL SNOW DEPTH 1d int NC000015/SDEN Snow density (liquid water content) 1d int NC000015/MSWEM Method of snow water equivalent measurement 1d int NC000015/SWEMS Snow water equivalent 1d string NC000015/SEQNUM CHANNEL SEQUENCE NUMBER 1d string NC000015/BUHD BULLETIN BEING MONITORED (TTAAii) 1d string NC000015/BORG BULLETIN BEING MONITORED (CCCC) 1d string NC000015/BULTIM BULLETIN BEING MONITORED (YYGGgg) 1d string NC000015/BBB BULLETIN BEING MONITORED (BBB) 1d int NC000015/RCTS RECEIPT TIME SIGNIFICANCE 1d int NC000015/RCYR YEAR - TIME OF RECEIPT 1d int NC000015/RCMO MONTH - TIME OF RECEIPT 1d int NC000015/RCDY DAY - TIME OF RECEIPT 1d int NC000015/RCHR HOUR - TIME OF RECEIPT 1d int NC000015/RCMI MINUTE - TIME OF RECEIPT 1d int NC000015/CORN CORRECTED REPORT INDICATOR

NC255161 Dimensioning Sub-paths:

Queries: Segmentation fault (core dumped)


I think the ioda (hash: b8281e5a with latest update at July 11th) in the current gdasapp is not updated yet.
rmclaren commented 1 month ago

Yes, the fix above has not been applied to the develop branch yet, so I wouldn't expect it to work yet. The fix is part of the feature/in_parallel branch which has not been merged yet..

rmclaren commented 1 month ago

You could try it with /scratch1/NCEPDEV/da/Ronald.McLaren/src/bufr-query/build/bin/show_queries.x