Closed iainrussell closed 1 year ago
Also one can exclude certain keys to speed up the unpacking the data section. See https://confluence.ecmwf.int/display/UDOC/Performance+improvement+by+skipping+some+keys+-+ecCodes+BUFR+FAQ
However, we already do this: https://github.com/ecmwf/pdbufr/blob/master/pdbufr/bufr_structure.py#L630
Hi @iainrussell,
what filter
option did you use for the single message file? That message does not containstationOrSiteName
.
Iterating through all the keys in the single message in Python takes 1.5 s on my Mac. While extracting airTemperature
with pdbufr takes 23 s with this code:
df = pdbufr.read_bufr(f,
columns=("year",'month', "hour", "minute","latitude", "longitude", "airTemperature"),
)
print(df)
I used the same filter, so it should reject it early, but I think that key is not in the header, so it searches the data section for it
I found the reason for the slowness. It is related to the suboptimal access pattern to keys/values. I will try to fix it in a PR.
@shahramn, Do you know what the fastest way is to get both the name and code for all the keys in a message? I know there is codes_get("temperature->code"), but I wonder if there is anything faster.
The slowness is cased buy calling
eccodes.codes_get(key + "->code"]
for each and every key in a message. In the single message case 66560 times!
Good investigative work Sandor!
The following refers to a BUFR file in ECMWF internal file system under my username:
$PERM/public/i0tp_07062020_00.buf
. It contains 364,359 messages and 715,925 subsets. The following call to pdbufr takes around 41 minutes on the HPC:I compared performance by running a pure eccodes-python script to iterate over the messages and unpack each one. That takes around 5 minutes.
I also modified the pdbufr code so as to avoid performing the actual filtering, and then it took around 5 minutes too.
I could see that certain sets of messages in the BUFR file were taking a lot longer to process than others. After putting a little profiling code into pdbufr and printing the indexes of the messages that were taking longer than 0.01 seconds to process, these were some of the message indexes (0-based) that consistently took longer:
Some standout times to process certain messages, which seemed consistent with multiple runs:
I copied a single message into
$PERM/public/one_msg_110773.bfr
(bufr_copy -wcount=110773 i0tp_07062020_00.buf one_msg_110773.bfr
, which is the message at my 0-based 110774 index printouts) and it takes 15 seconds to process by itself with the same pdbufr script. This particular message has what looks like 6632 time periods encoded in it, and the single message occupies almost 140KB on disk.Interestingly, calling bufr_dump on this message took only 0.6 seconds: