Open emilyhcliu opened 1 week ago
@emilyhcliu This is a thornier issue than I thought originally. I'm trying to think of a good way to merge data like this in order to keep things consistent. We can talk about the details later.
@rmclaren
One question about all_sub_categories()
yaml_path = './bufr2ioda_mtiasi_mapping.yaml'
input_path= './gdas.t00z.esiasi.tm00.bufr_d'
container = bufr.Parser(input_path, yaml_path).parse()
categories = container.all_sub_categories()
print(categories)
There are three categories for IASI: metop-a, metop-b and ,metop-c
I expect that the following output from print(categories)
[ 'metop-a', 'metop-b', 'metop-c']
But, I got the following:
[['metop-a'], ['metop-b'], ['metop-c']]
Why do we get lists inside of a list?
This is because you can categorize (split) on several parameters. So for example you could define two splits, satellite ID and hour. In which case you would get a list
[['metop-a', '2'], ['metop-a', '3'], ['metop-a', '4'], ['metop-b', '2'], ['metop-b', '3'], ['metop-b', '4'], etc...]
. So 'metop-a' and '2' are subcatgories. Sets of subcategories ex:['metop-a', '2']
make a category.
@rmclaren Can we add functionality so that users can request a sub-container from a container with categories? For example, my mapping has categories (e.g. goes-16, goes-17, goes-18) container,all_sub_categories = [['goes-16'], ['goes-17'], ['goes-18']]
Can we have some method to break down the container like the following: container1 = container('goes-16') container2 = container('goes-17') container3 = container('goes-18')
Good news.
I tested creating multiple obs spaces from one bufr file for satwind using script backend
and data cache
.
It worked great!!
For the satwind case, the BUFR contains g16 and g17. But, my mapping file defines three categories (g16, g17, g18). So, the output g18 should be empty with headers only. This also worked!!
So, I am closing issue #6 - about creating empty data file with header only for categories in BUFR but defined in the mapping file. Now, I realize that it is a good thing to create an empty data file with headers only under the circumstances described in the issue.
@rmclaren I need your suggestion for the following data conversion case:
For IASI data, there are three BUFR sources, each with its mapping file, but they all map to the same IODA variable fields.
Three containers need to be combined into one container (container = container1 + container2 + container3) The target container dimension for 2D variables is [Location1+Location2+Locaion3, 616)
So, the variables in container2 & container3, which have channels in the dimension, will need to be reorganized (from 500 to 616) The variables that need to be modified are:
variables/sensorChannelNumner
andvariables/spectralRadiance
For container2, I have modified these two variables so that their dimensions changed from [Location2, 500] to [Location2, 616] For container3, I have modified these two variables so that their dimensions changed from [Location3, 500 to [Location3, 616]
I tried
DataContainer.replace
:I got the following error message:
The error message was expected since I was trying to add data with dimension [Location2, 616] to a data path with dimension [Location2, 500].
@rmclaren Do you have any suggestions?