Masking Compositor information how to use

[gersmit]

I have a question about making parts of a scene image transparent. Is it possible to add transparency if, for example, the given temperature is higher or lower than, say, 240 Kelvin? I then want to keep the pixels that are 240 degrees and remove the rest, make them transparent.

@djhoese helped me out a bit already. But what I'd like to know is how can I mix a "Masking Compositor" into my Python script. For example, can I use it just like DayNightCompositor or should I use it differently?

Maybe someone can give me an example.

Thanks in advance for an answer

[pnuu]

Is there something missing from the class documentation that you need? See the documentation here. See also the built-in hrv_severe_storms_maskedcomposite that uses this the preferred way via configuration file here.

[gersmit]

@pnuu thanks for answer. These composites.MaskingCompositor is new for me. Can you tel me how to call the compositor with python, and show the result.

[pnuu]

The documentation I linked above shows how the compositor is used when called directly. The preferred way is to define the composite in the composite config and loaded with scn.load(['name_of_your_composite']).

[gersmit]

@pnuu In /home/pi/.local/lib/python3.7/site-packages/satpy/etc/composites/seviri,yaml I choose composites: --> ct_masked_ir: (compositor: !!python/name:satpy.composites.MaskingCompositor)

In python: os.environ['SATPY_CONFIG_PATH'] = '/home/pi/.local/lib/python3.7/site-packages/satpy/etc/composites/seviri.yaml' scn.load(['ct_masked_ir']) Then I Create an Area Get error --> Traceback (most recent call last): File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 976, in load self._dependency_tree.populate_with_keys(needed_datasets, query) File "/home/pi/.local/lib/python3.7/site-packages/satpy/dependency_tree.py", line 209, in populate_with_keys raise MissingDependencies(unknown_datasets, "Unknown datasets:") satpy.node.MissingDependencies: Unknown datasets: {DataQuery(name='ct'): {DataQuery(name='ct')}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/pi/a-satpy-testen/cursus_seviri.py", line 151, in scn.load(['ct_masked_ir']) File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 978, in load raise KeyError(str(err)) KeyError: "Unknown datasets: {DataQuery(name='ct'): {DataQuery(name='ct')}}"

As i use in the same seviri.yaml file "cloudtop", then it works oke. (compositor: !!python/name:satpy.composites.GenericCompositor) What do i have to doe to let the the satpy.composites.MaskingCompositor working

[pnuu]

For that particular composite in addition to SEVIRI channel data you'll also need the required ct product from NWC SAF. The cloudtop uses plain SEVIRI channels, while ct is a derived product. You can start from the hrv_severe_storms_masked composite I linked and modify the prerequisites and conditions to suit your needs. The first prerequisite is the data to be shown, the second is where the conditions are applied. In your case it seems both are the same. And naturally also adapt the name and standard_name to something more descriptive.

[gersmit]

It is gone be more difficult for me. What is the easyest way to connect ct product van NWC SAF. Do i have to download that and install? I use no HRV. Or get it from the yaml file

[djhoese]

@gersmit, let's take a step back. You originally said:

Is it possible to add transparency if, for example, the given temperature is higher or lower than, say, 240 Kelvin? I then want to keep the pixels that are 240 degrees and remove the rest, make them transparent.

Let's pick one product that you want to do that with. What product do you want to make partial transparent? What product do you want to use to determine the temperature (is it the same as the one you want to make transparent)? You named your composite ct_masked_ir so I assume you want a "cloud top" product to be masked by some IR product? If you need the NWC SAF "ct" product then yes I think you will need the output from that software (@adybbroe or @pnuu may be able to correct me or provide more information).

Next, do you only want to do this in Python code or would doing it in a YAML configuration file be ok? It looks like you tried editing the YAML files that Satpy comes with. This can be easier sometimes, but it is very easy to break things. @pnuu linked to our suggested way of doing this through a custom configuration directory (documented here).

You also do this:

os.environ['SATPY_CONFIG_PATH'] = '/home/pi/.local/lib/python3.7/site-packages/satpy/etc/composites/seviri.yaml'

There is no need to do this UNLESS you are creating your own custom configuration directory. You are pointing to the builtin satpy composites which are already included and you are pointing to a single file. You should be pointing to the etc/ directory only.

[pnuu]

ct_masked_ir is an existing SEVIRI composite, which uses NWC SAF Cloud Type level 2 product to mask infrared (10.8 um) data.

Taking the initial post and assuming masking with the IR data it self is sufficient, and that brightness temperatures above 240 Kelvin should be transparent, something like this could work:

  ir_masked_ir_240K:
    compositor: !!python/name:satpy.composites.MaskingCompositor
    prerequisites:
    # This data will be shown
    - name: IR_108
    # Data used in masking
    - name: IR_108
    standard_name: ir_masked_ir_240K
    conditions:
    # Data will be masked where brightness temperature is higher than 240 Kelvin
    - method: greater
      value: 240
      transparency: 100

Placing this in your own $SATPY_CONFIG_PATH/composites/seviri.yaml and doing scn.load(['ir_masked_ir_240K']) should give you a starting point. Note that the default enhancement of linear stretch will be applied if you don't add your own enhancement to $SATPY_CONFIG_PATH/enhancements/seviri.yaml with the same standard_name. See the documentation and built-in enhancements for inspiration if the linear stretch isn't what you want.

[gersmit]

@djhoese, @pnuu , Message to both, it will work as you have explained. Was it that I was going to edit the pixels with a Python routine, now can do in an easier way with Satpy. Thank you very much for the clear explanation. 👍

Small question next: Can I edit the result of the Mask now with color like I do with BackgroundCompositor('colorized_ir_clouds') or are there other composites for that ??

[pnuu]

Just replace the first IR_108 prerequisite in the MaskingCompositor example above with colorized_ir_clouds and that's it.

[gersmit]

@pnuu Thanks for the explanation, it works very nicely. It is often the case that you just have to find the beginning of how to deal with certain things.

I think with this explanation you will make many more amateurs happy who would also like to experiment in this area but do not immediately know the entrance.

I will continue to achieve my goal of filtering clouds by temperature.

Thanks from @gersmit

[gersmit]

@pnuu , small problem: Everything works fine a day a go. Now i have change something and it wil not work anymore. I see that is is busy bot no mask is coming.

Also the composite Cloudtop is not working anymore.

Maybe you can tell me what i did wrong. Thanks in advance

[pnuu]

Impossible to say if you don't show what you changed..

[gersmit]

I wrote to new file and use this for testing fast to see if it works and then build it out: def coor_box(lo, ln, ro, rn): global scn_cropped scn_cropped = scn.crop(ll_bbox=(lo, ln, ro, rn))

os.environ['SATPY_CONFIG_PATH'] = '/home/pi/.local/lib/python3.7/site-packages/satpy/etc/composites/seviri.yaml'

scn.load(['ir_masked_ir_240K'])

Europe

coor_box(-10,32,25,58) local_scn = scn_cropped

local_scn.show('IR_108')

[djhoese]

In your scn you load ir_masked_ir_240K but then in your .show you use IR_108. Is that what you want?

When it worked previously did you need to do any resampling?

[pnuu]

In addition you are pointing SATPY_CONFIG_PATH to a single file while it should point to a directory, and the said file is one of the built-in composite configs so it's already available to Satpy.

[gersmit]

@pnuu also @djhoese say that but no difference.

Earlier I did it like this in consultation with @pnuu . I actually see the holes appearing in IR_108. Later I want to give color and @pnuu indicates, use colorized_ir_clouds, and that worked.

I extended the script with areas_def etc and saw that at one point it stopped working properly. Created a new file with a short content of compositor ir_masked_ir_240k as I showed.

First I was also able to test with compositor cloudtop but that no longer works either.

I did copy seviri.yaml to enhancements in between to see if that also worked. (I read something about this)

If I make a text error in seviri.yaml I get an error. So I can see the yaml is called but returns no result.

Strange thing, all the more so it worked fine yesterday.

Is it that maybe i have to call "from satpy import ** "??

[gersmit]

@djhoese I am trying with a new image on my Raspberry pi to make a new Satpy etc. to see if that is my problem. When i install Satpy, get message ERROR: Minimum supported proj version is 6.2.0, installed version is 4.9.3. For more information see: https://pyproj4.github.io/pyproj/stable/installation.htm -- > How can i install a new Proj

When i run the Python script i get on two RPI Error, i never hat before: Traceback (most recent call last): File "/home/pi/a-satpy-testen/cursus3_seviri.py", line 17, in scn.load(scn.all_dataset_names()[1:]) File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 1163, in load self._read_datasets_from_storage(kwargs) File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 1183, in _read_datasets_from_storage return self._read_dataset_nodes_from_storage(nodes, kwargs) File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 1189, in _read_dataset_nodes_from_storage loaded_datasets = self._load_datasets_by_readers(reader_datasets, kwargs) File "/home/pi/.local/lib/python3.7/site-packages/satpy/scene.py", line 1214, in _load_datasets_by_readers new_datasets = reader_instance.load(ds_ids, kwargs) File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 945, in load ds = self._load_dataset_with_area(dsid, coords, kwargs) File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 1109, in _load_dataset_with_area ds = super(GEOFlippableFileYAMLReader, self)._load_dataset_with_area(dsid, coords, kwargs) File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 841, in _load_dataset_with_area ds = self._load_dataset_data(file_handlers, dsid, kwargs) File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 713, in _load_dataset_data proj = self._load_dataset(dsid, ds_info, file_handlers, kwargs) File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/yaml_reader.py", line 689, in _load_dataset projectable = fh.get_dataset(dsid, ds_info) File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/seviri_l1b_native.py", line 458, in get_dataset self._update_attrs(dataset, dataset_info) File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/seviri_l1b_native.py", line 582, in _update_attrs actual_lon, actual_lat, actual_alt = self.satpos File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/seviri_l1b_native.py", line 610, in satpos semi_minor_axis=self.mda['projection_parameters']['b'] File "/home/pi/.local/lib/python3.7/site-packages/satpy/readers/seviri_base.py", line 703, in get_satpos proj='geocent', a=semi_major_axis, b=semi_minor_axis, units='m' File "/home/pi/.local/lib/python3.7/site-packages/pyproj/crs.py", line 436, in init super(CRS, self).init(projstring) File "pyproj/_crs.pyx", line 1738, in pyproj._crs._CRS.init pyproj.exceptions.CRSError: Invalid projection: +proj=geocent +a=6378169.0 +b=6356583.800000001 +units=m +type=crs: (Internal Proj Error: proj_create: Invalid b value)

Can you explaine to me the best way to install the programs and/or is there maybe a image for Raspberry Pi complete to use?

[djhoese]

How did you install Satpy? I can't tell you what to change without knowing what you are already doing.

This is not a Satpy issue this is a pyproj/PROJ issue. It even provides a URL for you on information on the different ways it can be installed. You need a newer version of the PROJ C library. We typically recommend people install packages through conda via packages on the conda-forge channel (see our installation instructions). I'm not sure if that will work on an rpi though.

[gersmit]

Can I use as wel x32 as x64 image? Strange, it allways works fine and now everything is against me 😂 I am going to find out, thanks and until later

[djhoese]

You could try downgrading your version of pyproj to =2.2 (which is the lowest version of pyproj we support in satpy, pyresample might require 2.3). That version should have a slightly lower requirement for PROJ but 4.9.3 (what the error message says you have) is still pretty old.

Regarding 32-bit or 64-bit, I would say if you can do 64-bit then you should. Note as I said before this has nothing to do with Satpy. Satpy is a pure python library. The error you are getting is a compatibility issue with the version of pyproj you are installing and the version of PROJ (a C library) that you have installed. You could do a simple from pyproj import CRS which does not use Satpy at all and it should fail in the same way the complicated Satpy script does. You may want to contact pyproj or check their installation instructions, the ones they linked to in the error message, to see if they mention anything about rpi.

[gersmit]

On 6 august 2021 i wrote that : Everything works fine a day a go. Now i have change something and it wil not work anymore. I see that is is busy bot no mask is coming.

Now the masking is working again but cannot get colors by 'colorized_ir_clouds' I have 'colorized_ir_clouds' on the 5th row **ir_masked_ir_240K: compositor: !!python/name:satpy.composites.MaskingCompositor prerequisites:

This data will be shown

- name: colorized_ir_clouds
# Data used in masking
- name: IR_108**

What is the problem with this, can you explane that and how can i otherwise give color to masked image?

Thanks for helping me out.

[djhoese]

What are the two images you've put here? What does colorized_ir_clouds look like if you load it by itself?

[gersmit]

above image = IR_108, the lowest image = composite 'ir_masked_ir_240K' ( the composite from @pnuu ) the attachement = load_and_show_colorized_ir_clouds. This image show to match i think