Cached overlays are pale

[lobsiger]

Problem description

Cached overlay outline and fill colors are pale. See this thread:

https://groups.google.com/g/pytroll/c/7VVgsmstLkQ

@djhoese for unknown reason the google list deletes all my posts. I found out that the pycoast stored overlay.png is already pale. I made a composite with a clean image of Switzerland and the stored overlay file using ImageMagick. I get exactly the Satpy result for cached overlays. You can see with the bare eye that the cached overlay.png is pale.

[djhoese]

@lobsiger It looks like google groups decided to require approval of your messages...but then also didn't notice any of the owners of the list that they need approval. Would you like me to approve one of the messages? If so, which time?

[lobsiger]

@djhoese what do you mean with "which time"? Do I have to post a message to the google group at some known time? Or can you just approve my first post of this overlay problems thread?

[djhoese]

Google groups shows multiple messages from you with different timestamps:

I assumed you tried to resend the message multiple times.

[lobsiger]

@djhoese I logged in and could post and see my posts. But after logout the posts were deleted. I cleaned my gmail account, changed PW and approved my other e-mail account. Maybe this or some of your help gave me access to google group posts again. Regarding a fix for the pale cache file generation I will be of no help as this code is well over my head :-(. All I found out above is that the saved/cached overlay file is pale already. So the problem seems to be in the generation of this overlay file and not in the later application step.

[mraspaud]

Hi @lobsiger,

This was a long time ago, but I was the one writing this code at this time. From what I remember, the problem with the cache is that it can make no assumption on what color will be under it's coastlines, so it becomes lighter naturally. However, now that I think about it, maybe the way to apply the cached coastlines could be improved eg by multiplying instead of pasting the cache? As you can see here https://github.com/pytroll/pycoast/blob/main/pycoast/cw_base.py#L1215 we use the paste function to put the cache lines on top of the background. Maybe there are better methods for this? for example the documentation of paste seems to hint at using something called alpha_composite instead, maybe that's better? https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.paste

[lobsiger]

@mraspaud TBH I do not really understand why the overlay must know (??"can make make no assumptions"??) on top of what kind of background it will be applied later. If I write e.g. borders with 'outline': (255, 0, 0) and 'outline_opacity': 255 I would expect a cached *.png that is fully transparent except for fully opaque red border lines.

Anyway, maybe Image Magick gives a hint here? To apply file foreground.png on top of file background.png the command is:

convert background.png foreground.png -composite result.png

I use this e.g. to overlay MSLP charts on top of GEO satellite images. These charts have no clue what will be underneath.

[lobsiger]

... sorry, I should have taken the black MSLP chart. If you see all white above or even below (?) click on the invisible image :-).

[mraspaud]

The thing is that when using antialiased lines as drawn with the AGG backend, some pixels of the lines are somewhat transparent, hence the need to take care of alpha blending.

[lobsiger]

@mraspaud O.K. I grasp that writing a fine antialiased line on some background might not be the same thing as writing this line on a transparent canvas (I eliminate antialiasing in my MSLP charts). But what about the fill colors? When these are fully opaque I see no difference. But when I fill e.g. land masses with 'fill': (255,0,0) of 'fill_opacity': 127 I do not understand the difference seen between cached and non cached results below (red in cached image is much more dim).

[lobsiger]

@mraspaud @djhoese maybe this is one more hint: If I generate the above image without cache but 'fill' : (127, 0, 0) and 'fill_opacity': 127 I get (apparently) exactly the same red fill colors as with cached 'fill': (255, 0 ,0) and 'fill_opacity': 127. Is in the cached case the red (255, 0, 0) fill color arithmetically blended with black (0, 0, 0) somehow?

[djhoese]

Hhhmmm just a thought, but when the cache image is first created the base Image is created as all black with full transparency (RGBA=(0, 0, 0, 0)). I wonder if we did it (255, 255, 255, 0) if that would make a difference?

[djhoese]

Here is where the change would go:

https://github.com/pytroll/pycoast/blob/4687a0ea4fa4b3bf630baec907b92cd06a01c5ff/pycoast/cw_base.py#L1178

[lobsiger]

@djhoese I could try that but I do not expect it to work. If my idea of blending above has some truth then the problem is rather how the lines and fill values are written onto this black surface. The involved (0,0,0,0) pixels should be replaced. For pixels not touched (0,0,0,0) would remain as "no_data" value. Fine lines might suffer the antialiasing problem @mraspaud mentioned.

[djhoese]

I won't promise/guarantee that this will fix things, but I've seen something like this in the VisPy library that I maintain. When two colors are blended, especially when transparency is involved, the mix between black (0) and any color is a reduced version of that color, the mix between whitee (255) and any color should at least be the original color. ...this is my hope at least.

[lobsiger]

@djhoese O.K. I tested your proposal with 'fill':(255,0,0) and 'fill_opacity': 127 and overlay cache True. Besides different colors I would point to the fact that the minor grid lines now reappear. The antialiasing mentioned by @mraspaud seems to profit from the base RGBA = (255,255,255,0) white canvas. Pinky fill seems like a blend of white and red?

[djhoese]

Just to make sure when I test this that I'm doing it the same, could you paste the code you're running to generate the main image part of this (the overlay configuration at least)?

[lobsiger]

As a last variant I tested 'fill':(255,0,0) and 'fill_opacity': 127 with overlay cache True starting with a red RBGA = (255, 0, 0, 0) canvas for the overlay file. This basically gives me the original image with no cache ( (red+red)/2=red ??) while the thin minor grid lines now get some red from the @mraspaud mentioned antialiasing. I still think the problem has to do how the colors are applied on the empty base image. No idea whether and where this can be changed/improved.

[lobsiger]

@djhoese I'm not sure this is of any help. I make these images with my module LEOstuff.py that does about everything for LEO sats. I only copied the most relevant overlay parts used.

Area used

eurol10: description: Euro 10.0km area - Europe projection: proj: stere ellps: WGS84 lat_0: 90.0 lon_0: 0.0 lat_ts: 60.0 shape: height: 800 width: 1000 area_extent: lower_left_xy: [-3780000.0, -7644000.0] upper_right_xy: [3900000.0, -1500000.0]

#############################################################################
# sf is a scale factor depending on image size sf = (width + height) / 3000 #
#############################################################################

sf = (1000 + 800) / 3000

my_coasts  = {'outline': (255, 255,   0), 'outline_opacity': 255, 'width': 1.5*sf, 'level': 1, 'fill': (255, 0, 0), 'fill_opacity': 127}
my_borders = {'outline': (255,   0,   0), 'outline_opacity': 255, 'width': 1.0*sf, 'level': 1}
my_rivers  = {'outline': (  0,   0, 255), 'outline_opacity': 255, 'width': 1.0*sf, 'level': 3}

my_grid  = {'major_lonlat': (10,10), 'minor_lonlat': (2, 2),
            # Opacity 0 will hide the line, values 0 ... 255  EL
            'outline': (255, 255, 255) , 'outline_opacity': 255,
            'minor_outline': (200, 200, 200),'minor_outline_opacity': 127,
            'width': 1.5*sf, 'minor_width': 1.0*sf, 'minor_is_tick': False,
            'write_text': True, 'lat_placement': 'lr', 'lon_placement': 'b',
            'font': fontdir + '/DejaVuSerif.ttf',
            'fill': 'white', 'fill_opacity': 255, 'font_size': 30*sf}
            # minor_is_tick: False draws a line not ticks, True does ticks
            # label placement l,r,lr for latitude and t,b,tb for longitude

[djhoese]

I'm playing around with this in #96 (mostly doc fixes at the moment). But while I was working on making a failing unit test I discovered that if you don't use a background image (so always return the overlays with a transparent background) that the results are always equal. So the cached result, the image generated from reusing the cached image (expected), and generating the image without the cache. This is obvious when you think about it since we never combine the foreground with a background, but I just wanted to point it out.

[djhoese]

@lobsiger @mraspaud I've noticed an inconsistency with add_overlays_from_dict. The docstring suggests it should always return a transparent image with the overlays added on to it. However, I've noticed that this is only true if caching is enabled. If you disable caching and provide a background image then it returns the background image with the foreground overlays applied. I think this has to do with the way the code is written and how it treats foreground/background.

Is this OK with us? I think the inconsistency between cache and no cache in this sense makes it undesirable.

[djhoese]

hhhmm but maybe that's a performance issue since in the non-cached case you'd be generating a whole new image in memory and then applying it. :thinking:

[lobsiger]

@djhoese @mraspaud yes of course caching is before all a performance thing especially if you generate many composites of the same scene that are resampled with generate=False (as I do). TBH my English and/or brain is not good enough to understand what you stated 3 post above. Is there a way to cache an overlay without using an image file on a disk? I came to the conclusion that as long as we (must?) use the PIL Image.paste()

https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.paste

there is no way to make final results using cached image files equal to writing all overlays directly on top of the generated satellite image. If I understand it right the Image.paste() uses the A channel as mask. Only if a pixel has A=255 it is not combined with the background. That means the big differences will always be with fine lines (due to antialiasing) and in surfaces filled with some transparency (the lakes I added). That's how I finally noted the difference that must have been there for many years.

[djhoese]

@lobsiger I think the antialiasing is going to be an issue for lines of any size, but yes they are maybe more noticeable for fine lines since the percentage of the line that is semi-transparent is much higher.

Some questions for you:

The images you are pasting here in this issue, are they the exact image saved by your code or are they screenshots of what you are seeing? Either way, in your image viewer do you set a background color for images? I have my image viewer set to show a checkerboard pattern underneath an image so when an image is transparent I can see a checkerboard. Here is what I'm seeing with the text I'm making where the square in the upper-right has an opacity of 127. The below are screenshots, so they are what I see in my image viewer.

The final image using caching:

The final image without caching:

So it seems the blending done by .paste during caching is overwriting the alpha channel instead of adding (taking the maximum is maybe appropriate?) to it. I would expect for a black background with alpha=255 and an overlay shape of alpha=127 that the final result has an overall alpha of 255 but the green color of the square is mixed between black (0, 0, 0) and pure green (0, 255, 0).

Can someone sanity check me on this? I'll look into using some of the other masking and alpha_composite options and see how it changes things.

[lobsiger]

@djhoese the images sort from satpy as .png. Then in a last step I add the legend at left with Image Magick (IM) and change to .jpg. I do not see a difference of the satpy issued .png and the .jpg. Normally I finally make .webm images because these compress much better than .jpg. But github does not accept *.webm files added to comments. Except for IM nothing else is used.

[djhoese]

Ah JPEG doesn't support transparency anyway. I'm not sure why you wouldn't be seeing a difference between .png and .jpg unless your image viewer is using a black background behind the image. Anyway, I'll keep playing with this. I've got the caching output to be decently consistent and what I expect by using alpha_composite, but without caching it is still doing something funny (not using transparency on the green square so it shows full brightness).

[lobsiger]

I think the PIL docs of Image.paste() explain what we see: " If a mask is given, this method updates only the regions indicated by the mask. ...... Where the mask is 255, the given image is copied as is. Where the mask is 0, the current value is preserved. Intermediate values will mix the two images together, including their alpha channels if they have them."

As a mask we use the overlay (foreground) A channel. As long as I use colors with opacity 255 (like in the Cities and Points symbols and their labels added) there is no difference. In thin lines even the center pixels might have opacity < 255 and are mixed with the black background "including alpha". That's why borders and rivers are always dim red and dim blue when cached.

[lobsiger]

I know that .jpg has no transparency. Maybe my FireFox browser hides some differences that you see in special image processors. The only problem with transparencies I remember was Himawari .png images way outside the full disk. I always use nodata_values either black or white. If you think it helps somehow I can further shrink the eurol10 area used so far and post the output *.png without the IM step. As seen above with my all white "Bracknell" MSLP chart it changes when you click on it.

[djhoese]

Here's a very simple test I'm doing:

        from pycoast import ContourWriterAGG

        proj4_string = "+proj=longlat +ellps=WGS84"
        area_extent = (-10.0, -10.0, 10.0, 10.0)
        area_def = FakeAreaDef(proj4_string, area_extent, 200, 200)
        cw = ContourWriterAGG(gshhs_root_dir)

        test_shape_filename2 = tmp_path / "test_shapes2"
        with shapefile.Writer(test_shape_filename2) as test_shapes:
            test_shapes.field('name', 'C')
            test_shapes.poly([[[5.0, 10.0], [10.0, 10.0], [10.0, 5.0], [5.0, 5.0]]])
            test_shapes.record("upper-right-box")

        overlays = {
            "cache": {"file": os.path.join(tmp_path, "pycoast_cache")},
            "shapefiles": [
                {"filename": str(test_shape_filename2), "fill": (0, 255, 0), "outline": (255, 255, 0), "fill_opacity": 127},
            ],
        }

        # Create the original cache file
        print("Before initial cache")
        background_img1 = Image.new("RGBA", (200, 200), (0, 0, 0, 255))
        cached_image1 = cw.add_overlay_from_dict(overlays, area_def, background=background_img1)

        # Reuse the generated cache file
        print("Before cache reuse")
        background_img2 = Image.new("RGBA", (200, 200), (0, 0, 0, 255))
        cached_image2 = cw.add_overlay_from_dict(overlays, area_def, background=background_img2)

        # Create without cache
        overlays.pop("cache")
        print("Before no cache")
        background_img3 = Image.new("RGBA", (200, 200), (0, 0, 0, 255))
        nocache_image = cw.add_overlay_from_dict(overlays, area_def, background=background_img3)

        # Manually (no dict, no cache)
        background_img4 = Image.new("RGBA", (200, 200), (0, 0, 0, 255))
        cw.add_shapefile_shapes(background_img4, area_def, **overlays["shapefiles"][0])

So with changing things in the caching logic to use alpha_composite I get what I expect from these operations in the background_imgX objects. That is, I get a black background and a green square in the upper-right but the green square is "faded" like it has an opacity of 127 applied to it but the square is not transparent like the checkerboard images I showed above.

Now the problem is that in last two cases of the above code (no cache and "manual") the green square is completely bright like the opacity isn't having any effect. I'll have to continue debugging tomorrow.

[djhoese]

Nevermind, I did another shape that is fully opaque green and it is clear that the partially transparent one is paler in the cached image, less pale in the manual image, and both are not as bright as the fully opaque one. So...I might just be in the same position as I was when I started today.

[djhoese]

I wonder if this is a pre-multiplied alpha versus regular alpha kind of thing. Where the colors in the saved/cached image are already multiplied by alpha so when we do the alpha_composite is doesn't work right.

[lobsiger]

@djhoese as you suggested https://github.com/pytroll/pycoast/issues/95#issuecomment-1489190995 I add two (size reduced) .png exactly as those are written by satpy. I see no difference with the .jpg versions in the FireFox browser and in the Windows supplied image processing programs they also look the same (I see no remaining transparency or checkerboard). So it seems this adds nothing new.

With the knowledge I think to have now I'd rather close this issue. I see no solution for resolving the antialiasing issue. Writing an antialiased white line on a bunch of different composite images cannot be replaced by a cached line that does not yet know on what background it will be applied later. If not already there (and overlooked by me) a note in RTD might be appropriate.

[djhoese]

Interesting...but I think I've actually figured this out. I just need to implement it and test it. I'll push it to my PR and then you can let me know what you think. This is essentially a pre-multiplied alpha issue from what I can tell. The cache image being on a black background is causing any semi-transparent pixels to get multiplied by the alpha value (in combination with the black background) which reduces the color of the provided value. For example, drawing with a fill color of (0, 255, 0, 127) is resulting in the cached image being (0, 126, 0, 126). When it is then applied to the background image (which is (0, 0, 0, 255) in my tests) the final result is (0, 62, 0, 255). If you "reverse" the alpha multiplication you get a value near what you expect:

In [4]: 62 * (255 / 126)
Out[4]: 125.47619047619047

I'm sure there is some floating point issues going on here, but this is a pretty good sign from what I can see.

[lobsiger]

@djhoese what I said above is just my understanding of what the current PyCoast code with PIL Image.paste() does. I do not yet understand what your magic with alpha_composite is all about. That's just not my level. So let's see what you finally figure out ...

[djhoese]

@lobsiger Ok I've made the main "fix" commit to my PR (https://github.com/pytroll/pycoast/pull/96), but I need to continue testing. So far my background image is always pure black which isn't very realistic. So there is a chance that the background is covered by the overlay rather than blended.

[djhoese]

I added more tests and things seem to be as I expect. If you could test that branch of mine that'd be great. You should be able to do:

pip install git+https://github.com/djhoese/pycoast.git@bugfix-dull-cache

[lobsiger]

@djhoese I replaced half an hour ago by hand 2 existing lines with:

... fg = foreground.convert("RGBa") background.paste(fg, mask=fg) ... premult_foreground = self._foreground.convert("RGBa") self._background.paste(premult_foreground, mask=premult_foreground) ... And I get the same results as before. Did I miss something? Does your pip install line change much more?

[lobsiger]

Maybe it's a cache thing of the Firefox browser. I close and login again.

[djhoese]

Hm, those are definitely the most important lines. I don't think any of the other changes would have an effect. :thinking: :confused:

[lobsiger]

@djhoese I tried once more. Deleted the cached overlay file, closed and reopened the browser. Results are exactly (visually) as before https://github.com/pytroll/pycoast/issues/95#issuecomment-1486881288 Do your tests show something else?

[djhoese]

My unit tests show differences, but I just tried a real world case and you are correct, things don't look correct. I will triple check, but it looks like was I posted before. That is, the overlay polygons (filled coastlines, etc) retain their alpha value from the overlay so you get a transparent polygons showing the "background":

So in my image viewer the background is a checkerboard. It makes sense for the space pixels in the upper-left (invalid pixels), but the green portions are supposed to be combined with the opaque layer below. Although...I'm actually even more confused by the fact that I'm specifying a fill color for "coastlines" and it is being used for all of those countries in the Caribbean.

Edit: Oops I installed the main branch. Let me try my PR branch.

[djhoese]

Ok when I actually install the right version of pycoast (my PR) I get:

If you look closely at the green regions (not sure they show up in the tiny screenshot) you can see the clouds underneath the green.

[djhoese]

@lobsiger What version of Pillow do you have?

[lobsiger]

@djhoese my pillow is 9.2.0. Do you think your test (with the PR) above is now what you expect?

[djhoese]

Yes. It is the same exact (within 1 pixel value for the uint8 images) image without caching as with caching. Let me try upgrading my pillow (I'm at 9.1.1) and see what happens.

Edit: Updating to pillow 9.2.0 on my PopOS (think Ubuntu) system and I get the same results.

[djhoese]

@lobsiger Triple check that you don't have any of the other changes that we played with before I finished my PR (like the all-white cache starting color or playing with alpha_composite if you did that).

Edit: Or that you're modifying the installed version of the source code in the environment that you think you are and that you are reinstalling it if necessary after the changes.

[lobsiger]

@djhoese I made a backup copy when we started *.sik. Here is the diff:

(pytroll) eumetcast@kallisto:~/miniconda3/envs/pytroll/lib/python3.10/site-packages/pycoast$ diff cw_base.py.sik cw_base.py

1202c1202,1204
<                     background.paste(foreground, mask=foreground.split()[-1])
---
>                     # background.paste(foreground, mask=foreground.split()[-1]) DEBUG
>                     fg = foreground.convert("RGBa")
>                     background.paste(fg, mask=fg)
1215c1217,1219
<             self._background.paste(self._foreground, mask=self._foreground.split()[-1])
---
>             # self._background.paste(self._foreground, mask=self._foreground.split()[-1]) DEBUG
>             premult_foreground = self._foreground.convert("RGBa")
>             self._background.paste(premult_foreground, mask=premult_foreground)

[djhoese]

:confused: I...I...I don't know what else could be going wrong. If you are sure you are using that pytroll environment when you run your stuff then this should just work. It doesn't even matter if you have old cached images because all of these fixes should happen after loading the cached image, not during its creation.

What version of aggdraw? Are you using the ContourWriterAGG?

[lobsiger]

AFAIK only agg has opacity. aggdraw 1.3.16 py310hb761d4a_0 conda-forge Maybe it's not a good idea to edit files in a conda environment (but all other changes worked somehow)?

[djhoese]

I mean it isn't best practice, but it should work the way you did it. If you were on Windows then I might question something being cached or something weird, but what you have should work. I technically have aggdraw 1.3.14, but the changes between aggdraw patch versions are mostly for Python 3.10/3.11 compatibility and other build related things. I'm on Python 3.10 too.

Would you mind trying the pip install command I did before just to install my PR version and try again. If you still don't get the expect results you could run pytest --pyargs pycoast.tests and it should run all the tests.

Oh, another idea, when you run your script, are you running it from a pycoast source directory/git clone?

[lobsiger]

Sorry, I have no development environment here. This is just a plain Satpy install. So I'm afraid no tests available. I'm running from some sub directory I have all the scripts I'm testing ... I'm a little bit reluctant to pip install into a conda install. Do you think that will work?