DAS-2161: Updates libs with no user visible changes.

flamingbear commented 1 month ago

Description

This ticket was to bump the library versions and discover why regression tests stopped passing.

There was a change in some image metadata when the pillow (PIL) library was updated from v10.0.0 to v10.x.0.

The culprit is that GDAL adds a No Data value to the PNG metadata only when the colormap has 256 values and a single one of those is fully transparent, e.g RGBA(0,0,0,0).

pillow 10.0.0 always returned a 256 length colormap from the quantized image irrespective of the number of colors requested.

Pillow 10.x.0 changed this behavior to only return a palette of the unique values in the quantized image.

The regression tests failed because the shorter colortable passed to gdal did not meet GDAL's (presumed, not documented) feature of adding a No Data value to the output metadata.

By beginning with a 256-element all-black color table and writing extracted, truncated one into it, we retain the previous behavior by having a full length 256 value color table, which allows for some images to have a NoData Value in the PNG metadata.

Jira Issue ID

DAS-2161

Local Test Steps

Pull branch Look that the pip_requirements.txt file has updated libraries Run build and test the image.

❯ ./bin/build-image && ./bin/build-test && ./bin/run-test

Deploy image to Harmony-In-A-Box and run the HyBIG regression tests locally. They should pass as they did before with no changes to the regression image or tests.

No release. I think these changes can be picked up in the next release since there are no user facing changes.

PR Acceptance Checklist

[X] Jira ticket acceptance criteria met.
[x] CHANGELOG.md updated to include high level summary of PR changes.
[ ] ~~docker/service_version.txt updated if publishing a release.~~
[x] Tests added/updated and passing.
[ ] ~~Documentation updated (if needed).~~

owenlittlejohns commented 4 weeks ago

Oh, I looked at the Snyk issues. They are kind of tricky - looks like the packages need a version of Python that Snyk isn't running. Snyk looks like it is using Python 3.9. The Python version used to be non-configurable in the way we are using Snyk, and I think that might still be the case. It's a total pain and is a reason why some projects get false positives.

flamingbear commented 4 weeks ago

Oh, I looked at the Snyk issues. They are kind of tricky - looks like the packages need a version of Python that Snyk isn't running.

I got totally side tracked and didn't look into it. Thanks for the tip.

flamingbear commented 4 weeks ago

Oh, I looked at the Snyk issues. They are kind of tricky - looks like the packages need a version of Python that Snyk isn't running. Snyk looks like it is using Python 3.9. The Python version used to be non-configurable in the way we are using Snyk, and I think that might still be the case. It's a total pain and is a reason why some projects get false positives.

So This is the page I got to, but I can't even see as much information as you said following the gear symbol or the "about this error" link.

owenlittlejohns commented 4 weeks ago

With the Snyk thing, there is a little bit of this being a prior bee in my bonnet. But I looked up the "Python version not supported" issue via the page with the error codes that is linked to from "About this error". That mentioned that it was likely the version of one of the packages needing a different version of Python.

To dig out the version of Python being used, when you click on the gear icon, move from the "Settings" tab to the "Overview" one. That then lists Python 3.9. But, to my knowledge, the version of Python is limited to being the same throughout all our Snyk projects. I had raised that the Python version being old was causing false positives with the CI/CD team for EED-3 before, and (if I remember rightly) Snyk got us on to a beta version of Snyk that allowed for us to configure the global version of Python being used (I think the CI/CD team bumped us from 3.7.2 to 3.9), but we still don't have a granular per-project control that ultimately we need.

TL;DR - I'm kind of unsure how to proceed here. I don't think we can get around the Snyk issues as they stand.

flamingbear commented 4 weeks ago

Python version not supported" issue via the page with the error codes that is linked to from "About this error".

I just did this again because I was sure I did this. But this time I saw it. I must have been typing it wrong (again and again).

But we can fix with a .synk file? Alternatively, add a .snyk file for Python version selection override.

flamingbear commented 3 weeks ago

@owenlittlejohns following discussions with Jon and Venku. I'm adding this snyk file so they can bump the default version to 3.9. But there's still some wonder as to why snyk is actually failing at 3.9 and 3.11 since 3.7 isn't used anywhere in the project

nasa / harmony-browse-image-generator