search

pdf-association / pdf-issues

Industry-based resolutions for issues and errata reported against any PDF-related specification
https://pdf-issues.pdfa.org/
64 stars 2 forks source link

ICCBased Lab colorspace #31

Open faceless2 opened 3 years ago

faceless2 commented 3 years ago

This text is from ISO32K2:2020, bottom of page 191:

PDF writers shall only use the profile types shown in "Table 67 — ICC profile types" for specifying calibrated colour spaces for colouring graphics objects. Each of the indicated fields shall have one of the values listed for that field in the second column of the table. Profiles shall satisfy both the criteria shown in the table. The terminology is taken from the ICC specifications. ... Note 1. XYZ and 16-bit Lab profiles are not listed.

and here's table 67:

Header Field Required Value
deviceClass icSigInputClass ('scnr')
icSigDisplayClass ('mntr')
icSigOutputClass ('prtr')
icSigColorSpaceClass ('spac')
colorSpace icSigGrayData ('GRAY')
icSigRgbData ('RGB ')
icSigCmykData ('CMYK')
icSigLabData ('Lab ')

I've a few minor issues with this.

First, "The terminology is taken from the ICC specifications." - maybe, but there's no "icSigNNN" anywhere in ICC v4.3 or 2.4. Not a big deal, but replacing "icSigDisplayClass ('mntr')" with just "mntr" might be an improvement.

Second, the note explicitly states "16 Bit Lab is not listed" - not listed doesn't really tell us anything. The text was "not supported" in PDF1.7, which I think was a better phrasing.

Third, despite being not listed (or supported), Lab is listed in table 67. It's not clear what's supposed to be allowed - 8 bit Lab only? In fact there's no such thing as "16 bit Lab" in ICC: there's just Lab. It may have an 8-bit table, a 16 bit lookup table or a parametric curve. I've never seen an 8-bit one, but neither 16-bit nor parametric curve Lab profiles are supported in Acrobat.

I think the intent is just to exclude Lab as a colorSpace type, in which case I'd suggest dropping the "Lab" line from Table 67, and changing the Note to "Note 1. in particular, XYZ and Lab profiles are not supported." - although it's a bit redundant given the lack of those entries in table 67.

petervwyatt commented 3 years ago
  1. I also cannot find "icSigXxx" in any ICC document. I also cannot find "deviceClass" or "colorSpace" so it possibly from some legacy proprietary API. But I'd go as far to suggest that the PDF spec should use the exact phrasing from ICC specs (Table 18 in ICC:2010) - such as "Input device profile 'scnr'" as that way they are both descriptive and technically accurate. So Table 67 would become:
ICC Header Field Required Value
Profile/device class field Input device profile ('scnr')
Display device profile ('mntr')
Output device profile ('prtr')
ColorSpace profile ('spac')
Data colour space field Gray ('GRAY')
RGB ('RGB')
CMYK ('CMYK')
CIELAB or PCSLAB ('Lab')
  1. In regards to "NOTE 1 XYZ and 16-bit L*a*b* profiles are not listed." - I think this is a typo as I believe it may apply to the preceeding bullet points (most likely the 2nd last bullet as the last bullet is relative new), and not to the Table 67! 'Lab' is definitely in Table 67 and '`XYZ '' is definitely not!

Further note that 'Lab' is described as context dependent as highlighted by footnote (b) in ICC:2010 Table 19:

"The signature 'Lab' refers to CIELAB or PCSLAB depending upon the context."

In which case this is most likely a comment about ICC PCS limitations. See ICC:2010 6.3.4:

"The colorimetric data defined in 6.3.2 and 6.3.3 may be specified either as PCSXYZ or PCSLAB data. When specified as PCSXYZ data they shall be encoded using 16 bits per component while when specified as PCSLAB data they shall be encoded as either 8 bits per component or 16 bits per component. Additionally, within the context of DToBx and BToDx tags PCSXYZ or PCSLAB data shall be encoded using float32Number values that directly express PCSXYZ or PCSLAB calorimetry."

And if this is the case then I would think we can strike the entire note as it is very old (first added in PDF 1.3) and full ICC support should be assumed.

But of course I could also be entirely incorrect. @lrosenthol? @MPBailey?

@faceless2 - if you could share your ICC profiles that may also help. Thanks.

MPBailey commented 3 years ago

If you go back to one of the earlier Adobe specs (I looked at 1.4) the text is "As of version 1.4, PDF supports only the profile types shown in Table 4.17; other types may be supported in the future. (In particular, note that XYZ and 16-bit Lab* profiles are not supported.)" I think that supports faceless2's interpretation (and my prior assumption) that it's a confirmation that they are deliberately omitted from the table. And I agree that "not supported" represents that intention better.

I also found some of the icSigxxx names when I went back to CC.1:1998-09, In what was then appendix H (C Header File Example). As you say, it looks like an API definition rather than terminology used in the standard itself, and it’s not there in 2020. I agree with your suggested rewording.

But you need to be careful about spaces inside the quotes in the “required value” volume; the current 32000-2 is correct on those. No spaces in any 4-letter values, a space at the end for RGB and Lab.

Sorry, this is on the edge of my expertise; I have no comments on the 16-bit Lab question, although it does look as if it needs addressing.

faceless2 commented 3 years ago

I'm almost certain the intention is not to exclude Lab as a PCS space - I have a number of regular CMYK profiles that use a Lab PCS space, it's very common.

Here are the profiles (from Elle Stone):

and, just to see if it works (it doesn't): a sample PDF with the new 7-color FOGRA55 beta profile is at https://bfo.com/misc/color/icc-fogra55-7clr.pdf.

I'm rapidly approaching the edge of my expertise here too. All I can say for sure is none of the above PDFs meet the current wording of ISO32K, and none of them work in Acrobat.

If the intention was to allow only ICC profiles with an "Data colour space field" of gray, rgb or cmyk, then I agree we could just say that in Table 67 - delete the explanatory note, and delete any mention of Lab from Table 67. That seems to match what Acrobat is expecting.

faceless2 commented 3 years ago

A bit more to add - I found a Lab profile with an 8-bit LUT (there's one in the "PDFUA-Reference-10_(Danish-form).pdf" document which is part of the PDF/UA testsuite).

This works in Acrobat without error, but there is some sort of special handling going on - I have to change the components, from the range [0..1] which is normal for ICCBased colors, to the range 0..100, -128..127 and -128..127 which is the range used for the LabColorSpace (which also means LabColorspace and this ICC space are effectively interchangeable)

So now the logic of this test is lost on me. I have a feeling we'll need Leonard for this one.

petervwyatt commented 3 years ago

@lrosenthol - seeking your input...

faceless2 commented 3 years ago

To clarify where we're at - Acrobat allows:

I suppose the why's behind this are irrelevant at this point - if the intention is to limit the format to just these types of profiles, I'd suggest something building from this text. I think Peter's table in the first comment gets us there with GRAY, RGB and CMYK, it's really only the Lab text that was confusing.

MPBailey commented 3 years ago

I've just tried all five of the sample PDFs attached above in one of our implementations. Only the 7-channel Fogra case errors; the other all give output that appears at least halfway sensible. @faceless2 : just to help me get further as easily as possible; Are the three PDFs using Lab profiles designed to give equivalent appearance, or should I expect them to look different from each other?

faceless2 commented 3 years ago

If all the profiles were rendered correctly, for the 3 "Lab" files we'd expect to see:

Acrobat DC gives this rendering for icc-lab-8bit.pdf and fails to load the others. If you see black/red/purple/yellow-brown, the Lab profile is being treated as RGB (macOS PDF viewer does this)

For the icc-xyz.pdf profile we'd expect to see:

The FOGRA55 test has 7 stripes, one per ink: cyan, magenta, yellow, black, orange, green, violet.

The 7-color one is a stretch for sure, but most ICC engines should be able to handle the others without any effort - the XYZ profile in particular is literally just an identity matrix.

MPBailey commented 3 years ago

Thanks @faceless2. I've kicked this over to a couple of our colour developers for confirmation of what I'm seeing.

FWIW, the challenge with the 7CLR profile is not handling the profile itself, it's the implications for what might need to be done around live PDF transparency if you could tag individual graphics with that profile, or, more challenging, use it as a blending color space.

The current (32000-2:2020) standard says that Gray, RGB and CMYK spaces may be used as blending colour spaces, but Lab and ICCBased may not: "spaces that represent lightness and chromaticity separately (such as Lab, Luv, and HSV) shall not be used as blending colour spaces because the compositing computations in such spaces do not give meaningful results when applied separately to each component."

By that logic a 7CLR profile, if allowed at all, should be supported as a blending colour space, but I think we're a long way from specifying what that would actually mean in practice.

faceless2 commented 3 years ago

I'm getting well beyond my expertise level here so I'll stop posting except to say that the thought of blending in a 7CLR space fills me with an existential dread. Thanks for digging into this Martin.

MPBailey commented 3 years ago

Hmmm. FWIW the ICC Profile inspector can't open the three 'Elle' profiles.

My developers have also pointed out that the Lab profiles are all "abstract" class profiles, which are not supported in PDF. That's why the 16-bit and matrix PDF files are showing up as using RGB in Harlequin; we're falling back on the alternate space; in all cases that's DeviceRGB (because N = 3 with no Alternate key).

petervwyatt commented 3 years ago

@MPBailey please switch to https://github.com/InternationalColorConsortium/DemoIccMAX (previously known as RefIccMax). You can then run iccDumpProfile -v file.icc ALL or use the GUI. This supports all of V2, V4 and V5 (aka both ICC.1 and ICC.2).

MPBailey commented 3 years ago

It looks as if some of the colour values in the sample files are incorrect as well.

The 8 bit case has:

0 0 0 scn 20 30 40 130 re f 100 0 0 sc 60 30 40 130 re f 75 0 50 sc 100 30 40 130 re f 75 50 0 sc 140 30 40 130 re f

But the 16 and v4 matrix case has:

0 0 0 scn 20 30 40 130 re f 1 0 0 sc 60 30 40 130 re f 0.75 0 0.5 sc 100 30 40 130 re f 0.75 0.5 0 sc 140 30 40 130 re f

So I guess near-black is actually expected for icc-lab-16bit_v2.pdf and icc-lab-matrix-v4.pdf.

Table 68 gives ranges for Lab* as "L∗: [0 100]; a∗ and b∗: [−128 127]"

faceless2 commented 3 years ago

Yes I noted that in https://github.com/pdf-association/pdf-issues/issues/31#issuecomment-773997640 - there's some special handling going on, it appears for 8-bit Lab profles only, Acrobat is substituting the regular Lab colorspace - which means the ranges are no longer [0..1]

MPBailey commented 2 years ago

Note the agreed amendment within this section shown at https://pdf-issues.pdfa.org/32000-2-2020/clause08.html#H8.6.5.5

But that does not change Table 67 or the text immediately above it at all, so does not affect this issue.

Working on the basis that improving wording to match the behaviour of a well-known implementation should be the route to go. That means:

petervwyatt commented 2 years ago

@lrosenthol - could you please research a well-known implementation for the points in Martin's last comment so we can decide if we should strictly prohibit, or be more flexible and possibly allow "implementation dependent" statements for the issues described here?

lrosenthol commented 2 years ago

PDF supports ICC Profiles in Gray, RGB, CMYK and Lab spaces. Period. (the main reason for this, as noted above, is that the current transparency algorithms aren't designed to work in any other space - especially N-Color).

I am not aware of "well known implementation" doing anything special with Lab profiles - but I am happy to investigate.

I do agree that the text uses old terms that should be updated to reflect the current ICC.1 spec.

petervwyatt commented 2 years ago

@faceless2 - I found these 2 ICC profiles on the 'net which both have Data Color Space=XYZ and ProfileClass=ColorSpace and for which the AtoB0 and BtoA0 tags use lut16. They are very slightly different. According to wording in both ISO 32000-1 and ISO 32000-2 these ICCs are valid in PDFs.

XYZ-iccs.zip

But according to your comment above on 17 Feb 2021 you say this is still not supported by Acrobat. The Adobe PDF 1.4 note that Martin mentioned in his comment on 5 Feb 2021 (and I also confirmed was still in Adobe PDF 1.7), but which didn't make it into any ISO spec. I assume because it was thought to be an implementation limitation by a specific vendor when ISO-ised as I know as least one implementation that supports this. Could you make some test PDFs with these ICCs for evaluation please? @lrosenthol - could you please evaluate too?

So I think one decision now is whether the legacy Adobe 1.x statement "(In particular, note that XYZ and 16-bit Lab* profiles are not supported.)" is now an ISO requirement (and it should be reworded as a file format requirement!), or left as implementation dependent (so maybe as an informative note). I support the latter.

faceless2 commented 2 years ago

@petervwyatt Good find! The "xyz identity" profile I found in https://github.com/pdf-association/pdf-issues/issues/31#issuecomment-773947764 was of the wrong class ("Abstract") which means it was not valid for ISO32K - my error. These are profile class "ColorSpace" and that seems to make the difference..

Both work perfectly in Acrobat, Foxit, macos Preview, ABBY and with our PDF API too.

The red, green, blue and white color bars should look like a single bar; if the bars appear divided into vertical segments the PDF viewer is struggling with this type of ICC profile. The SVG is just for interest; the top-two rows will work in Safari only.

petervwyatt commented 2 years ago

Thanks Mike!

With a bit more searching I also found Data Color Space=Lab and ProfileClass=ColorSpace and for which the AtoB0 and BtoA0 tags also use lut16, which covers the other part of the legacy statement "(In particular, note that XYZ and 16-bit Lab* profiles are not supported.)" . ICCs with Data Color Space=Lab and ProfileClass=ColorSpace that use lut8 are relatively common so I won't bother providing here.

ITULab.zip

So if this works (I suspect it might) then all parts of the legacy Adobe statement that did not make it into ISO 32K was clearly left out for good reason.

To @lrosenthol's comment: 11.3.4 para 2 defines blending colour spaces as a inclusive list of specific PDF colour spaces as well as the statement "The Lab space and ICCBased spaces that represent lightness and chromaticity separately (such as Lab, Luv, and HSV) shall not be used as blending colour spaces because the compositing computations in such spaces do not give meaningful results when applied separately to each component. In addition, an ICCBased space used as a blending colour space shall be bi-directional; that is, the ICC profile shall be capable of both device to PCS and PCS to device transformations.". Table 145 also states "... exclude Lab and lightness-chromaticity ICCBased colour spaces..." but doesn't have the specific inclusive list - but it does have 2 cross-references to 11.3.4 so I would assert it is clear enough.

So should an updated Table 67 also contain XYZ as a valid Data colour space?

MPBailey commented 2 years ago

I found these 2 ICC profiles on the 'net which both have Data Color Space=XYZ and ProfileClass=ColorSpace [...]. According to wording in both ISO 32000-1 and ISO 32000-2 these ICCs are valid in PDFs.

Could you explain why you believe that those profiles are valid according to 32000-2? The 2020 edition says, in the bullets just above Table 67:

PDF writers shall only use the profile types shown in "Table 67 — ICC profile types" for specifying calibrated colour spaces for colouring graphics objects. Each of the indicated fields shall have one of the values listed for that field in the second column of the table. Profiles shall satisfy both the criteria shown in the table. The terminology is taken from the ICC specifications.

And XYZ is not included on the colorSpace row of Table 67.

I'm not arguing that it shouldn't be supported, with some suitable wording around transparency blending spaces, just that I don't believe that it is right now.

faceless2 commented 2 years ago

Re. Peter's ITULab.icc it's a 16-bit Lab with ProfileClass=ColorSpace, and (as with the XYZ profiles) my earlier 16-bit Lab test was ProfileClass=Abstract, which invalidates my earlier results for 16-bit Lab.

This one behaves identically to the 8-bit Lab ICC profile I tested above in all PDF viewers I tested: specifically

I've put a PDF at https://bfo.com/misc/color/icc-lab-test2.pdf, and here's a screenshot from Acrobat: the numbers in each box are the components given to the sc operator.

image

If there was no special processing on the components I would expect the top two rows to be similar. If there is special scaling on the components, the bottom two rows are similar.

petervwyatt commented 2 years ago

@MPBailey

I found these 2 ICC profiles on the 'net which both have Data Color Space=XYZ and ProfileClass=ColorSpace [...]. According to wording in both ISO 32000-1 and ISO 32000-2 these ICCs are valid in PDFs.

Could you explain why you believe that those profiles are valid according to 32000-2? The 2020 edition says, in the bullets just above Table 67

That was my bad- I meant to say except that they are XYZ (I was originally going to load up both XYZ and Lab ICCs but found that the Lab files I had were bad so changed my mind at the last minute and didn't change my comment). Sorry for any confusion!

I 100% agree that the original and current specs do not include DataColorSpace=XYZ.

seehuhn commented 2 months ago

The discussion above mentions the existing text "XYZ and 16-bit L*a*b* profiles are not listed" (just above Table 67), but I couldn't make out whether or not this came to a conclusion.

To me this text didn't make sense even in the PDF 1.3 spec. There it still read as follows:

PDF 1.3 supports only the profile types shown in Table 4.17; [...] (In particular, note that XYZ and 16-bit L*a*b* profiles are not supported.)

The phrase "in particular" seems to indicate that the table excludes 16-bit L*a*b* profiles, but I don't think it does (the 'Lab ' color space is explicitely listed as being allowed, and whether the resolution is 8-bit or 16-bit cannot be seen from the information in the table.)

Question. In PDF-2.0, are 16-bit L*a*b* profiles supported or not?