search

opengeospatial / geotiff

18 stars 9 forks source link

New working item for BigTIFF community standard #103

Open joanma747 opened 2 years ago

joanma747 commented 2 years ago

BigTIFF has become more necessary that ever with lager images, multiband images and multiple resolutions (COG). The Testbed 17 has a proposal for a new draft specification for BigTIFF: https://gitlab.ogc.org/ogc/T17-D046-COG-Specification-ER/-/tree/master/standardDrafts/BigTIFF

We would like to move it to an OGC community standard in this group. The process is a bit different form other community standards. There is only an "informal" web page that defines BigTIFF. IT is clear and comprehensible but not sufficiently formal, so we cannot adopt any preexisting document directly. It was needed to reformatted it into clear requirements.

The issue is identified in https://github.com/opengeospatial/geotiff/issues/12 where an analysis of the implications for BigTIFF tags in GeoTIFF keys could have is presented

ogcscotts commented 2 years ago

@joanma747 - two questions on the Community Standard.

  1. Do the owners of BigTIFF agree to submit the document?
  2. If the Community Standard is being written as a result of Testbed activity, we will need to get the owner of BigTIFF to agree to the text and that it represents an authoritative specification for BigTIFF before we could consider this for a Community Standard. It may also be worth considering whether the BigTIFF ownership could be transferred to OGC.
cmheazel commented 2 years ago

An alternate approach, add a BigTIFF conformance class where we document the differences between TIFF and BigTIFF. No need for an external standard.

joanma747 commented 2 years ago

Let's advertise our initiative in the LibTIFF email list and look for their support. We should strongly underline that we are fully compatible with the previous documentation and implementation of the BigTIFF.

Consider to make BigTIFF a public repo

MOTION to establish an OGC BigTIFF standards development effort, using a public Git (gitlab or github) repository and invite participation of the BigTIFF community Moved by Chuck Heazel Second by Even Rouault Discussion: We are going to use the LibTIFF email list. IT seems that it will be and OGC standard with support of the community. It is not clear if this is an "OGC community standard" or a "pure OGC standard with the support of the community" at this point. NOTUC

We will ask Greg for a new repository for this working item. It makes configuration management easier

Let's draft some simple text to do the first contact to the LibTIFF list.

@cmheazel, the requirement class for BigTIFF is here: https://gitlab.ogc.org/ogc/T17-D046-COG-Specification-ER/-/blob/master/standardDrafts/BigTIFF/standard/clause_7_normative_text.adoc

joanma747 commented 2 years ago

The repository to work in this working item is here: https://github.com/opengeospatial/BigTIFF

joanma747 commented 2 years ago

The repository is now populated with the materials created from Testbed17. The intention is to keep them aligned (manually) until the end of Testbed 17, when the work will exclusively continue in the GeoTIFF.SWG

joanma747 commented 2 years ago

Proposal of text email for the LibTIFF list:

A group of people in the Open Geospatial Consortium (OGC, www.ogc.org) would like to move the BigTIFF into the OGC standards process. The intention is to create a OGC standard that is fully compatible with the previous documentation and implementation of the BigTIFF. We would like to do this in a open process and make sure that the community that was behind the BigTIFF specification are invited to participate in the process and are well recognized for their previous work. The OGC will provide a process to maintain the standard in the future and will work to make GeoTIFF and Cloud Optimized GeoTIFF (COG) standards to support both TIFF and BigTIFF. All OGC standards are available for free in the OGC website.

The process will be done in a open GitHub repository and the first attempt can be found here: https://github.com/opengeospatial/BigTIFF You are invited to submit issues or to create pull requests as part of the way to provided comments and improve the standard draft.

Is there any objection or comment before we start this process?

joanma747 commented 2 years ago

Even Rouault will send this to the LibTIFF email list. Thanks for that!

rouault commented 2 years ago

email just sent

rouault commented 2 years ago

Good question from Rob Tillaart regaring TIFF tags in general:

Rob Tillaart: Will the OGC put a process in place for BigTiff tags?

me: > What do you call exactly as BigTIFF tags ?

Rob Tillaart: In the past there was a lot of discussion about (un)documented / (un)registered tags and how to handle them. Will the OGC play a role in this process of documenting tags? I can imagine that they only are interested in Geospatial related tags and not in e.g. medical tags

rouault commented 2 years ago

""" Hi Even et al:

To my knowledge, the community that was behind the BigTIFF specification is now effectively Leica (medical, not Geosystems). The initial specification came from Ole Eichhorn, then at Aperio, a digital pathology company which was acquired by Leica. The involvement of Aware systems as far as I know was only hosting a discussion on their mailing list, and documenting essentially a copy of Ole's work. I don't believe the participants were from Aware. Anyone do please correct me if my understanding is incomplete or incorrect.

However, this unofficial specification only covers a migration of TIFF to 64-bits, and says nothing in particular about tiling, multi-resolutions, multi-channels, bit depths, compression, TIFFtags/metadata, etc, all of which need to be defined per application at a higher level. Various stabs at this for multiresolution imaging have been made over the years: Aperio.svs, COG, ZIF (my own), OME-TIFF (used ZIF as a reference point), and many 99% similar flavors of pyramidal tiled (big)TIFF. Shame we can't all get it together.

Given that GeoTIFF and COG specifications exist, what is OGC intending to add beyond that? Is it just the "64-bit"ness, or metadata?

There is no formal maintainer; the AWARE systems site does not maintain the BigTIFF specification, that is just a reference; another is at http://bigtiff.org, a site now owned by Leica.

And I'm not sure the wording on https://github.com/opengeospatial/BigTIFF is as intended?: "it becomes an OGC standard" sounds like OGC intends to take over ownership of BigTIFF? Leica might have something to say about that, although the specification has been put in the public domain. Personally, I really wish Adobe as the owner of the TIFF specification would step up to the plate with this, and update the actual TIFF specification to cover BigTIFF. Their focus has been elsewhere for many years, but this would be a great community service.

Regards,

Kemp Watson Objective Pathology Services Toronto, Canada """

rouault commented 2 years ago

Some clarification given by Joris Van Damme on the non-central implication of Aperio in the BigTIFF design: """ Kemp,

To my knowledge, the community that was behind the BigTIFF specification is now effectively Leica (medical, not Geosystems).

Incorrect. Leica, as a company, was only involved in that they were one of four sponsors of the BigTIFF implementation inside LibTIFF. They were completely unrelated to the design. It's possible a number of people that contributed to the design in the mailing list were employed by Leica at the time, but if so they were not speaking for that company.

The initial specification came from Ole Eichhorn, then at Aperio, a digital pathology company which was acquired by Leica.

No. Original specification was a community effort, on the mailing list, and to this day you can read up on that and verify it in the mailing list archive. Aperio, at that time, had an alternative design, partly incompatible, that they put into an independent implementation, without consulting the community. It made things quite awkward for a while. They still own bigtiff.org. I hope that at least the stuff they offer on that site is no longer incompatible with our own specifications, but I haven't checked.

The involvement of Aware systems as far as I know was only hosting a discussion on their mailing list

Incorrect. AWare Systems is me, Joris Van Damme. I actively contributed to the design phase quite a bit, though it truly was a community effort. I also did the implementation in LibTIFF, sponsored by four other companies.

Various stabs at this for multiresolution imaging have been made over the years

The SubIFDs tag allows for any number of any resolution child ifds. It's a specification totally independent from BigTIFF, SubIFDs can and should be used in classic TIFF and BigTIFF. You're not making sense.

I can go on correcting your other statements, but I fail to see how that's worth the effort at this point. It's ancient history. Why spin these things, what is your angle?

Given that GeoTIFF and COG specifications exist, what is OGC intending to add beyond that? Is it just the "64-bit"ness, or metadata?

Like the man said, it is the BigTIFF part, so, sure, the "64-bit"ness. BigTIFF is just TIFF with 64bit offsets, by design.

I feel the current activity in the GeoTIFF community is a good thing that will surely result in more widespread BigTIFF adoption, and more widespread confidence in its validity and the fact that it's here to stay. Even, thanks for letting us know.

Best regards,

Joris Van Damme AWare Systems """

joanma747 commented 2 years ago

It is so great that we have such support from Joris Van Damme. I love the tone of the last paragraph. This email clarifies the history and helps a lot.

rouault commented 2 years ago

Project of response to the reactions from the libtiff mailing list

Hi,
thanks to all who have provided feedback. Our group just met and we'd like to provide the following clarifications:
- we'll change the readme of the github repository to mention that the current BigTIFF specification is hosted on the awaresystems.be website rather than maintained.
- we do not plan, at least at that stage, to maintain a full TIFF tag registry. OGC only intends to maintain the GeoTIFF tags, as well as tags for geospatial metadata.
- the scope of the BigTIFF specification maintained by our OGC group will be exactly the same as the one mentioned above. As you may see from the draft, it is a relatively short specification, documenting only the differences between BigTIFF and classic TIFF, that is the evolution of the header, the directory structure, and more generally all the 64-bits indexing / offsets extension of TIFF6 specification. The TIFF 6 spec will be the reference for all aspects not modified by BigTIFF.
- "who is responsible for BigTIFF?" That's indeed the main issue that triggered our initiative. Our review of the current situation was that no one was formally a "owner" or in charge, or showing hints to be willing to be in charge, hence we've decided to step up. Our motivation to do that is clearly to be able to use BigTIFF in a GeoTIFF context. But BigTIFF having not a formal enough specification, this makes it hard to comply with the requirements of a standards process. We've considered to put BigTIFF as an annex of the GeoTIFF specification, but that would not have been a clean approach. We believe that by making BigTIFF a standalone specification, it might also be useful for other communities, beyond the geospatial one, and will make further adoption of the technology easier in contexts where a formal specification is needed.

Can we answer if there will be an ETS for BigTIFF ?

EmDevys commented 2 years ago

@rouault (and all) 2 comments:

joanma747 commented 2 years ago

I agree with the answers @rouault provides with the good additions that @EmDevys suggests.

rouault commented 2 years ago

I've integrated Emmanuel's suggestions. Should we wait next week meeting to validate the message ? (that should be fine)

rouault commented 2 years ago

The whole discussion on the libtiff mailing list is archived at https://www.asmail.be/msg0055229449.html

joanma747 commented 2 years ago

De : Scott Foshee [mailto:sfoshee@adobe.com] Envoyé : lundi 26 octobre 2020 21:53 À : Emmanuel Devys; Scott Simmons (ssimmons@ogc.org) Cc : wprotzman@dcscorp.com; 'Wilkes, Graham (NRCAN/RNCAN)' (graham.wilkes@canada.ca); Jennifer Hum-Miller; Roger Lott (EPSG); Even Rouault (even.rouault@spatialys.com); Charles.M.Heazel.ctr@nga.mil; Larry Beck; geotiff.swg@lists.opengeospatial.org; Scott Foshee Objet : Re: Adobe is in full control of the TIFF standard

Hi Emmanual and Scott…

Since Adobe is the copyright holder for Adobe TIFF, you can not create a Geospatial TIFF standard that’s based on Adobe TIFF without a license from Adobe.

I explained that this was not an option in an earlier e-mail.

I would be happy to convey this directly to the OGC legal department. Scott, are you the correct person at OGC for Adobe to convey copyright issues?

Also, I know the folks at ISO and will update them.

Thanks,

Scott

Where this boom comes from?. If I interpret this text literally (you can not create a Geospatial TIFF standard that’s based on Adobe TIFF without a license from Adobe.), this could affect also COG as well as GeoTIFF itself!. That could be the end of any of these initiatives (except if Adobe grants a licence to the OGC for free). Not everybody is for open standards. Lets hope that "OGC diplomacy" can navigate the situation.

EmDevys commented 2 years ago

Joan and all As you know, GeoTIFF is not a new standard, it dates from the mid 90’s and was submitted by JPL and SPOT Image Corp. You may check the original GeoTIFF 1.0 at http://geotiff.maptools.org/spec/geotiffhome.html which includes: Copyright Portions of this specification are copyrighted by Niles Ritter and Mike Ruth. Permission to copy without fee all or part of this material is granted provided that the copies are not made or distributed for direct or commercial advantage and this copyright notice appears. Licenses and Trademarks Aldus and Adobe are registered trademarks, and TIFF is a registered trademark of Aldus Corp., now owned by Adobe. SPOT Image, ESRI, ERDAS, ARC/Info, Intergraph and Softdesk are registered trademarks. This topic has been discussed when OGC decided to handle GeoTIFF and maintain it (when GeoTIFF.SWG was created, under Ted Haberman as HDF Group). And there was at this time agreement in the OGC that there is (or should be) no legal issue behind the usage of TIFF specification by GeoTIFF standard. I guess what Scott Foshee pushed at this time (and that you underlined) was some excessive statement, presumably from some kind of legal staff unaware of technical staff.

If I remember correctly, when this email was received, Scott (Simmons) checked with OGC layer, and confirm there was no stopper for the OGC GeoTIFF standardisation action. There is some statement in OGC GeoTIFF 1.1 in iii. Preface about patent rights, which was reviewed and tuned with OGC staff.

I let Scott Simmons adjust as necessary (or correct) if anything would be incorrect.

In conclusion, I guess (and hope) it is not necessary to re-open this debate (and discussion with Adobe/Scott Foshee), unless there are solid objections and reasons for that.

De : Joan Masó @.*** Envoyé : jeudi 30 septembre 2021 13:25 À : opengeospatial/geotiff Cc : Emmanuel Devys; Mention Objet : Re: [opengeospatial/geotiff] New working item for BigTIFF community standard (#103)

De : Scott Foshee @.**@.>] Envoyé : lundi 26 octobre 2020 21:53 À : Emmanuel Devys; Scott Simmons @.**@.>) Cc : @.**@.>; 'Wilkes, Graham (NRCAN/RNCAN)' @.**@.>); Jennifer Hum-Miller; Roger Lott (EPSG); Even Rouault @.**@.>); @.**@.>; Larry Beck; @.**@.>; Scott Foshee Objet : Re: Adobe is in full control of the TIFF standard

Hi Emmanual and Scott…

Since Adobe is the copyright holder for Adobe TIFF, you can not create a Geospatial TIFF standard that’s based on Adobe TIFF without a license from Adobe.

I explained that this was not an option in an earlier e-mail.

I would be happy to convey this directly to the OGC legal department. Scott, are you the correct person at OGC for Adobe to convey copyright issues?

Also, I know the folks at ISO and will update them.

Thanks,

Scott

Where this boom comes from?. If I interpret this text literally (you can not create a Geospatial TIFF standard that’s based on Adobe TIFF without a license from Adobe.), this could affect also COG as well as GeoTIFF itself!. That could be the end of any of these initiatives (except if Adobe grants a licence to the OGC for free). Not everybody is for open standards. Lets hope that OGC diplomacy can navigate the situation.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/opengeospatial/geotiff/issues/103#issuecomment-931234138, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACDHG2XXY2YSCLJHZUQJRNLUERCKHANCNFSM5CWE45CA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

joanma747 commented 2 years ago

I understand your points with the GeoTIFF and I'm sure all was considered at the time.

The question now is if this is red flag for our activities in BigTIFF and COG. Can we assume that a similar approach will, in the end, work and continue the technical work?. I hope so!

rouault commented 2 years ago

The question now is if this is red flag for our activities in BigTIFF and COG. Can we assume that a similar approach will, in the end, work and continue the technical work?. I hope so!

My opinion: Adobe is just throwing FUD (Fear Uncertainty Doubt) to try to extract a few more $$$ from their 1994's Aldus acquisition, and they don't own any copyright or trademark on BigTIFF, nor as they do on GeoTIFF.

EmDevys commented 2 years ago

I do hope so, but for sure for bigTIFF the relationship with TIFF is strong, bigTIFF is only “64-bit”ness extension of TIFF. So it should be doubled-checked that there is no issue with Adobe, and that the bigTIFF communities accept the OGC offer/proposal. This includes Adobe. This is the merit of the public announce on the libtiff mailing-list. So I guess let’s double-check there is no real issue, and if there is any, as there is a need for bigTIFF (for geospatial, medical and probably also photographic and printing usages), I hope the people objecting will propose an alternate solution (Adobe-based, or based on some ISO committee, or any other).

For COG, the relationship with TIFF is not changing TIFF (or bigTIFF), it is basically some BP based on TIFF (and potentially GeoTIFF) for cloud-optimised usage of imagery. If there would be a need to insert some license / trademark information for the usage of TIFF in GeoTIFF, probably it would be careful to do the same for COG.

De : Joan Masó @.*** Envoyé : jeudi 30 septembre 2021 16:54 À : opengeospatial/geotiff Cc : Emmanuel Devys; Mention Objet : Re: [opengeospatial/geotiff] New working item for BigTIFF community standard (#103)

I understand your points with the GeoTIFF and I'm sure all was considered at the time.

The question now is if this is red flag for our activities in BigTIFF and COG. Can we assume that a similar approach will, in the end, work and continue the technical work?. I hope so!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/opengeospatial/geotiff/issues/103#issuecomment-931396531, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACDHG2UAE7MHMUR7HDLNOBDUER2W7ANCNFSM5CWE45CA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

ogcscotts commented 2 years ago

We seem to have a contact at Adobe again who is engaged in the discussion - OGC staff will see if we can get more firm legal insight from Adobe.

joanma747 commented 2 years ago

Is this settling the issue for good?. It looks like it does!!!

Leonard Rosenthol via Tiff tiff@lists.osgeo.org Oct 4, 2021, 9:14 PM (11 hours ago) to Emmanuel, Kemp, Joris, geotiff.swg@lists.opengeospatial.org, Scott, tiff@lists.osgeo.org

Scott Foshee (who manages all imaging standards for Adobe) and I spoke today and we would like the community to know that we are supportive of BigTIFF being moved from its current state into a formal open international standard by building on top of an already existing open standard for TIFF – that being TIFF/EP (https://www.iso.org/standard/29377.html). We would recommend that the work take place at ISO as well, so that others can then build on top of it as they have with TIFF/EP itself, and would also make it easier for Adobe to participate (if that is desirable by the community). However, we understand if the OGC or others would prefer some other formal SDO.

I will note that this is response is ONLY about BigTIFF and does not have any impact on any other conversations around GeoTIFF or others that may be ongoing.

Leonard

EmDevys commented 2 years ago

It seems that on the bigTIFF topic and its libtiff mailing-list / contributors there are some in favour of pushing bigTIFF specification outside of the OGC, and Adobe is proposing ISO (ISO 12234-2:2001 Electronic still-picture imaging — Removable memory — Part 2: TIFF/EP image data format) with the main drawback that the specification is not freely available, and many in favour of the OGC proposal, with a freely available open-standard. Is there any alternative (such as IEC – several standards are shared between ISO and IEC, IEC sometimes proposing a freely available version)? If not, I don’t know whether a ballot among the bigTIFF community (developers, users) – which is uneasy to delimitate, and probably best represented by the libtiff mailing-list) would be a valid process to elaborate a decision

De : Joan Masó @.*** Envoyé : mardi 5 octobre 2021 08:24 À : opengeospatial/geotiff Cc : Emmanuel Devys; Mention Objet : Re: [opengeospatial/geotiff] New working item for BigTIFF community standard (#103)

Is this settling the issue for good?. It looks like it does!!!

Leonard Rosenthol via Tiff @.**@.> Oct 4, 2021, 9:14 PM (11 hours ago) to Emmanuel, Kemp, Joris, @.**@.>, Scott, @.**@.>

Scott Foshee (who manages all imaging standards for Adobe) and I spoke today and we would like the community to know that we are supportive of BigTIFF being moved from its current state into a formal open international standard by building on top of an already existing open standard for TIFF – that being TIFF/EP (https://www.iso.org/standard/29377.html). We would recommend that the work take place at ISO as well, so that others can then build on top of it as they have with TIFF/EP itself, and would also make it easier for Adobe to participate (if that is desirable by the community). However, we understand if the OGC or others would prefer some other formal SDO.

I will note that this is response is ONLY about BigTIFF and does not have any impact on any other conversations around GeoTIFF or others that may be ongoing.

Leonard

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/opengeospatial/geotiff/issues/103#issuecomment-934104151, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACDHG2RNV3TU77BGNXZKEY3UFKKYPANCNFSM5CWE45CA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

joanma747 commented 2 years ago

Transcribing the discussion of the today telecon of the SWG:

We seem to have two requirements emerging from the discussion:

There are many precedents of standards done by the OGC that comply with both requirements. We propose to start in the OGC and propose it to ISO using JAG or a similar mechanism. There are concerns on how much extra work going to ISO this will require. It will require a country to propose it as a working item.

We need to start a discussion with the OGC staff about the best way to label the document and bring it to the ISO process (e.g. usign the JAG).

EmDevys commented 2 years ago

Just one comment on the OGC / ISO process. The JAG is facilitating / handling the coordination between OGC and ISO TC211 only, not all ISO committees. I guess submitting TIFF to ISO TC211 – Geographic information would not be well accepted by other communities. Adobe has proposed the ISO/TC 42 Photography. It could also be handled under a more neutral ISO/IEC JTC1 Information technology. I would be of interest to check whether OGC has a liaison with ISO/IEC JTC1 or ISO/TC 42 or any other committee, in order to facilitate communication and mutual understanding.

De : Joan Masó @.*** Envoyé : mardi 5 octobre 2021 15:25 À : opengeospatial/geotiff Cc : Emmanuel Devys; Mention Objet : Re: [opengeospatial/geotiff] New working item for BigTIFF community standard (#103)

We seem to have two requirements emerging from the discussion:

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/opengeospatial/geotiff/issues/103#issuecomment-934412585, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ACDHG2SDP5KKJR2DSEZREZ3UFL4CZANCNFSM5CWE45CA. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

desruisseaux commented 1 year ago

This issue is labelled as "done". Is this label accurate? From my reading of above discussion, the legal question seems to still open. If this issue is indeed done, can we close this issue with a comment summarizing the outcome?