search

geotiffjs / geotiff.js

geotiff.js is a small library to parse TIFF files for visualization or analysis. It is written in pure JavaScript, and is usable in both the browser and node.js applications.
https://geotiffjs.github.io/
MIT License
878 stars 183 forks source link

replaced txml with xml-utils #245

Closed DanielJDufour closed 3 years ago

DanielJDufour commented 3 years ago

This pull request introduces xml-utils and removes txml. xml-utils doesn't have any dependencies (other than the test runner flug for development) and it doesn't use any NodeJS builtins. There shouldn't be any configuration required for use with webpack or any other bundler. This should fix https://github.com/geotiffjs/geotiff.js/issues/241

Note: I didn't see any tests covering getGDALMetadata, but I might have missed them. I don't have time at the moment to add any more tests (prepping for FOSS4G!), but would have time in a week or so.

DanielJDufour commented 3 years ago

Sounds good. I'll start working on writing some tests next week that cover getGDALMetadata.

ahocevar commented 3 years ago

I look forward to this getting merged - my attempt to fix the txml import issues in #243 unfortunately does not work correctly in Webpack 5.

constantinius commented 3 years ago

@DanielJDufour do you think you can provide some tests?

DanielJDufour commented 3 years ago

@constantinius , yes, working on adding some tests now :-)

tschaub commented 3 years ago

In case it is useful, here is a rebased version of your original commit: https://github.com/geotiffjs/geotiff.js/commit/5057c11b8cae5c99ce4ed3e35e24e74a470ce3a5

And a commit that adds a test: https://github.com/geotiffjs/geotiff.js/commit/10fe751d3f1da727185376c22919af47a465d842

The test reads metadata from the TIFF in this archive: metadata.zip

These commits are part of this branch: https://github.com/tschaub/geotiff.js/tree/xml-utils

constantinius commented 3 years ago

@DanielJDufour @tschaub

I'm using this script to set up test data. Maybe we can set GDAL metadata there? I think adding metadata is quite easy using gdalinfo -stats right? I'm trying to minimize downloads from outside of the repo as they may vanish at some point.

tschaub commented 3 years ago

@constantinius - I agree it makes sense to reduce downloads.

I created the metadata.tiff above with:

gdal_translate nasa_raster.tiff metadata.tiff -mo "FOO=BAR"

I'm not 100% clear on when stats shows up as GDAL_METADATA and when it might get added as non-gdal-specific tags. Poking around some of the other data in the test directory, I found some with stats but without a GDAL_METADATA tag.

DanielJDufour commented 3 years ago

I created https://github.com/GeoTIFF/test-data/tree/main/files which holds a bunch of smaller tiffs from various sources that have GDAL_METADATA. I created this before I saw your message about downloading externally, but they are relatively small (max 11MB), so you should be able to copy over into this repo's Github files once I submit the PR. For what it's worth, they all come from real-world uses cases / users of GeoRasterLayer.

constantinius commented 3 years ago

Hm, okay, I think we could either borrow one or two of those (we don't need that many tests anyways, right?) or download them in the setup script (but only if the commit hash is explicit). Sounds fair?

DanielJDufour commented 3 years ago

I'm testing with abetow-ERD2018-EBIRD_SCIENCE-20191109-a5cf4cb2_hr_2018_abundance_median.tiff, which I believe comes from the eBird STAC Catalog on stacindex.org.

When I don't pass in sample, getGDALMetadata() returns 

{
  STATISTICS_MAXIMUM: '10.436447143555',
  STATISTICS_MEAN: 'nan',
  STATISTICS_MINIMUM: '0',
  STATISTICS_STDDEV: 'nan'
}

which is the the results for the last sample and not the whole data set.

Question If sample = null, should we ignore items where the sample attribute is set, so we return truly dataset-level stats or nothing. EDITED: With the exception where there appears to be only one band. In that case <Item sample="0"> does represent the dataset-level, too

Here's the full GDAL_METADATA:

<GDALMetadata>
  <Item name="STATISTICS_MAXIMUM" sample="0">9.7544279098511</Item>
  <Item name="STATISTICS_MEAN" sample="0">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="0">0</Item>
  <Item name="STATISTICS_STDDEV" sample="0">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="1">8.0268106460571</Item>
  <Item name="STATISTICS_MEAN" sample="1">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="1">0</Item>
  <Item name="STATISTICS_STDDEV" sample="1">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="2">7.5575194358826</Item>
  <Item name="STATISTICS_MEAN" sample="2">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="2">0</Item>
  <Item name="STATISTICS_STDDEV" sample="2">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="3">8.1030435562134</Item>
  <Item name="STATISTICS_MEAN" sample="3">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="3">0</Item>
  <Item name="STATISTICS_STDDEV" sample="3">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="4">7.8772916793823</Item>
  <Item name="STATISTICS_MEAN" sample="4">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="4">0</Item>
  <Item name="STATISTICS_STDDEV" sample="4">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="5">8.1386680603027</Item>
  <Item name="STATISTICS_MEAN" sample="5">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="5">0</Item>
  <Item name="STATISTICS_STDDEV" sample="5">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="6">8.0033168792725</Item>
  <Item name="STATISTICS_MEAN" sample="6">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="6">0</Item>
  <Item name="STATISTICS_STDDEV" sample="6">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="7">7.8162579536438</Item>
  <Item name="STATISTICS_MEAN" sample="7">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="7">0</Item>
  <Item name="STATISTICS_STDDEV" sample="7">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="8">7.7347140312195</Item>
  <Item name="STATISTICS_MEAN" sample="8">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="8">0</Item>
  <Item name="STATISTICS_STDDEV" sample="8">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="9">7.3182582855225</Item>
  <Item name="STATISTICS_MEAN" sample="9">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="9">0</Item>
  <Item name="STATISTICS_STDDEV" sample="9">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="10">7.2544522285461</Item>
  <Item name="STATISTICS_MEAN" sample="10">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="10">0</Item>
  <Item name="STATISTICS_STDDEV" sample="10">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="11">6.4744348526001</Item>
  <Item name="STATISTICS_MEAN" sample="11">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="11">0</Item>
  <Item name="STATISTICS_STDDEV" sample="11">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="12">5.9061455726624</Item>
  <Item name="STATISTICS_MEAN" sample="12">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="12">0</Item>
  <Item name="STATISTICS_STDDEV" sample="12">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="13">6.0884804725647</Item>
  <Item name="STATISTICS_MEAN" sample="13">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="13">0</Item>
  <Item name="STATISTICS_STDDEV" sample="13">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="14">6.1409721374512</Item>
  <Item name="STATISTICS_MEAN" sample="14">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="14">0</Item>
  <Item name="STATISTICS_STDDEV" sample="14">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="15">6.9471464157104</Item>
  <Item name="STATISTICS_MEAN" sample="15">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="15">0</Item>
  <Item name="STATISTICS_STDDEV" sample="15">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="16">6.7919926643372</Item>
  <Item name="STATISTICS_MEAN" sample="16">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="16">0</Item>
  <Item name="STATISTICS_STDDEV" sample="16">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="17">7.1424775123596</Item>
  <Item name="STATISTICS_MEAN" sample="17">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="17">0</Item>
  <Item name="STATISTICS_STDDEV" sample="17">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="18">7.8350172042847</Item>
  <Item name="STATISTICS_MEAN" sample="18">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="18">0</Item>
  <Item name="STATISTICS_STDDEV" sample="18">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="19">8.4386291503906</Item>
  <Item name="STATISTICS_MEAN" sample="19">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="19">0</Item>
  <Item name="STATISTICS_STDDEV" sample="19">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="20">8.8700618743896</Item>
  <Item name="STATISTICS_MEAN" sample="20">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="20">0</Item>
  <Item name="STATISTICS_STDDEV" sample="20">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="21">10.158813476562</Item>
  <Item name="STATISTICS_MEAN" sample="21">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="21">0</Item>
  <Item name="STATISTICS_STDDEV" sample="21">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="22">10.562696456909</Item>
  <Item name="STATISTICS_MEAN" sample="22">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="22">0</Item>
  <Item name="STATISTICS_STDDEV" sample="22">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="23">9.9366149902344</Item>
  <Item name="STATISTICS_MEAN" sample="23">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="23">0</Item>
  <Item name="STATISTICS_STDDEV" sample="23">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="24">10.163953781128</Item>
  <Item name="STATISTICS_MEAN" sample="24">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="24">0</Item>
  <Item name="STATISTICS_STDDEV" sample="24">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="25">9.8941698074341</Item>
  <Item name="STATISTICS_MEAN" sample="25">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="25">0</Item>
  <Item name="STATISTICS_STDDEV" sample="25">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="26">9.4827003479004</Item>
  <Item name="STATISTICS_MEAN" sample="26">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="26">0</Item>
  <Item name="STATISTICS_STDDEV" sample="26">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="27">8.7912473678589</Item>
  <Item name="STATISTICS_MEAN" sample="27">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="27">0</Item>
  <Item name="STATISTICS_STDDEV" sample="27">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="28">8.0593023300171</Item>
  <Item name="STATISTICS_MEAN" sample="28">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="28">0</Item>
  <Item name="STATISTICS_STDDEV" sample="28">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="29">8.3999433517456</Item>
  <Item name="STATISTICS_MEAN" sample="29">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="29">0</Item>
  <Item name="STATISTICS_STDDEV" sample="29">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="30">8.6393766403198</Item>
  <Item name="STATISTICS_MEAN" sample="30">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="30">0</Item>
  <Item name="STATISTICS_STDDEV" sample="30">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="31">9.123348236084</Item>
  <Item name="STATISTICS_MEAN" sample="31">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="31">0</Item>
  <Item name="STATISTICS_STDDEV" sample="31">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="32">9.5504579544067</Item>
  <Item name="STATISTICS_MEAN" sample="32">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="32">0</Item>
  <Item name="STATISTICS_STDDEV" sample="32">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="33">10.156976699829</Item>
  <Item name="STATISTICS_MEAN" sample="33">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="33">0</Item>
  <Item name="STATISTICS_STDDEV" sample="33">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="34">10.826090812683</Item>
  <Item name="STATISTICS_MEAN" sample="34">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="34">0</Item>
  <Item name="STATISTICS_STDDEV" sample="34">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="35">10.796345710754</Item>
  <Item name="STATISTICS_MEAN" sample="35">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="35">0</Item>
  <Item name="STATISTICS_STDDEV" sample="35">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="36">11.531585693359</Item>
  <Item name="STATISTICS_MEAN" sample="36">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="36">0</Item>
  <Item name="STATISTICS_STDDEV" sample="36">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="37">11.836143493652</Item>
  <Item name="STATISTICS_MEAN" sample="37">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="37">0</Item>
  <Item name="STATISTICS_STDDEV" sample="37">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="38">12.17227268219</Item>
  <Item name="STATISTICS_MEAN" sample="38">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="38">0</Item>
  <Item name="STATISTICS_STDDEV" sample="38">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="39">12.052537918091</Item>
  <Item name="STATISTICS_MEAN" sample="39">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="39">0</Item>
  <Item name="STATISTICS_STDDEV" sample="39">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="40">11.457691192627</Item>
  <Item name="STATISTICS_MEAN" sample="40">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="40">0</Item>
  <Item name="STATISTICS_STDDEV" sample="40">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="41">11.130515098572</Item>
  <Item name="STATISTICS_MEAN" sample="41">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="41">0</Item>
  <Item name="STATISTICS_STDDEV" sample="41">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="42">11.44605255127</Item>
  <Item name="STATISTICS_MEAN" sample="42">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="42">0</Item>
  <Item name="STATISTICS_STDDEV" sample="42">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="43">11.915058135986</Item>
  <Item name="STATISTICS_MEAN" sample="43">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="43">0</Item>
  <Item name="STATISTICS_STDDEV" sample="43">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="44">11.952184677124</Item>
  <Item name="STATISTICS_MEAN" sample="44">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="44">0</Item>
  <Item name="STATISTICS_STDDEV" sample="44">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="45">12.19713306427</Item>
  <Item name="STATISTICS_MEAN" sample="45">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="45">0</Item>
  <Item name="STATISTICS_STDDEV" sample="45">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="46">12.336188316345</Item>
  <Item name="STATISTICS_MEAN" sample="46">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="46">0</Item>
  <Item name="STATISTICS_STDDEV" sample="46">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="47">12.513179779053</Item>
  <Item name="STATISTICS_MEAN" sample="47">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="47">0</Item>
  <Item name="STATISTICS_STDDEV" sample="47">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="48">12.301235198975</Item>
  <Item name="STATISTICS_MEAN" sample="48">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="48">0</Item>
  <Item name="STATISTICS_STDDEV" sample="48">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="49">11.876184463501</Item>
  <Item name="STATISTICS_MEAN" sample="49">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="49">0</Item>
  <Item name="STATISTICS_STDDEV" sample="49">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="50">11.081751823425</Item>
  <Item name="STATISTICS_MEAN" sample="50">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="50">0</Item>
  <Item name="STATISTICS_STDDEV" sample="50">nan</Item>
  <Item name="STATISTICS_MAXIMUM" sample="51">10.436447143555</Item>
  <Item name="STATISTICS_MEAN" sample="51">nan</Item>
  <Item name="STATISTICS_MINIMUM" sample="51">0</Item>
  <Item name="STATISTICS_STDDEV" sample="51">nan</Item>
</GDALMetadata>
DanielJDufour commented 3 years ago

Hi, @constantinius and @tschaub and @ahocevar , I added some tests. Let me know if there's anything else that you would like me to do. Happy to remove/add more as you prefer. I'm not sure if you want me to add the FOO=BAR test to this PR or have @constantinius do that.

Honestly, I added probably more tests than necessary for my peace of mind. But everything is all green now, so we can scale back if you want.

constantinius commented 3 years ago

@DanielJDufour Looks good, thanks! I just added a note to use a specific commit hash for the test data download. Otherwise I really like it.

DanielJDufour commented 3 years ago

@constantinius , sorry I missed your comment earlier. Should be fixed now.

p.s. that was really cool. I didn't realize you could download a zip from a specific commit/version of a repo until you told me!

constantinius commented 3 years ago

Great, thanks for the contribution!

tschaub commented 3 years ago

Looks great, @DanielJDufour! Looking forward to making use of this when released.

bjornharrtell commented 3 years ago

@DanielJDufour nice work!

constantinius commented 3 years ago

This is now included in v1.0.8

sunkeWEB commented 3 years ago

可以降低ol版本 "ol": "6.3.1"