search

openpreserve / jpylyzer

JP2 (JPEG 2000 Part 1) validator and properties extractor. Jpylyzer was specifically created to check that a JP2 file really conforms to the format's specifications. Additionally jpylyzer is able to extract technical characteristics.
http://jpylyzer.openpreservation.org/
Other
69 stars 28 forks source link

Add plt, plm segments to properties output #170

Closed VirtualTim closed 3 years ago

VirtualTim commented 4 years ago

I didn't add any validation, as for large files this could take up a significant percentage of the output file. This will significantly increase the output size for datasets that use plt's. The output of a ~6GB file with plt's went from ~90KB to ~90MB.

The logic here could also be used to parse the plm segment. However I don't have any sample datastes that use that, so I didn't implement this.

VirtualTim commented 4 years ago

I've added all the iplt's as a single comma separated xml element, as adding a tag for each iplt will get ugly fast.

The output of the tilePart section will now look something like this (for datasets using plt's):

<tilePart>
  <sot>
    <lsot>10</lsot>
    <isot>0</isot>
    <psot>0</psot>
    <tpsot>0</tpsot>
    <tnsot>0</tnsot>
  </sot>
  <plt>
    <lplt>2955</lplt>
    <zplt>0</zplt>
    <iplt>086B,0682,05F3,17E6,1273,10DF,2853,20C0,1D33,1224,0F3F,0DED,12B0,0F93,0DFB,08FC,078A,0690,21C1,0A60,072F,225B,0C28,08A5,1F1E,0D37,0A19,25A4,0ED0,0AF3,24F5,0D4E,09CC,20F2,0C92,0946,2141,0CAC,0945,1F93,0CAD,0914,1FC6,0CF8,0949,1071,019A,00FB,1499,01E7,0151,18A1,02A3,0178,1612,02AF,01BE,12F1,0303,01A3,14AE,02DF,01A9,1972,0493,02D3,144A,01BB,017D,163D,026D,018C,1BCE,03B3,0211,1916,0357,0277,1852,0502,03B4,1908,04F6,02EB,1800,045E,0324,1714,043E,02AD,1B7A,02AC,01DF,1331,027F,01E1,192C,04B4,02BF,1AC8,036A,023B,1972,028B,01CA,18FA,025A,016A,17FE,02E1,01C0,12A9,01B6,013B,1761,0348,01F0,19BB,02B8,01F0,14DD,03C6,020F,1686,0322,0169,15FA,02E6,0181,1B33,0433,0260,155C,02C8,0175,15E3,0272,0141,14E6,036B,01FC,1368,03CD,025A,16B6,02CB,0173,1569,0333,01E8,12ED,02A0,0183,04C0,3C,26,05BD,31,03,0598,39,1B,05FC,31,0D,06FC,32,12,0550,2B,0B,0776,0A,03,05A4,38,23,0718,5C,1D,084F,45,10,0965,36,0B,0703,21,03,0177,0B,03,0240,03,03,01CF,03,03,04B1,12,0B,04BA,15,03,05F3,4A,10,0927,53,15,0698,44,2D,05BE,48,1E,0704,7E,24,090B,76,33,0772,18,0B,02FC,03,03,07A9,30,0D,0491,03,03,03A7,03,03,0482,1E,03,05F0,54,0D,0B07,0099,32,1157,48,1B,0606,55,16,087E,3C,12,11AC,00DD,6D,06FB,65,4B,09B8,5A,21,0B00,00AB,1C,034F,1A,13,069E,1F,03,049E,1B,03,0731,2D,03,0A93,00AA,2D,13F3,3C,0B,0AFF,0086,3F,0F6C,77,17,1332,00D8,6F,0708,53,46,04F7,50,03,08B0,0083,14,08FA,70,29,072F,32,0B,073B,53,0A,0670,2A,03,0A96,71,44,1338,46,27,057B,008C,45,0C70,008F,35,138C,00D9,67,07A2,3A,0B,04CA,44,0F,0ADB,78,17,08F8,3F,0E,097B,42,03,0912,28,03,09B2,7E,16,0AA9,33,0E,09EC,1B,1C,06A2,40,03,05D4,32,03,0D73,00A8,26,0743,3D,03,06B0,18,03,09DF,47,03,0989,40,16,080C,34,03,0984,2B,03,080F,2B,03,07CD,58,15,082D,1E,03,07C7,0B,03,05A7,03,03,0A8E,008D,15,0742,33,1C,0483,0B,03,062E,2B,03,056D,26,03,0530,0B,03,056A,03,03,05CA,20,03,061E,52,03,0634,16,03,0375,0B,03,05E2,18,03,092F,55,0B,04CF,0E,0B,066C,1C,03,03FC,3D,46,041C,28,0B,04C4,4F,2F,0314,2F,03,01E0,20,03,0204,1A,03,0459,34,03,0682,36,0B,0641,2C,03,056C,27,0B,0537,17,03,0566,0B,03,06D2,0F,03,04E6,2C,03,0540,69,03,0936,4C,03,0986,46,03,0A66,61,03,0A10,65,11,0C07,00BC,29,0911,50,0C,0A92,00AE,10,035F,2A,0B,049B,1F,03,0463,22,0D,052D,34,03,02F8,47,03,03C6,4D,28,05EC,4B,03,0848,0C,0B,0539,38,13,04CE,43,0B,04DC,2E,03,098B,008B,1C,02BE,18,03,067D,4F,03,03F6,1F,0D,0266,0B,03,0357,15,03,03A6,52,15,06A4,38,0B,04F4,33,03,07AB,008F,19,083B,00A9,16,06D6,45,13,07AF,46,0B,01E7,1C,0B,0096,03,03,00CA,0B,03,57,0B,03,0110,03,03,00B7,0B,03,60,03,03,44,03,03,0093,03,03,6B,03,03,6A,0B,03,62,03,03,1E,03,03,3D,03,03,31,03,03,008E,03,03,00AB,03,03,65,03,03,47,03,03,00E2,0B,03,7E,03,03,73,0F,03,00E5,03,03,00D2,03,03,24,03,03,4E,03,03,00F9,13,03,0092,0B,03,67,03,03,009B,0B,03,67,03,03,63,03,03,00AB,1A,03,00A1,18,03,011B,13,03,3A,0B,03,03,03,03,49,03,03,46,03,03,4E,0B,03,59,03,03,0088,1D,03,0097,03,03,00E6,0B,03,00BD,10,03,009E,15,03,01A6,10,03,0110,0B,03,27,03,03,00D7,03,03,75,03,03,0087,03,03,6D,03,03,00B9,03,03,5F,03,03,0095,03,03,00C0,03,03,00D0,03,03,00CC,0C,03,01CC,0B,03,25,03,03,3E,03,03,010C,0D,03,13,03,03,20,03,03,62,03,03,03,03,03,2D,0F,03,44,13,03,03,03,03,00EE,03,03,010A,03,03,2E,03,03,03,03,03,03,03,03,00E2,03,03,03,03,03,0C,03,03,03,03,03,00FA,03,03,00B6,03,03,0097,03,03,0096,03,03,00AD,17,03,03,03,03,2D,03,03,018F,0B,03,00D8,0B,03,68,0B,03,68,0E,0E,3C,0B,0D,5A,0D,03,6D,03,0D,0167,0E,03,0238,0D,03,00CE,03,03,1B,03,03,29,03,03,5F,03,03,00FD,03,03,00E9,03,03,0108,0C,03,0109,03,03,00B8,03,03,00E3,03,03,00C0,03,03,00C2,0B,03,0121,0E,03,00B4,0B,03,00DF,10,03,021C,0B,03,033C,03,03,0188,13,03,6E,03,03,0086,0B,03,65,03,03,00AD,0B,03,0237,0C,03,0272,16,03,010B,1B,03,0B,03,03,0083,03,03,00FB,03,03,012E,14,03,01AB,03,03,61,03,03,66,03,03,0D,03,03,44,03,03,5B,03,03,6F,03,03,0164,17,03,009F,0B,03,0B,03,03,01B8,10,03,03BE,03,03,03E8,0B,03,0122,0D,03,0B,03,03,00C9,03,03,02C6,03,03,0346,13,03,031A,29,03,011C,0B,03,21,03,03,01EE,0B,03,01B7,0E,03,019D,16,03,0187,1A,03,5D,03,03,00D6,03,03,01A3,0B,03,46,03,03,3D,03,03,00B4,03,03,019E,03,03,00AD,0B,03,48,03,03,01AA,1A,03,0399,0C,03,0403,13,03,01B8,19,03,20,03,03,72,03,03,03A5,03,03,0429,03,03,0325,2D,03,0141,0B,03,30,03,03,0127,14,03,018A,13,03,01AC,0B,03,0172,23,03,41,03,03,00CE,03,03,00F0,03,03,46,03,03,0091,12,03,009E,03,03,00CE,03,03,00C5,13,03,15,03,03,0225,10,03,03C6,03,03,03DA,13,03,0222,0D,03,016D,03,03,019B,1E,03,0399,1B,03,0415,03,03,02B6,26,03,013A,20,03,1C,03,03,00A2,0F,03,0164,1B,03,0162,03,03,00AD,0B,03,0082,0B,03,00B5,03,03,03,03,03,55,03,03,0F,03,03,17,03,03,0080,03,03,5F,03,03,00B8,0C,03,01FA,0B,03,03D0,03,03,03F5,10,03,010A,13,03,0086,0B,03,00DF,24,03,03A1,20,03,0402,03,03,02DF,23,03,012D,03,03,11,03,03,03,03,03,00D3,16,03,0140,1A,03,0143,0D,03,0154,16,03,0217,15,03,01AB,1A,03,0170,14,03,01AB,13,03,01D8,23,03,0155,0B,03,0112,03,03,19,03,03,023D,0C,03,032F,0B,03,03B5,03,03,39,15,03,45,0B,03,46,10,03,02CE,0F,03,03C8,03,03,02EC,26,03,011A,0B,03,20,03,03,69,10,03,014C,0F,03,01D2,17,03,00FA,0B,03,00F8,03,03,01DB,17,03,01D6,0B,03,0117,03,03,00E3,03,03,0222,03,03,019C,0C,03,0190,1E,03,00C4,03,03,01ED,0B,03,010E,03,03,01FB,0B,03,6F,0F,03,008D,0B,03,60,0D,03,00C1,0B,03,016D,03,03,0212,0F,03,00D4,15,03,0D,03,03,6F,03,03,0126,0D,03,01F2,14,03,0129,13,03,0106,1B,03,012C,03,03,0209,16,03,00C4,0B,03,015D,0B,0D,0169,03,03,015A,18,03,0124,1B,03,5F,03,03,01F6,03,03,01C5,03,03,56,03,03,0089,03,03,00DB,03,03,55,03,03,0095,03,03,0195,03,03,01FC,24,03,018C,03,03,26,03,03,01AF,03,03,010E,03,03,021B,0E,03,017E,03,03,47,03,03,0240,03,03,01CD,0E,03,00D7,03,03,013C,0C,03,0198,0F,03,01DA,03,03,0080,03,03,28,0B,03,0125,0B,03,0168,03,03,00B1,03,03,011A,03,03,0145,03,03,017A,03,03,00A5,03,03,0117,0B,03,0133,1E,03,012D,0B,03,25,03,03,010A,03,03,7D,03,03,0179,03,03,0130,03,03,70,03,03,0110,03,03,0132,03,03,0105,03,03,00E1,03,03,01B0,03,03,01BB,0B,03,009C,0B,03,3F,03,03,00E5,0B,03,0129,0D,03,52,03,03,00B9,03,03,00A7,03,03,75,03,03,28,03,03,0157,0B,03,0190,0E,03,00EA,03,03,0D,03,03,00C8,03,03,74,03,03,0147,03,03,00C1,03,03,00AC,0B,03,6C,03,03,0086,03,03,00D6,03,03,3A,03,03,0121,03,03,0190,0B,03,00C5,03,03,00AB,11,03,00E4,22,03,015B,0B,03,0091,03,03,1B,03,03,1E,03,03,3C,03,03,0097,03,03,013A,03,03,01AA,1E,03,00C5,03,03,1D,03,03,5D,03,03,03,03,03,0D,03,03,33,03,03,2A,03,03,23,03,03,28,03,03,44,03,03,10,03,03,19,03,03,62,03,03,16,03,03,2D,03,03,13,03,03,17,03,03,2E,03,03,1B,03,03,008E,03,03,00CE,03,03,61,03,03,4E,03,03,6E,03,03,4E,03,03,03,03,03,00C6,03,03,4B,13,03,0089,0B,03,009A,10,03,65,0B,03,63,03,03,6A,16,03,23,03,03,10,03,03,0082,03,03,0098,0B,03,74,13,03,5B,0B,03,72,03,03,54,03,03,72,03,03,58,03,03,1B,03,03,50,03,03,5A,03,03,33,03,03,26,03,03,00B8,03,03,03,03,03,012A,0F,03,0100,03,03,4F,03,03,61,0E,03,52,03,03,0112,0B,03,00C8,13,03,0100,1F,03,0089,0E,03,5C,0B,03,18,03,03,03,03,03,03,03,03,03,03,03,62,03,03,00D2,13,03,0103,0B,03,011E,14,03,0134,03,03,00BB,03,03,0088,0B,03,0128,03,03,00AC,03,03,39,03,03,4D,03,03,00AE,03,03,0181,03,03,0097,03,03,46,03,03,00AF,03,03,00A1,13,03,44,03,03,5E,03,03,00A2,0B,03,6A,03,03,75,03,03,76,0B,03,5A,0C,03,0130,0F,03,018B,11,03,00E8,21,03,69,1B,03,39,03,03,3C,03,03,33,03,03,00BF,0B,03,0097,03,03,0B,03,03,19,03,03,2C,0B,03,58,03,03,01A3,03,03,00B6,03,03,4D,03,03,13,03,03,3C,13,03,0084,03,03,00BB,03,03,009E,03,03,75,03,03,39,03,03,61,03,03,00E3,0B,03,4F,0B,03,009B,1F,03,00CA,1D,03,7F,0D,03,00E9,23,03,00B7,1E,03,00AF,21,03,50,0D,03,03,03,03,0112,03,03,0D,03,03,54,03,03,4E,03,03,0109,15,03,03,03,03,03,03,03,03,03,03,37,03,03,00E7,11,03,00AE,0B,03,008E,03,03,64,03,03,00A0,03,03,014E,03,03,4B,03,03,72,16,03,65,03,03,60,03,03,00E0,0E,03,0102,18,03,00FB,1D,03,22,03,03,03,03,03,49,03,03,77,03,03,00BD,10,03,03,03,03,31,03,03,1B,03,03,0E,03,03,29,03,03,03,03,03,52,03,03,7B,0B,03,008F,0E,03,00A3,03,03,0090,03,03,25,03,03,5F,03,03,5F,0E,03,59,0B,03,45,03,03,009D,0D,03,00A3,03,03,009F,03,03,69,03,03,03,03,03,0096,13,03,00DD,0E,03,00C1,0B,03,6F,0B,03,0B,03,03,2B,03,03,27,03,03,03,03,03,28,03,03,00C6,21,03,0173,13,03,0144,0B,03,28,03,03,00AA,03,03,00AB,0B,03,59,0B,03,7B,03,03,00C1,0D,03,0108,0F,03,00D3,03,03,00C8,0B,03,00D8,0B,03,3E,03,03,03,03,03,6A,03,03,7F,03,03,1C,03,03,00D6,03,03,03,03,03,20,03,03,19,03,03,17,03,03,0B,03,03,5C,03,03,009A,03,03,49,03,03,2B,0B,03,46,14,03,20,0E,03,76,11,03,009F,13,03,0085,0B,03,38,03,03,03,03,03,0082,0B,03,64,03,03,03,03,03,11,03,03</iplt>
  </plt>
</tilePart>
bitsgalore commented 4 years ago

Many thanks for this contribution, and your other contributions as well. This looks like something that would be good to include in the upcoming jpylyzer 2.1 release, which is planned for November.

BTW if you're looking for samples files with a plm marker segment, check out file "oj-plm-main-header.jp2" in the jpylyzer-test-files dataset.

VirtualTim commented 4 years ago

No worries. I think it's quite handy having a jp2 metadata/validation tool written outside of the other existing implementations.

I could take a quick look at the plm segment tomorrow, I think the complex stuff is already in this PR. Would you prefer me to tack it onto this one, or create a separate PR?

Also I see you don't have any specific plt datasets. I could re-encode oj-plm-main-header.jp2 ot oj-plt-tilepart-header.jp2, if that would be useful?

bitsgalore commented 4 years ago

Hi Tim,

Thanks for having a go at the PLM code. I don't really have a preference as to adding it to the current PR vs creating a separate one. I would suggest to do whatever works best for you.

The dataset already contains 3 images with the PLT segment (sentinel.jp2; palettedImage.jp2 and reference.jp2), so as far as I'm concerned there's no need for another test image for this feature.

VirtualTim commented 4 years ago

With these change the output of validating oj-plm-main-header.jp2 now includes:

<plm>
  <lplm>12</lplm>
  <zplm>0</zplm>
  <nplm>0</nplm>
  <iplm>00,01,00,05,04,04,00,00</iplm>
</plm>

I don't know if this is actually correct, as the other tool I've been using for validation (GDALs dump_jp2.py) doesn't log this.

VirtualTim commented 4 years ago

@bitsgalore I think this PR is all done now.

bitsgalore commented 3 years ago

Merged into jpylyzer 2.1 dev branch https://github.com/openpreserve/jpylyzer/commit/5756cb88e3a1d9193549a2dee76a813aa3adac44

VirtualTim commented 3 years ago

There's something a bit odd about how this and #167 was merged, it looks like the PR was rejected. I don't think it matters, but may just confuse people who look at the list of PRs.

bitsgalore commented 3 years ago

@VirtualTim Not 100% sure, but this might be because I merged both PRs into the the development branch for jpylyzer 2.1 instead of the master branch. Once this release is complete, I will merge this branch back into the master branch, which (I expect?) should fix this. This might take a while though, because the actual release is scheduled for q4 of this year.