search

Ymagis / ClairMeta

Clairmeta is a python package for Digital Cinema Package (DCP) probing and checking.
BSD 3-Clause "New" or "Revised" License
84 stars 22 forks source link

Error in dcp_check_subtitle.py test will crash on certain DCPs #155

Closed jamiegau closed 4 years ago

jamiegau commented 4 years ago

Hi, I have found current release will crash the subtitle test on certain DCPs.

Below is an example running against the EDCF SMPTE test DCP. EDCF-SMPTE-Bv2_TST_S_EN-EN_OV_51-HI-VI_2K_20161020_DTU_SMPTE_OV

It appear to be parsing the subtitles differently resulting in attempting to unique_chars.add(char) where char is a dict and not what is expected.. The error is self explanatory.

Jamess-MacBook-Pro:by_code jamieg$ python3 -m clairmeta.cli check -type dcp -progress VXAcsqRX7zWdFQakAhieTE/a/EDCF-SMPTE-Bv2_TST_S_EN-EN_OV_51-HI-VI_2K_20161020_DTU_SMPTE_OV
2020-08-05 17:52:29,926 - Clairmeta - INFO - Probing DCP : VXAcsqRX7zWdFQakAhieTE/a/EDCF-SMPTE-Bv2_TST_S_EN-EN_OV_51-HI-VI_2K_20161020_DTU_SMPTE_OV
2020-08-05 17:52:31,993 - Clairmeta - INFO - Total time : 2.07 seconds
2020-08-05 17:52:32,193 - Clairmeta - INFO - Checking DCP : VXAcsqRX7zWdFQakAhieTE/a/EDCF-SMPTE-Bv2_TST_S_EN-EN_OV_51-HI-VI_2K_20161020_DTU_SMPTE_OV
[  17.98 kiB in 0.00 sec (at 143.78 MBytes/s)      ] 100.00% - 41354b54-530a-4261-a883-cf7454591c66.xml
[  204.63 MiB in 0.37 sec (at 576.16 MBytes/s)     ] 100.00% - 64e3231b-b1aa-425d-bc56-bbafd1c3aed2.mxf
[  83.44 MiB in 0.16 sec (at 550.82 MBytes/s)      ] 100.00% - ac8a84b8-b617-4bb0-a95b-ca21d637e1d5.mxf
[  136.57 kiB in 0.00 sec (at 443.68 MBytes/s)     ] 100.00% - 3befc196-eb7d-4338-a15e-a8e1e5ebbf48.mxf
[  188.58 MiB in 0.34 sec (at 575.02 MBytes/s)     ] 100.00% - 14b3c5e6-6a2e-43c1-b6f6-6eb01e7dab65.mxf
[  79.04 MiB in 0.14 sec (at 572.75 MBytes/s)      ] 100.00% - bce2ca26-8830-4775-a45d-adbb713c4d88.mxf
[  140.38 kiB in 0.00 sec (at 349.93 MBytes/s)     ] 100.00% - 6c25c591-cca6-4508-97d0-c3986c0198ab.mxf
[  270.69 MiB in 0.49 sec (at 583.72 MBytes/s)     ] 100.00% - 492cf3e2-23f1-478d-acfe-4d9663e63dac.mxf
[  101.02 MiB in 0.19 sec (at 561.01 MBytes/s)     ] 100.00% - bce36a3f-ce91-4325-8463-0c3f498351d9.mxf
[  142.58 kiB in 0.00 sec (at 453.60 MBytes/s)     ] 100.00% - 3d95bd5a-6ae5-4a8b-b158-6d84f0ae9ec3.mxf
2020-08-05 17:52:33,986 - Clairmeta - ERROR - Check unknown error
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_base.py", line 119, in run_check
    check_res = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_subtitle.py", line 408, in check_subtitle_cpl_font_glyph
    unique_chars.add(char)
TypeError: unhashable type: 'dict'

2020-08-05 17:52:34,085 - Clairmeta - ERROR - Check unknown error
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_base.py", line 119, in run_check
    check_res = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_subtitle.py", line 408, in check_subtitle_cpl_font_glyph
    unique_chars.add(char)
TypeError: unhashable type: 'dict'

2020-08-05 17:52:34,124 - Clairmeta - ERROR - Check unknown error
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_base.py", line 119, in run_check
    check_res = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_subtitle.py", line 408, in check_subtitle_cpl_font_glyph
    unique_chars.add(char)
TypeError: unhashable type: 'dict'

2020-08-05 17:52:34,147 - Clairmeta - INFO - DCP : VXAcsqRX7zWdFQakAhieTE/a/EDCF-SMPTE-Bv2_TST_S_EN-EN_OV_51-HI-VI_2K_20161020_DTU_SMPTE_OV
2020-08-05 17:52:34,147 - Clairmeta - INFO - Size : 927.84 MiB
2020-08-05 17:52:34,147 - Clairmeta - INFO - Error(s) :
    check_assets_cpl_metadata - 41354b54-530a-4261-a883-cf7454591c66.xml (Asset 3befc196-eb7d-4338-a15e-a8e1e5ebbf48.mxf) : IntrinsicDuration metadata mismatch, CPL claims 552 but MXF 10000
    check_assets_cpl_metadata - 41354b54-530a-4261-a883-cf7454591c66.xml (Asset 6c25c591-cca6-4508-97d0-c3986c0198ab.mxf) : IntrinsicDuration metadata mismatch, CPL claims 504 but MXF 10000
    check_assets_cpl_metadata - 41354b54-530a-4261-a883-cf7454591c66.xml (Asset 3d95bd5a-6ae5-4a8b-b158-6d84f0ae9ec3.mxf) : IntrinsicDuration metadata mismatch, CPL claims 744 but MXF 10000
    check_certif_role - 165d11b6-ac01-4b59-a6b0-b6bda44842c5.pkl.xml (Certificate : 1109) : Expecting CS role in CommonName (SM.tdc-uk-user01)
    check_certif_role - 41354b54-530a-4261-a883-cf7454591c66.xml (Certificate : 1109) : Expecting CS role in CommonName (SM.tdc-uk-user01)
    check_subtitle_cpl_font_glyph - Check unknown error
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_base.py", line 119, in run_check
    check_res = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_subtitle.py", line 408, in check_subtitle_cpl_font_glyph
    unique_chars.add(char)
TypeError: unhashable type: 'dict'

    check_subtitle_cpl_font_glyph - Check unknown error
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_base.py", line 119, in run_check
    check_res = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_subtitle.py", line 408, in check_subtitle_cpl_font_glyph
    unique_chars.add(char)
TypeError: unhashable type: 'dict'

    check_subtitle_cpl_font_glyph - Check unknown error
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_base.py", line 119, in run_check
    check_res = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/clairmeta/dcp_check_subtitle.py", line 408, in check_subtitle_cpl_font_glyph
    unique_chars.add(char)
TypeError: unhashable type: 'dict'

2020-08-05 17:52:34,147 - Clairmeta - INFO - Warning(s) :
    check_dcnc_compliance - ContentTitle must have 12 parts, 11 found
Field Studio not found in ContentTitle
2020-08-05 17:52:34,147 - Clairmeta - INFO - Total check : 105
2020-08-05 17:52:34,148 - Clairmeta - INFO - Total time : 1.91 sec
2020-08-05 17:52:34,148 - Clairmeta - INFO - Validation : Fail

DCP - VXAcsqRX7zWdFQakAhieTE/a/EDCF-SMPTE-Bv2_TST_S_EN-EN_OV_51-HI-VI_2K_20161020_DTU_SMPTE_OV - Check failed
jamiegau commented 4 years ago

Hi, I created a pull request to try and fix this error, but it appears more sinister than that..

For example.. image

I take it that this is a multi line subtitle, one entry per line, tried to use asdcp-unwrap, but was not able to pull the subtitles out for some reason. I am not familiar with SMPTE subtitle files and extracting them.. But this image indicates, it may not be pulling the dtaa right into the object. The first line does not have the "Text" attribute. (Why the code fails) What's your opinion @remia ?

Will need to dig into other sections of the code to figure that one out..

remia commented 4 years ago

Hi @jamiegau,

I tried to download the DCP at https://www.smptedcp.com/resources/test-package but the subtitle are encrypted so I can't reproduce this error yet. I'm not sure who / what server the KDMs inside the package are targeting.

As for how we extract the subtitles, you should look at https://github.com/Ymagis/ClairMeta/blob/develop/clairmeta/dcp_check_subtitle.py#L187 and the unwrap_mxf helper functions.

Didn't really really took the time to understand your last post as I prefer to reproduce the bug myself first.

Thanks for the report

matmat commented 4 years ago

@jamiegau is it possible to make this package available for download somewhere? I would like to test what is going wrong. EDCF does not seem to have this package for download anymore.

The logs unfortunately doesn't say what the CPL uuid of your package is..

I have the following which is similair but not exactly the same:

ContentTitleText: EDCF-SMPTE-Bv2_TST_S_EN-EN_OV_51-HI-VI-Atmos_2K_20161020_DTU_SMPTE_OV CPL uuid: 362ec1b8-363c-439c-bda0-a5d0a181c8b0

The above package passes the clairmeta checks without crashing.

jamiegau commented 4 years ago

Those who want the DCP that causes the error, please send me an email and I'll mailsend it to you james . gardiner @ digitall net au

remia commented 4 years ago

Any update on this ?

jamiegau commented 4 years ago

Hj, I have a the main culprit DCP on its way to you now.. (Via a filemail.com) That DCP and another that clairmeta just dope out with an error on the probe.

James

remia commented 4 years ago

Thanks, please see PR #164 that should address this.