Unlisted Dicom tags cause KeyError, and failure for all other purposes

[GoogleCodeExporter]

What steps will reproduce the problem?
1.  For dicom files with tags that are not listed in the _dicom_dict.py file, 
the files can not be read, even if the dicom tags are not of any use to the 
application.
2.  Try to read a dicom file that has a tag (FFFF,FFFF) or (3F03,1001)  (I 
don't know or care what those tags hold)

What is the expected output? What do you see instead?
I would expect the program to ignore these tags, or to fill them out with some 
clearly bogus placeholders to indicate that they are going to silently be 
ignored.

What I see instead is that the dicom.file_read() throws a KeyError stating that 
the key is not in the pre-defined dictionary, and the parts of the file that 
are extractable are not available for querying.

What version of the product are you using?
I tried both 0.9.4 and 0.9.5rc1.

Please provide any additional information below.

A hack that works for one of the data sets that was causing problems was to add 
the missing values to the dictionary.  This is not very satisfying because I 
can not anticipate all the other instances of not reported tags that will 
encountered in the future.
[johnsonhj@neuron mpy-svn-stats]$ tail -4 
/Library/Python/2.6/site-packages/dicom/_dicom_dict.py
'7Fxx0040': ('OW', '1', "Variable Coefficients SDDN", 'Retired'),
'3F031001': ('NONE', '1', "Unknown Item", ''),
'FFFFFFFF': ('NONE', '1', "Unknown Item", ''),
}

===========================
Proposal:

In dicom/datadict.py

replace: 
 41 def get_entry(tag):
 42     """Return the tuple (VR, VM, name, is_retired) from the DICOM dictionary
 43 ····
 44     If the entry is not in the main dictionary, check the masked ones,
 45     e.g. repeating groups like 50xx, etc.
 46     """
 47     tag = Tag(tag)
 48     try:
 49         return DicomDictionary[tag]
 50     except KeyError:
 51         mask_x = mask_match(tag)
 52         if mask_x:
 53             return RepeatersDictionary[mask_x]
 54         else:
**55**          raise KeyError, "Tag %s not found in DICOM dictionary" % 
Tag(tag)
 56

=============  with =======================================
 41 def get_entry(tag):
 42     """Return the tuple (VR, VM, name, is_retired) from the DICOM dictionary
 43 ····
 44     If the entry is not in the main dictionary, check the masked ones,
 45     e.g. repeating groups like 50xx, etc.
 46     """
 47     tag = Tag(tag)
 48     try:
 49         return DicomDictionary[tag]
 50     except KeyError:
 51         mask_x = mask_match(tag)
 52         if mask_x:
 53             return RepeatersDictionary[mask_x]
 54         else:
**55**          return ('NONE', '1', "Unknown Item : "+str(tag), '')
 56

Original issue reported on code.google.com by hans.j.j...@gmail.com on 18 Sep 2010 at 1:27

[GoogleCodeExporter]

This is strange. The two tags mention that are giving problems are private tags 
(group number is odd) which pydicom has always been able to ignore, since many 
would not be in the private dictionary anyway.

In pydicom's testfiles directory, I tried the following:
>>> ds=dicom.read_file("CT_small.dcm")
>>> ds.AddNew((0x3F03,0x1001),"UN", "hi")
>>> ds.save_as("3f03")
>>> ds2=dicom.read_file("3f03")
>>> ds2
and it reads okay, and displays the new tag fine. Same for the (FFFF,FFFF) tag, 
but group FFFF is reserved by the DICOM standard (but not used as far as I can 
recall) so that is a strange one.

Maybe there is something different about your file. Is it implicit or explicit 
VR? Could you provide an anonymized example file for this? 

Original comment by darcymason@gmail.com on 18 Sep 2010 at 4:06

• Changed state: Accepted

[GoogleCodeExporter]

Here is a dicom file that should allow you to reproduce the error.

Original comment by OScu...@gmail.com on 1 Oct 2010 at 9:16

Attachments:

• IM000001

[GoogleCodeExporter]

I was able to reproduce the error using the supplied file. I looked at the hex 
dump -- the data element is unusual. The file is implicit VR, and the 
troublesome element has undefined length (FFFFFFFF) followed by a sequence item 
marker, so it appears to be a SQ. Tried 'SQ' instead of 'NONE' in your 
_dicom_dict line, and it read and displayed okay, but other than the first 
item, the sequence looks a bit strange.

I'll need to do some more hex dump inspection to really understand what the 
original file is trying to do. But I'll hold on that until after release of 
pydicom 0.9.5 as I don't want to introduce and new bugs with the fix for this.

Original comment by darcymason@gmail.com on 3 Oct 2010 at 6:23

[GoogleCodeExporter]

Has this issue been resolved in 0.9.5? I'm encountering something that's 
probably related in one of my Philips MR DICOM files. Could provide file if 
needed.

Original comment by stefan.r...@gmail.com on 18 Feb 2011 at 12:45

[GoogleCodeExporter]

No, it is still present in 0.9.5. I believe these issues (issue 91, issue 97, 
and this one) are probably related.

Another file would be great though; it can help to see several files to see the 
commonalities

Original comment by darcymason@gmail.com on 18 Feb 2011 at 1:17

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

see attached. following the fix from issue 97 (thanks for pointing me there - 
should have looked more carefully) I can read the file in all its beauty. But 
why are private tags (x2001XXXX and x2005XXXX) looked for in the dictionary 
which has little chance of knowing about them?

Original comment by stefan.r...@gmail.com on 18 Feb 2011 at 6:26

Attachments:

• IMAGE8

[GoogleCodeExporter]

Got the file (thanks) and confirmed the error. I've also tried this in pydicom 
0.9.3, and the error does not occur there. So I think it is related to the 
switch to "raw data elements" done as part of the faster reading efforts. As 
part of that, SQ items were not parsed at first unless they were undefined 
length. Will look into it further.

Original comment by darcymason@gmail.com on 18 Feb 2011 at 12:08

[GoogleCodeExporter]

This issue was closed by revision 861d859b4f.

Original comment by darcymason@gmail.com on 21 Feb 2011 at 5:48

• Changed state: Fixed