mof compile failure with pragma namespace and invalid namespace

KSchopmeyer commented 3 years ago

pywbem_mock fails with either a ModelError or CIMError naming a profile class if the MOF includes a pragma statment that refers to a namespace that does not exist or that does exist and no fleshed out interop namespace exists to be able to process the dmtf defined create namespace methods.

The errors are:

If the namespace does not exist:

ModelError: Interop namespace could not be determined (tried ['interop', 'root/interop', 'root/PG_Interop'])
If the interop namespace exists but is not complete:

CIMError: 6 (CIM_ERR_NOT_FOUND): Class 'CIM_RegisteredProfile' not found in namespace 'interop'.

The first case can be duplicated with the following mof file


    // Test mof that fails when compiled with a pywbem mock definition
    // pywbemcli -m test_nspragma.mof
    // Pragma defines namespace that does not exist
    #pragma namespace ("root/blah")

   #pragma locale ("en_US")
   Qualifier Association : boolean = false,
       Scope(association),
       Flavor(DisableOverride, ToSubclass);

and pywbemcli called as follows

Fails as follows:

ppp -m test_nspragma.mof class enumerate -MOF compile failed: Interop namespace could not be determined (tried ['interop', 'root/interop', 'root/PG_Interop']) Aborted!



See the next comment for the other tests and result

This is because the MOF compiler qualifier declaration directly calls the server.create_namespace which is expecting some means of communicating with the server to create a namespace but when there is no interop namespace, or there is just an empty  interop namespace the compiler fails with exceptions like ModelError or CIMError 

Since this set of errors is difficult to tie to the original issue, a namespace pragma in MOF being compiled without a corresponding namespace It would be good if the user received an exception that helped them understand what the error was.

NOTE: THis should be moved to pywbem. and post the mof that causes this.

KSchopmeyer commented 3 years ago

Failure when a mock python script that uses mof with namespace pragma with invalid namespace failure

NOTE: Works if namespace is "root/cinv2"

Mock file

def setup(conn, server, verbose):
    # pylint: disable=unused-argument
    mof = """
        #pragma namespace ("root/blah")
        #pragma locale ("en_US")
        Qualifier Association : boolean = false,
            Scope(association),
            Flavor(DisableOverride, ToSubclass);
    """
    conn.add_namespace('interop')
    conn.compile_mof_string(mof, verbose=True)

Command and result


    pywbemcli -m test_nspragma.py

/Mock script test_nspragma.py failed:
Traceback (most recent call last):

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_inmemoryrepository.py", line 243, in validate_namespace
    self._repository[namespace]

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_nocasedict.py", line 78, in __getitem__
    return super(NocaseDict, self).__getitem__(key)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/nocasedict/_nocasedict.py", line 304, in __getitem__
    raise key_error  # pylint: disable=raise-missing-from

KeyError: "Key 'root/blah' not found"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_baseprovider.py", line 149, in validate_namespace
    self.cimrepository.validate_namespace(namespace)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_inmemoryrepository.py", line 246, in validate_namespace
    format(namespace))

KeyError: 'Namespace "root/blah" does not exist in repository'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_mof_compiler.py", line 822, in p_mp_setQualifier
    p.parser.handle.SetQualifier(qualdecl, namespace=ns)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_mockmofwbemconnection.py", line 373, in SetQualifier
    self.conn.SetQualifier(qual, namespace=ns)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_cim_operations.py", line 9681, in SetQualifier
    has_return_value=False)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/mock/mock.py", line 1100, in __call__
    return _mock_self._mock_call(*args, **kwargs)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/mock/mock.py", line 1104, in _mock_call
    return _mock_self._execute_mock_call(*args, **kwargs)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/mock/mock.py", line 1167, in _execute_mock_call
    result = effect(*args, **kwargs)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_wbemconnection_mock.py", line 1081, in _mock_imethodcall
    result = methodnameattr(namespace, **params)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_wbemconnection_mock.py", line 1530, in _imeth_SetQualifier
    QualifierDeclaration=params['QualifierDeclaration'])

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_mainprovider.py", line 1139, in SetQualifier
    self.validate_namespace(namespace)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_baseprovider.py", line 154, in validate_namespace
    namespace))

pywbem._exceptions.CIMError: 3 (CIM_ERR_INVALID_NAMESPACE): Namespace does not exist in CIM repository: 'root/blah'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_baseprovider.py", line 398, in get_class
    klass = class_store.get(classname, copy=True)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_inmemoryrepository.py", line 105, in get
    .format(name, self._cim_object_type))

KeyError: "Name CIM_RegisteredProfile not in <class 'pywbem._cim_obj.CIMClass'> object store"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/kschopmeyer/pywbem/pywbemtools/pywbemtools/pywbemcli/mockscripts/__init__.py", line 119, in setup_script
    module.setup(conn=conn, server=server, verbose=verbose)

  File "test_nspragma.py", line 21, in setup
    conn.compile_mof_string(mof, verbose=True)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_wbemconnection_mock.py", line 619, in compile_mof_string
    mofcomp.compile_string(mof_str, namespace)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_mof_compiler.py", line 2877, in compile_string
    rv = self.parser.parse(mof, lexer=lexer)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/ply/yacc.py", line 333, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/ply/yacc.py", line 1120, in parseopt_notrack
    p.callable(pslice)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_mof_compiler.py", line 828, in p_mp_setQualifier
    p.parser.server.create_namespace(ns)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_server.py", line 426, in create_namespace
    ws_profiles = self.get_selected_profiles('DMTF', 'WBEM Server')

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_server.py", line 656, in get_selected_profiles
    'RegisteredOrganization')

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_valuemapping.py", line 262, in for_property
    IncludeQualifiers=True)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem/_cim_operations.py", line 9174, in GetClass
    PropertyList=PropertyList)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/mock/mock.py", line 1100, in __call__
    return _mock_self._mock_call(*args, **kwargs)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/mock/mock.py", line 1104, in _mock_call
    return _mock_self._execute_mock_call(*args, **kwargs)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/mock/mock.py", line 1167, in _execute_mock_call
    result = effect(*args, **kwargs)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_wbemconnection_mock.py", line 1081, in _mock_imethodcall
    result = methodnameattr(namespace, **params)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_wbemconnection_mock.py", line 1455, in _imeth_GetClass
    PropertyList=params.get('PropertyList', None))

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_mainprovider.py", line 800, in GetClass
    property_list=PropertyList)

  File "/home/kschopmeyer/virtualenvs/pycli3/lib/python3.6/site-packages/pywbem_mock/_baseprovider.py", line 403, in get_class
    classname, namespace))

pywbem._exceptions.CIMError: 6 (CIM_ERR_NOT_FOUND): Class 'CIM_RegisteredProfile' not found in namespace 'interop'.

KSchopmeyer commented 3 years ago

We should consider using the cause special attribute to stop the chain of exceptions.

andy-maier commented 3 years ago

I believe the error is fixed when using pywbem PR https://github.com/pywbem/pywbem/pull/2581.

That pywbem change will be picked up by pywbemtools as follows: pip -y uninstall pywbem make install after that pywbem PR is merged.

Please verify whether the error messages are resolved by that.

The exception handling in pywbemtools still may need to be improved by handling the outermost exception of the chain you saw.

andy-maier commented 3 years ago

When running your mock script shown above with the latest code (pywbem PR 2581 and pywbemtools PR 878), I get a different error behavior. It now reports that the CIM_ObjectManager class is not found in the interop namespace. Which is true. This happens during the compile_file() call in the setup() function of the mock script. Apparently there is no exception handling around that script in order to let programming errors surface with traceback, but it is a bit inconvenient in this case.

This happens when processing the direct approach for creating the namespace in WBEMServer.create_server() when it determines which creation class it should use (we use PG_Namespace for Pegasus and CIM_Namespace for all others). It does that by determining the WBEM server brand, which it does by looking at the CIM_ObjectManager instance representing it.

Here is the full traceback (tests/unit/interop_noprovider_mock_script.py is your mock script shown above):

$ pywbemcli -m tests/unit/interop_noprovider_mock_script.py class tree
-Generating LALR tables
Target namespace 'root/cimv2'
Switching target namespace to 'root/blah'
Setting qualifier 'Association' in namespace 'root/blah'
Creating namespace 'root/blah'
Mock script tests/unit/interop_noprovider_mock_script.py failed:
Traceback (most recent call last):

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_inmemoryrepository.py", line 243, in validate_namespace
    self._repository[namespace]

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_nocasedict.py", line 78, in __getitem__
    return super(NocaseDict, self).__getitem__(key)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/nocasedict/_nocasedict.py", line 304, in __getitem__
    raise key_error  # pylint: disable=raise-missing-from

KeyError: "Key 'root/blah' not found"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_baseprovider.py", line 149, in validate_namespace
    self.cimrepository.validate_namespace(namespace)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_inmemoryrepository.py", line 245, in validate_namespace
    raise KeyError('Namespace "{}" does not exist in repository'.

KeyError: 'Namespace "root/blah" does not exist in repository'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_mof_compiler.py", line 836, in p_mp_setQualifier
    p.parser.handle.SetQualifier(qualdecl, namespace=ns)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_mockmofwbemconnection.py", line 373, in SetQualifier
    self.conn.SetQualifier(qual, namespace=ns)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_cim_operations.py", line 9677, in SetQualifier
    self._imethodcall(

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/mock/mock.py", line 1100, in __call__
    return _mock_self._mock_call(*args, **kwargs)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/mock/mock.py", line 1104, in _mock_call
    return _mock_self._execute_mock_call(*args, **kwargs)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/mock/mock.py", line 1167, in _execute_mock_call
    result = effect(*args, **kwargs)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_wbemconnection_mock.py", line 1100, in _mock_imethodcall
    result = methodnameattr(namespace, **params)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_wbemconnection_mock.py", line 1547, in _imeth_SetQualifier
    self._mainprovider.SetQualifier(

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_mainprovider.py", line 1139, in SetQualifier
    self.validate_namespace(namespace)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_baseprovider.py", line 151, in validate_namespace
    raise CIMError(

pywbem._exceptions.CIMError: 3 (CIM_ERR_INVALID_NAMESPACE): Namespace does not exist in CIM repository: 'root/blah'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_baseprovider.py", line 417, in get_class
    klass = class_store.get(classname, copy=True)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_inmemoryrepository.py", line 104, in get
    raise KeyError('Name {} not in {} object store'

KeyError: "Name CIM_ObjectManager not in <class 'pywbem._cim_obj.CIMClass'> object store"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_mainprovider.py", line 1441, in EnumerateInstances
    cl = self.get_class(namespace, ClassName, local_only=False)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_baseprovider.py", line 419, in get_class
    raise CIMError(

pywbem._exceptions.CIMError: 6 (CIM_ERR_NOT_FOUND): Class 'CIM_ObjectManager' not found in namespace 'interop'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/Users/maiera/PycharmProjects/pywbem/pywbemtools/pywbemtools/pywbemcli/mockscripts/__init__.py", line 119, in setup_script
    module.setup(conn=conn, server=server, verbose=verbose)

  File "tests/unit/interop_noprovider_mock_script.py", line 19, in setup
    conn.compile_mof_string(mof, verbose=True)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_wbemconnection_mock.py", line 632, in compile_mof_string
    mofcomp.compile_string(mof_str, namespace)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_mof_compiler.py", line 2923, in compile_string
    rv = self.parser.parse(mof, lexer=lexer)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/ply/yacc.py", line 333, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/ply/yacc.py", line 1120, in parseopt_notrack
    p.callable(pslice)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_mof_compiler.py", line 842, in p_mp_setQualifier
    p.parser.server.create_namespace(ns)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_server.py", line 482, in create_namespace
    if self.brand == "OpenPegasus":

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_server.py", line 317, in brand
    self._determine_brand()

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_server.py", line 1222, in _determine_brand
    cimom_insts = self._conn.EnumerateInstances(

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem/_cim_operations.py", line 2654, in EnumerateInstances
    result = self._imethodcall(

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/mock/mock.py", line 1100, in __call__
    return _mock_self._mock_call(*args, **kwargs)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/mock/mock.py", line 1104, in _mock_call
    return _mock_self._execute_mock_call(*args, **kwargs)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/mock/mock.py", line 1167, in _execute_mock_call
    result = effect(*args, **kwargs)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_wbemconnection_mock.py", line 1100, in _mock_imethodcall
    result = methodnameattr(namespace, **params)

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_wbemconnection_mock.py", line 1275, in _imeth_EnumerateInstances
    instances = self._mainprovider.EnumerateInstances(

  File "/Users/maiera/virtualenvs/pywbemtools38/lib/python3.8/site-packages/pywbem_mock/_mainprovider.py", line 1444, in EnumerateInstances
    raise CIMError(

pywbem._exceptions.CIMError: 5 (CIM_ERR_INVALID_CLASS): Class 'CIM_ObjectManager' not found in namespace 'interop'.

Aborted!

andy-maier commented 3 years ago

If I analyze the tracebacks above, the one non-obvious link is between the 3rd and 4th traceback. The corresponding source code is _mof_compiler.py", line 836 as reported by the top item in the 3rd traceback:

    try:
        p.parser.handle.SetQualifier(qualdecl, namespace=ns)       # _mof_compiler.py", line 836
    except CIMError as ce:
        if ce.status_code == CIM_ERR_INVALID_NAMESPACE:
            if p.parser.verbose:
                p.parser.log(
                    _format("Creating namespace {0!A}", ns))
            p.parser.server.create_namespace(ns)    # the next exception occurs inside here (tracebacks 4, 5, 6)
            if p.parser.verbose:
                p.parser.log(
                    _format("Setting qualifier {0!A}", qualdecl.name))
            p.parser.handle.SetQualifier(qualdecl, namespace=ns)
        elif ce.status_code == CIM_ERR_NOT_SUPPORTED:

The 4th, 5th and 6th traceback are again linked in an obvious way, and the 4th and 5th traceback are the details on how the 6th traceback raised its exception. The link back to the 3rd traceback is in the middle of the 6th traceback.

The non-obvious link between tracebacks 3 and 4 is caused by the code shown above, where the next exception in the handler is not directly raised, but occurs inside of a function that is called, in this case create_namespace().

So I thought about whether the MOF compiler should really create missing namespaces when processing object creation directives in this deferred way, now that pragma namespace works. The only target namespace that is not defined in pragma namespace directives is the default namespace. However, I'm not sure we should always create it because if the first thing in a MOF is to set a new target namespace via pragma namespace, we would have created the default namespace needlessly. The only other choice would be to require that the default namespace must exist instead of creating it, but I'm not sure that is a good idea either. So the deferred namespace creation currently implemented is not so bad after all, it seems.

In fact, in a pragma namespace directive, we are creating the namespace unconditionally, not knowing whether it will actually be used. The pragma namespace directive could be the last thing in a MOF, or it could be followed by another pragma namespace directive. So basically we have two approaches for namespace creation currently: The default namespace is created when needed, but the pragma namespace namespaces are created immediately. Let's discuss.

andy-maier commented 3 years ago

I think from the above, there are multiple DISCUSSION points:

Should we make the namespace creation more consistent?
Should we break the lengthy traceback chains by using __cause__ = None when transforming exceptions?
Should we change the exception handling around calling the mock scripts? (see issue #881 for that)

KSchopmeyer commented 3 years ago

This issue should be fixed with pr #882 Karl will confirm that this works.

pywbem / pywbemtools