Double nested fields

[netixx]

I am using the library to access hardware asset information (alm_hardware table).

I have the following code:

class ModelCategory(ModelSchema):
    sys_id = aiosnow.fields.String(is_primary=True)
    name = aiosnow.fields.String()
    parent_cateogry = aiosnow.fields.String()
    cmdb_ci_class = aiosnow.fields.String()

class HardwareModel(ModelSchema):
    sys_id = aiosnow.fields.String()
    name = aiosnow.fields.String()
    manufacturer = aiosnow.fields.StringMap()
    description = aiosnow.fields.String()
    model_number = aiosnow.fields.String()
    barcode = aiosnow.fields.String()
    power_consumption = aiosnow.fields.Integer()
    rack_units = aiosnow.fields.Integer()
    cmdb_model_category = ModelCategory

class Hardware(aiosnow.TableModel):
    sys_id = aiosnow.fields.String(is_primary=True)
    asset_tag = aiosnow.fields.String()
    install_status = aiosnow.fields.StringMap()
    location = aiosnow.fields.String(pluck=Pluck.DISPLAY_VALUE)
    company = aiosnow.fields.StringMap()
    ci = aiosnow.fields.String(pluck=Pluck.DISPLAY_VALUE)
    serial_number = aiosnow.fields.String()
    model = HardwareModel
    sys_class_name = aiosnow.fields.String(pluck=Pluck.DISPLAY_VALUE)
    model_category = ModelCategory

As you can see the ModelCategory (table cmdb_model_category) is nested in HardwareModel (cmdb_model) which is a field in Hardware (alm_hardware).

I am having trouble with this setup : the cmdb_model_category field in the HardwareModel is not present in the result of the query.

I get a exception/warning while using this setup:

.../lib/python3.7/site-packages/aiosnow/models/common/schema/base.py:90: UserWarning: Unexpected field in response content: cmdb_ci_class, skipping...
  f"Unexpected field in response content: {key}, skipping..."

and the field is missing:

{
'asset_tag': 'P1000443', 
'ci': 'ThinkStation S20', 
'company': <StringMapping [key=...ME Japan]>, 
'install_status': <StringMapping [key=...lisation]>, 
'location': '2-12-1 Ookayama, Meg...-ku, Tokyo',
 'model': {
   'barcode': 'S20 (4105R9U)', 
   'description': '<p>\n<h3>Ordinateur d...></ul></p>', 
   'manufacturer': <StringMapping [key=...e=Lenovo]>, 
   'model_number': 'S20(4105R9U)', 
   'name': 'ThinkStation S20', 
   'power_consumption': None, 
   'rack_units': 1, 'sys_id': 'a9a2d0c3c6112276010d...6c5ddd3461'
}, 
'model_category': {
   'cmdb_ci_class': 'cmdb_ci_computer', 
   'name': 'Computer', 
   'parent_cateogry': None, 
   'sys_id': '81feb9c137101000deea...c8bcbe5dc4'
}, 
'serial_number': 'GSN-700-V69187-LX', 
'sys_class_name': 'Matériel', 
'sys_id': '01a9ec0d3790200044e0...c8bcbe5d93'
}

In this particular case, there is a workaround, as the model category is also a field of the alm_hardware.

[rbw]

Definitely a bug.

The actual data is fetched but considered unknown by aiosnow as the nested schema registration failed, which also explains the raised warning.

Looking into this shortly. Thanks for reporting.

[rbw]

...or, well, perhaps more of a missing feature.

The idea is of course to support any level of nesting, but this was never actually implemented.

What needs to be done is basically to store absolute document paths with the GetRequest.nested_fields items, and a new GetRequest.get_subdocument(content) that uses these paths to return sub-documents that can be merged with the main document upon response deserialization. Also, this support should be added to the query builder serializer (given the ServiceNow API supports any level of nesting in queries).

My time is somewhat limited over the coming days, but I'll work on this next (unless someone else wants to).

[rbw]

Support for any level of nesting has been implemented in https://github.com/rbw/aiosnow/pull/97

@netixx - can you test it out?

Note! the cmdb_model_category field used in your example doesn't work for me, as that field doesn't reference an object in my ServiceNow test environment. My example below uses product_catalog_item instead.

Model declaration

class ProductCatalogItem(ModelSchema):
    sys_id = aiosnow.fields.String()
    no_order_now = aiosnow.fields.Boolean()
    delivery_time = aiosnow.fields.DateTime()
    sys_name = aiosnow.fields.String()

class HardwareModel(ModelSchema):
    sys_id = aiosnow.fields.String()
    name = aiosnow.fields.String()
    manufacturer = aiosnow.fields.StringMap()
    model_number = aiosnow.fields.String()
    barcode = aiosnow.fields.String()
    power_consumption = aiosnow.fields.Integer()
    rack_units = aiosnow.fields.Integer()
    product_catalog_item = ProductCatalogItem

class Hardware(aiosnow.TableModel):
    sys_id = aiosnow.fields.String(is_primary=True)
    asset_tag = aiosnow.fields.String()
    install_status = aiosnow.fields.StringMap()
    location = aiosnow.fields.String(pluck=Pluck.DISPLAY_VALUE)
    company = aiosnow.fields.StringMap()
    ci = aiosnow.fields.String(pluck=Pluck.DISPLAY_VALUE)
    serial_number = aiosnow.fields.String()
    model = HardwareModel
    sys_class_name = aiosnow.fields.String(pluck=Pluck.DISPLAY_VALUE)
    product_catalog_item = ProductCatalogItem

Response data

[ { 'asset_tag': 'P1000479',
    'ci': 'MacBook Pro 15"',
    'company': <StringMapping [key=81fdf9ebac1d55eb4cb89f136a082555, value=ACME China]>,
    'install_status': <StringMapping [key=1, value=In use]>,
    'location': '2500 West Daming Road, Shanghai',
    'model': { 'barcode': 'MD318LL/A',
               'manufacturer': <StringMapping [key=b7e9e843c0a80169009a5a485bb2a2b5, value=Apple]>,
               'model_number': 'MD318LL/A',
               'name': 'MacBook Pro 15"',
               'power_consumption': None,
               'product_catalog_item': { 'delivery_time': datetime.datetime(1970, 1, 3, 0, 0),
                                         'no_order_now': False,
                                         'sys_id': '2ab7077237153000158bbfc8bcbe5da9',
                                         'sys_name': 'Apple MacBook Pro 15"'},
               'rack_units': 1,
               'sys_id': 'd501454f1b1310002502fbcd2c071334'},
    'product_catalog_item': { 'delivery_time': datetime.datetime(1970, 1, 3, 0, 0),
                              'no_order_now': False,
                              'sys_id': '2ab7077237153000158bbfc8bcbe5da9',
                              'sys_name': 'Apple MacBook Pro 15"'},
    'serial_number': 'BQP-854-D33246-GH',
    'sys_class_name': 'Hardware',
    'sys_id': 'd501454f1b1310002502fbcd2c071334'}]

[netixx]

I confirm that your example works in my setup as well. Thank you for the quick code!

Regarding the cmdb_model_category field, I was mistaken:

• for Hardware, it's a direct relation to ModelCategory
• for HardwareModel, it's actually a list of ModelCategory

I tried to use cmdb_model_category = marshmallow.fields.List(ModelCategory) to represent this, be this does not seem to be the right way. How are list of objects handled ?

[rbw]

I tried to use cmdb_model_category = marshmallow.fields.List(ModelCategory) to represent this, be this does not seem to be the right way. How are list of objects handled ?

Currently, this isn't supported. However, as it's a must-have feature, I'll go ahead and add it to #97 (improve-schema-nesting) while I'm at it. Shouldn't be too much work.

So.. question is how to implement this. We can't use native marshmallow fields, as that'd break stuff - like completion and query serialization features of ModelSchema.

Adding optional support for passing an object of a nested schema, which accepts many in it's constructor, could be a way forward. Default would be False, and using a class directly would still work.

Something like this perhaps:

cmdb_model_category = ModelCategory(many=True)

What do you think? Anyone else with an opinion?

[rbw]

I've done some looking into the List [list of references] type, and it seems to expect a comma-separated list of IDs, but no actual links, which is somewhat surprising.

So.. for this to work, aiosnow needs to be made aware of where to find the referenced objects. I think the cleanest and most generic way to do this is via the schema constructor, using an endpoint parameter, i.e.:

cmdb_model_category = ModelCategory(many=True, endpoint="table_name_of_model_category")

Anyway, this means more work, so I'll handle it separately from #97.

[rbw]

Nested registration, resolver and expansion code has been updated in 0.5.3, available on pypi.

https://github.com/rbw/aiosnow/releases/tag/0.5.3

I'll be working on the other issue with reference lists in https://github.com/rbw/aiosnow/issues/98