AAC mapping rules, file encoding & idempotency

[sygilber]

Just found that if the file encoding of the OAuth mapping rules file is utf-8 (instead of ascii) then the following code comparing both mapping rule content will silently fail and always return False even though both contents are identical (from ibmsecurity.isam.aac.mapping_rules.py):

def update(isamAppliance, name, content, check_mode=False, force=False):
    """
    Update a specified mapping rule
    """
    update_required = False
    ret_obj = search(isamAppliance, name)
    id = ret_obj['data']
    if force is False:
        if id:
            ret_obj_content = _get(isamAppliance, id)
            # Having to strip whitespace to get a good comparison (suspect carriage returns added after save happens)
            **if (ret_obj_content['data']['content']).strip() != (content).strip():**
                update_required = True

With some logger.debug() statements added here and there, found that that I was getting UnicodeDecodeError crash when trying to display content of mapping rules file.

At this point, I'm not sure what is the definitive solution if we opt to support utf-8 encoding for input mapping rules files. For now, I am just making sure that the input mapping rules file encoding is ascii, and this resolves idempotency behavior observed when running 'set_mapping_rules' role.

[ram-ibm]

Looked at options - here is one explanation for what's happening: https://stackoverflow.com/questions/3400171/python-utf-8-comparison

[jfmainville]

To add additional details regarding the set_mapping_rule role idempotence issue, it looks like the "End of Line Sequence" (EOLS) could also be the culprit as I recently encountered the same problem while trying to add new mapping rules.

Also, I was able to fix this issue by converting the "End of Line Sequence" (EOLS) from CRLF (Windows) to LF (Unix).

Furthermore, here's a comparison between a working and non-working file using the "file" program:

file mapping_rule_WORKING.js => mapping_rule.js: ASCII text, with very long lines file mapping_rule_NON_WORKING.js => mapping_rule.js: ASCII text, with very long lines, with CRLF line terminators

Maybe by adding a condition to convert the EOLS format to LF on the input file in the _check method could resolve this idempotency issue?