Support parsing of expanded IPM parameter files

[makafanpeter]

Hi there,

I'm encountering an issue while trying to parse a Mastercard TT067 file using the mci_ipm_param_to_csv CLI with the following command:

mci_ipm_param_to_csv -o TT067T0.2024-01-25-01-00-25.IP0095T1 --in-encoding ascii --debug IP0095T1 --config config.json

The problem is that the output file only contains headers and no data. I'd appreciate any guidance on how to obtain the correct results.

For your reference, I've attached the relevant files in a ZIP archive named files.zip

Thanks

[adelosa]

I'll have a look this weekend Peter and let you know what I find out. I don't have a lot of users of these functions so I suspect there could be some issues in there.

[makafanpeter]

I'll have a look this weekend Peter and let you know what I find out. I don't have a lot of users of these functions so I suspect there could be some issues in there.

Thank you! I appreciate you taking the time to look into this.

[adelosa]

The current tool is assuming the use of the compressed parameter file format.

In the compressed format, the effective date uses Julian format (YYMMM) and the table code is a 3 digit code instead of the full table value.

Below is a compressed (1) and expanded (2) IP0040T1 table entry.. I have added periods/dots in the compressed version where the fields differ to show they are actually the same type of record.

17111...14A036.....5116545113000000000MCC5116545113999999999MCC020000000152710084563AUS036CMCC NNYMCC N0000000362
2024012414AIP0040T15417750570000000000MPL5417751329999999999MCC010000000177510080140USA8401MPL NYNMPL7N00000084020000000000000000000000000000 000000NY   000000NNNN0NDNN0NNNN  NNDNYNN                     000000 000000000000 0                          00000000000               000000                         

To support expanded format, I will need to make some changes. May take me a couple of weeks to implement. Will require a new command line option to support. If you need it now, you could look at using a compressed version of the parameter file. The Pre-edit tool is used to expand the file.

[adelosa]

OK.. so there are a couple of issues with this, especially for config. I would like to incorporate a complete config for table extraction in the library as most people will likely only ever use the same config if provided. The issue is the offset of the fields changes depending on whether using the compressed or expanded forms of the config. My view is that the config should be based on the expanded form, and the library will adjust the position if compress is selected. This also means that the standard fields should not be provided in the config as their position will change depending on the format.. So effective timestamp, active/inactive flag and table id should all be omitted from the config entries and will be automatically included in all extracts.

[adelosa]

I have updated the library to support both the compressed and expanded formats of the IPM parameter files. I have updated the config to remove the common fields that are auto-populated. I had a quick look at your config file. You will need to update it to use the expanded table format and not include the effective timestamp, active inactive code and table name fields in the config - these will be automatically extracted. As example, here is config for IP0095T1.

 {
        "IP0006T1": {
            "card_program_id": {"start": 19, "end": 22},
            "data_element_id": {"start": 22, "end": 25},
            "data_element_name": {"start": 25, "end": 82},
            "data_element_format": {"start": 82, "end": 85}
}

@makafanpeter - Can you please test the new version and let me know how you go. You can install the new version from git using the command pip install --upgrade https://github.com/adelosa/cardutil/tarball/master Once you confirm, I'll push a new release.

[adelosa]

@makafanpeter - Are you able to provide feedback on the changes?

[makafanpeter]

Hello @adelosa , sorry I don't have access to my computer at the moment, pls I would give feedback shortly. Thanks

[makafanpeter]

Hello @adelosa ,

I've made the update to the package and ran the following command:

mci_ipm_param_to_csv -o files/TT067T0.2024-01-25-01-00-25.001.IP0006T1 --in-encoding ascii files/TT067T0.2024-01-25-01-00-25.001 IP0006T1 --config \files\config.json

The output file is still empty. I'm not sure if this is the expected behavior based on the feedback you provided. Could you please help clarify, or if there might be an issue with the command or configuration?

For your reference, I've attached the relevant files in a ZIP archive named files.zip

Thanks

[adelosa]

You need to add the --expanded option to process an expanded file. Sorry. Forgot to mention that.

[makafanpeter]

Hello @adelosa , Here's the output file generated by running the following command:

mci_ipm_param_to_csv -o files/TT067T0.2024-01-25-01-00-25.001.IP0040T1 --in-encoding ascii files/TT067T0.2024-01-25-01-00-25.001 IP0040T1 --config \files\config.json --expanded

You can download the output file here: TT067T0.2024-01-25-01-00-25.001.IP0040T1.zip

Looks good, thanks for your help!

[makafanpeter]

I didn't mention this before, but I also use the following code to read the files programmatically. Would the update support expanded files as well?

       records = []
        try:
            with open(path, 'rb') as test_param_stream:
                test_param_stream.seek(0)
                reader = IpmParamReader(test_param_stream, encoding=encoding, table_id=table_id, blocked=blocked,
                                        param_config=my_config)
                for record in reader:
                    records.append(record)

[adelosa]

IpmParamReader now has expanded parameter. Just set this to True in your code.

IpmParamReader(..., expanded=True)

Let me know how you go.

[makafanpeter]

Hello @adelosa ,

When I add a custom field to the configuration file for any table within mci_parameter_tables, it seems to be ignored.

 "data": {
                        "start": 0,
                        "end": 2048
                  }

my use case for this, is for updating my configs, because sometimes it doesn't match whats on Mastercard IPM Table Layouts docs

[adelosa]

You will need to provide more details for me to understand what is happening.

[makafanpeter]

Hello @adelosa ,

In the other version if I define a mci_parameter_tables  congfig as

{
"IP0040T1": {
"data": {"start": 0, "end": 2048}
}
}

other version sample code

encoding = "cp500"
blocked = True
my_config = {
"IP0040T1": {
"data": {"start": 0, "end": 2048}
}
}

records = []
 with open(path, 'rb') as test_param_stream:
                test_param_stream.seek(0)
                reader = IpmParamReader(test_param_stream, encoding=encoding, table_id=table_id, blocked=blocked,
                                        param_config=my_config)
                for record in reader:
                    print(record)

The output

{'data': '2410314A0379752306400000000000PVL9752306499999999999PVL020000001242230081715SWE752DPVL NNNPVL N00000075220000000000000000000000000000 000000NN   000000NNNN0NUNN0NNNN99NUUNNNNN                   '}, {'data': '2410314A0379883880000000000000MCC9883889999999999999MCC020000000118010001924HKG344CMCC NNNMCC N00000034420000000000000000000000000000 000000NN   000000NNNN0NUNN0NNNN99NNUNNNNN                   '}, {'data': '2410314A0379883880000000000000PVL9883889999999999999PVL010000000118030001924HKG344CPVL NNNPVL N00000034420000000000000000000000000000 000000NN   000000NNNN0NUNN0NNNN99NUUNNNNN                   '}]

In the new version sample code

encoding = "cp500"
blocked = True
my_config = {
"IP0040T1": {
"data": {"start": 0, "end": 2048}
}
}

records = []
 with open(path, 'rb') as test_param_stream:
                test_param_stream.seek(0)
                reader = IpmParamReader(test_param_stream, encoding=encoding, table_id=table_id, blocked=blocked,
                                        param_config=my_config, expanded=expanded)
                for record in reader:
                    print(record)

output:

{'table_id': 'IP0040T1', 'effective_timestamp': '2410314', 'active_inactive_code': 'A', 'data': '        '}, {'table_id': 'IP0040T1', 'effective_timestamp': '2410314', 'active_inactive_code': 'A', 'data': '        '}, {'table_id': 'IP0040T1', 'effective_timestamp': '2410314', 'active_inactive_code': 'A', 'data': '        '}, {'table_id': 'IP0040T1', 'effective_timestamp': '2410314', 'active_inactive_code': 'A', 'data': '        '}, {'table_id': 'IP0040T1', 'effective_timestamp': '2410314', 'active_inactive_code': 'A', 'data': '        '}, {'table_id': 'IP0040T1', 'effective_timestamp': '2410314', 'active_inactive_code': 'A', 'data': '        '}, {'table_id': 'IP0040T1', 'effective_timestamp': '2410314', 'active_inactive_code': 'A', 'data': '        '}]

I noticed that no value is read for the data property in the output of the new version. Please let me know if I'm doing something wrong.

[makafanpeter]

Hello @adelosa ,

Here's my workaround using your library to solve the technical problem I opened this issue for.

1. Extend VbsReader

class CustomIpmParamReader(VbsReader):
    """
    IPM Param reader can be used to iterate through an IPM parameter extract file.
    The record is returned as a dictionary containing the parameter keys.

    ::

        from cardutil.mciipm import IpmParamReader
        with open('param.bin', 'rb') as param_in:
            reader = IpmParamReader(param_in, table_id='IP0040T1')
            for record in reader:
                print(record)

    If the parameter file is 1014 block format, then set the ``blocked`` parameter to True.

    ::

        from cardutil.mciipm import IpmParamReader
        with open('blocked_param.bin', 'rb') as param_in:
            reader = IpmParamReader(param_in, table_id='IP0040T1', blocked=True)
            for record in reader:
                print(record)

    """
    # layout for IP0000T1
    _IP0000T1_KEY = slice(11, 19)
    _IP0000T1_TABLE_ID = slice(19, 27)
    _IP0000T1_TABLE_SUB_ID = slice(243, 246)

    # compressed table type for all except IP0000T1 - get this 3 letter code from self.table_keys
    _C_EFF_TIMESTAMP = slice(0, 7)
    _C_ACTIVE_INACTIVE_CODE = slice(7, 8)
    _C_TABLE_SUB_ID = slice(8, 11)

    # expanded table common fields
    _X_EFF_TIMESTAMP = slice(0, 10)
    _X_ACTIVE_INACTIVE_CODE = slice(10, 11)
    _X_TABLE_ID = slice(11, 19)

    def __init__(self, param_file: typing.BinaryIO, table_id: str, encoding: str = None, param_config: dict = None,
                 expanded: bool = False,
                 **kwargs):
        """
        Create a new IpmParamReader

        :param param_file: the file object to read
        :param table_id: the IPM parameter table to read
        :param encoding: the parameter file encoding
        :param param_config: config dict with key bit_config
        """
        self.encoding = encoding if encoding else 'latin_1'
        self.param_config = param_config if param_config else cardutil_config.config.get('mci_parameter_tables')
        self.table_id = table_id
        self.table_index = dict()
        self.expanded = expanded
        super(CustomIpmParamReader, self).__init__(param_file, **kwargs)

        # check if config available for table id
        if not self.param_config.get(table_id):
            raise MciIpmDataError(f'Parameter config not available for table {table_id}')

        # load the table index
        trailer_record_found = False
        while True:
            try:
                vbs_record = super(CustomIpmParamReader, self).__next__()
            except StopIteration:
                break
            record = vbs_record.decode(self.encoding)
            if record[self._IP0000T1_KEY] == 'IP0000T1':
                self.table_index[record[self._IP0000T1_TABLE_SUB_ID]] = record[self._IP0000T1_TABLE_ID]
            if record.startswith('TRAILER RECORD IP0000T1'):
                trailer_record_found = True
                break
        LOGGER.debug('IP0000T1 records: {}'.format(self.table_index))
        if not trailer_record_found:
            raise MciIpmDataError('parameter file missing IP0000T1 trailer record')

    def __next__(self) -> dict:
        while True:
            record = super(CustomIpmParamReader, self).__next__()
            if self.expanded:
                record_table_id = record[self._X_TABLE_ID].decode(self.encoding)
            else:
                record_table_id = self.table_index.get(record[self._C_TABLE_SUB_ID].decode(self.encoding))

            LOGGER.debug(f"{record_table_id=}, {record=}")
            if record_table_id == self.table_id:
                record_dict = dict()
                for field in self.param_config[record_table_id]:
                    record_dict[field] = self._get_param_field(record, field)
                return record_dict

    def _get_param_field(self, record, field):
        if self.expanded:
            record_table_id = record[self._X_TABLE_ID].decode(self.encoding)
        else:
            record_table_id = self.table_index.get(
                record[self._C_TABLE_SUB_ID].decode(self.encoding)
            )
        return record[self.param_config[record_table_id][field]["start"]:
                      self.param_config[record_table_id][field]["end"]
               ].decode(self.encoding)

2. Define a custom config

CUSTOM_CONFIG = {
    'IP0072T1': {
        "effective": {"start": 0, "end": 7},
        "active": {"start": 7, "end": 8},
        "sequence_number": {"start": 8, "end": 11},
        "member_id": {"start": 11, "end": 22},
        "filler_1": {"start": 22, "end": 24},
        "interchange_region": {"start": 24, "end": 25},
        "iei_qualifier": {"start": 25, "end": 27},
        "internal_member_id_indicator": {"start": 27, "end": 28},
        "acquirer_switch": {"start": 28, "end": 29},
        "atm_indicator": {"start": 29, "end": 30},
        "rcl_region": {"start": 30, "end": 31},
        "endpoint": {"start": 31, "end": 38},
        "world_member_id_cbk_switch": {"start": 38, "end": 39},
        "world_card_acc_mcc_group_cd1": {"start": 39, "end": 40},
        "world_card_acc_mcc_group_cd2": {"start": 40, "end": 41},
        "world_card_acc_mcc_group_cd3": {"start": 41, "end": 42},
        "world_card_acc_mcc_group_cd4": {"start": 42, "end": 43},
        "world_card_acc_mcc_group_cd5": {"start": 43, "end": 44},
        "filler_2": {"start": 44, "end": 45},
        "member_name": {"start": 45, "end": 75},
        "country_code": {"start": 75, "end": 78},
        "country_code_iso": {"start": 78, "end": 81},
        "filler_3": {"start": 81, "end": 102},
        "outbound_format_indicator": {"start": 102, "end": 103},
        "mc_electronic_card_indicator": {"start": 103, "end": 104},
        "eea_acquirer_country_service_ica": {"start": 104, "end": 105},
        "filler_4": {"start": 105, "end": 106},
        "national_payment_gateway_processor_switch": {"start": 106, "end": 107},
        "payment_transfer_activity_participant_indicator": {"start": 107, "end": 108},
        "data": {"start": 0, "end": 2048}
    }
}

3. Define offset config

OFFSET_CONFIG = {
    'IP0040T1': 8,
    'IP0072T1': 8,
}

4. Helper class to parse the files

class MasterCardParamTableUtil:
    def __init__(self):
        self.config = self.get_config()

    @staticmethod
    def get_config():
        card_util_config = cardutil_config.config.get('mci_parameter_tables')
        card_util_config['IP0040T1']["effective_timestamp"] = {"start": 0, "end": 7}
        card_util_config['IP0040T1']["authentication_indicator"] = {"start": 160, "end": 161}
        card_util_config['IP0040T1']["cardholder_currency_indicator"] = {"start": 169, "end": 170}
        card_util_config['IP0040T1']["data"] = {"start": 0, "end": 2048}
        card_util_config.update(CUSTOM_CONFIG)
        return card_util_config

    @staticmethod
    def adjust_offsets(config, offset_config):
        ignore_keys = {
            "effective_timestamp": {"start": 0, "end": 10},
            "active_inactive_code": {"start": 10, "end": 11},
            "table_id": {"start": 11, "end": 19}
        }
        for key, properties in config.items():
            if key in offset_config:
                offset = offset_config[key]
                for prop in properties:
                    if prop in ignore_keys:
                        properties[prop] = ignore_keys[prop]
                    else:
                        start = properties[prop]['start']
                        end = properties[prop]['end']
                        if start == 0:
                            pass
                        else:
                            properties[prop]['start'] += offset
                            properties[prop]['end'] += offset
        return config

    def get_config_param(self, param, expanded):
        config = {param: self.config.get(param, {}), "IP0000T1": CUSTOM_CONFIG.get("IP0000T1", {})}
        config_copy = copy.deepcopy(config)
        if expanded:
            return self.adjust_offsets(config_copy, OFFSET_CONFIG)
        return config_copy

    def get_records_from_table(self, path, table_id, expanded):
        if expanded:
            encoding = "ascii"
        else:
            encoding = "cp500"
        blocked = True
        my_config = self.get_config_param(table_id, expanded)
        records = []
        try:
            with open(path, 'rb') as test_param_stream:
                test_param_stream.seek(0)
                reader = CustomIpmParamReader(test_param_stream, encoding=encoding, table_id=table_id, blocked=blocked,
                                              param_config=my_config, expanded=expanded)
                for record in reader:
                    records.append(record)
        except MciIpmDataError as me:
            print(f"Error: {me}")
        except Exception as e:
            print(f"Error: {e}")
        return records

I really appreciate your help with my challenges, thank you so much! I see you're working on a C# version of this library. Would you be interested in some help?

[adelosa]

Apologies for the late response.. Its been a busy week. I am not sure what the issue you are having with your program. I have just tried running your original program code with the param file you provided earlier and I got the expected results:

from cardutil.mciipm import IpmParamReader

if __name__ == '__main__':

    my_config = {
        "IP0040T1": {
            "data": {"start": 0, "end": 2048}
        }
    }

    # param_data.bin is the param file you attached (TT067T0.2024-01-25-01-00-25.001)
    with open('param_data.bin', 'rb') as test_param_stream:
        test_param_stream.seek(0)
        reader = IpmParamReader(
            param_file=test_param_stream,
            encoding='ascii',
            table_id='IP0040T1',
            blocked=True,
            param_config=my_config,
            expanded=True
        )
        for record in reader:
            print(record)

The output was as below (not full, just first couple of records)

{'table_id': 'IP0040T1', 'effective_timestamp': '2024012414', 'active_inactive_code': 'A', 'data': '2024012414AIP0040T12231130000000000000MCG2231130099999999999MCC010000002152110073966BRA076BMCG NNNMCG N00000098620000000000000000000000000000 000000NN   000000NNNN0NDNN0NYNN  DNDNNNN                     000000 000000000000 0                          00000000000               000000                         '}
{'table_id': 'IP0040T1', 'effective_timestamp': '2024012414', 'active_inactive_code': 'A', 'data': '2024012414AIP0040T12231130100000000000MBK2231130199999999999MCC010000002152110073966BRA076BMBK NNYMBK N00000098620000000000000000000000000000 000000NN   000000NNYY0NDNN0NNNN  DNDNNNN                     000000 000000000000 0                          00000000000               000000                         '}
{'table_id': 'IP0040T1', 'effective_timestamp': '2024012414', 'active_inactive_code': 'A', 'data': '2024012414AIP0040T12231130200000000000MPL2231130299999999999MCC010000002152110073966BRA076BMPL NNYMPL N00000098620000000000000000000000000000 000000NN   000000NNYY0NDNN0NNNN  DNDNNNN                     000000 000000000000 0                          00000000000               000000                         '}
{'table_id': 'IP0040T1', 'effective_timestamp': '2024012414', 'active_inactive_code': 'A', 'data': '2024012414AIP0040T12231130300000000000CIR2231130399999999999CIR020000002152110080255BRA076B    NNNCIR N00000098620000000000000000000000000000 000000NN   000000NNNY0NDNN0NNNN  DNDNNNN                     000000 000000000000 0                          00000000000               000000                         '}

Your file has a CP500 encoding (mainframe) so its obviously not the same parameter file. Are you able to provide the actual file you are having the issue with as I can't recreate the issue. I would like the tool to work without all the additional code you have added to get it working.

The C# library is something I was thinking about as it seems to be the language most people ask for. My biggest challenge is getting enough time to work on it.

[adelosa]

@makafanpeter Any update you can share?

[makafanpeter]

Hello @adelosa ,

Sorry for the late reply. I've attached both parameter files for your reference(cp500 and ascii). You no longer need to take any action, as I've found a solution to the problem. Thanks IPM.zip

[adelosa]

I assume from your comment that you identified the issue on your side rather than cardutil. Are you ok for me to close this issue. If that is the case I will release a new version to PYPI with the enhancements.

[makafanpeter]

Yes, Thanks for your help!

[charisad]

@adelosa I dont think cardutil has an issue as the config file i shared with you https://github.com/adelosa/cardutil/issues/8#issuecomment-1287567604 sometime ago captures can be updated by @makafanpeter to get the particular table he wants