Can't Convert Some IPM Files to CSV

[patricklangat]

I use mideu to extract IPM files to CSV, about 75% of the files are extracted successfully while about 25% are not, I get the error message below for those that do not extract

C:\Data\pyenv\Scripts>mideu extract TT112T0.2020-05-29-01-83-63.001 c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\extract.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. config = yaml.load(config_file) Traceback (most recent call last): File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\runpy.py", line 192, in _run_module_as_main return _run_code(code, main_globals, None, File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Data\pyenv\Scripts\mideu.exe__main__.py", line 9, in File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\mideu.py", line 27, in cli_entry _main(args) File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\mideu.py", line 53, in _main args.func(args) File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\extract.py", line 48, in extract_command output_list = [ File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\cli\extract.py", line 49, in get_message_elements( File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\mciutil.py", line 326, in get_message_elements _process_element(bit, File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\mciutil\mciutil.py", line 369, in _process_element field_length = int(field_length_string) ValueError: invalid literal for int() with base 10: b'\x90\x94\x90'

How can I resolve the above error?

[willianrosa]

Hi @patricklangat I am using cardutil, because mciutil "THIS PACKAGE HAS BEEN DEPRECATED AND WILL NOT BE UPDATED GOING FORWARD", please change your application for this, maybe this problems was fixed in this new lib.

The replacement module is cardutil - see https://cardutil.readthedocs.io

att,

[adelosa]

I would definitely have a look at the new module. Has a command called mci_ipm_to_csv that provides the same function as mideu extract. Its Py3 only which should not be an issue for you as you are using 3.8. The new version does not support custom config. I assume you are using the standard config. Resolving these issues can be difficult without data that actually caused the issue and that is hard to share as it is often production data. If you can recreate the issue using test data and provide I would be happy to have a look..

[patricklangat]

Hi @willianvieira thanks for the link, I have downloaded cardutil and run mci_ipm_to_csv and it converts successfully and also fails on the same files mideu extracted, please see the output below

C:\Users\plangat\AppData\Local\Programs\Python\Python38-32\Scripts>mci_ipm_to_csv TT112T0.2020-06-03-00-46-36.001 Traceback (most recent call last): File "C:\Users\plangat\AppData\Local\Programs\Python\Python38-32\Scripts\mci_ipm_to_csv-script.py", line 11, in load_entry_point('cardutil==0.3.0', 'console_scripts', 'mci_ipm_to_csv')() File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\cli\mci_ipm_to_csv.py", line 10, in cli_entry cli_run(vars(cli_parser().parse_args())) File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\cli\mci_ipm_to_csv.py", line 19, in cli_run mci_ipm_to_csv(in_ipm=in_ipm, out_csv=out_csv, kwargs) File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\cli\mci_ipm_to_csv.py", line 45, in mci_ipm_to_csv dicts_to_csv(IpmReader(in_ipm, encoding=in_encoding, blocked=blocked), config['output_data_elements'], out_csv) File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\outputter.py", line 13, in dicts_to_csv filtered_data_list = filter_data_list(data_list, field_list) File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\outputter.py", line 36, in filter_data_list return [filter_dictionary(item, field_list) for item in data_list] File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\outputter.py", line 36, in return [filter_dictionary(item, field_list) for item in data_list] File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\mciipm.py", line 303, in next return iso8583.loads(vbs_record, encoding=self.encoding) File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\iso8583.py", line 121, in loads return _iso8583_to_dict(b, iso_config, encoding, hex_bitmap) File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\iso8583.py", line 170, in _iso8583_to_dict return_message, message_increment = _iso8583_to_field( File "c:\users\plangat\appdata\local\programs\python\python38-32\lib\site-packages\cardutil\iso8583.py", line 288, in _iso8583_to_field field_data = field_data.decode(encoding) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 14: ordinal not in range(128)

Have any idea how to fix this?

[patricklangat]

Hi @adelosa thanks for your response, I have tried the new cardutil but with the same results, please see output above, let me see how I can get a sample file that fails so you can troubleshoot and see what could be going wrong

[patricklangat]

@adelosa and @willianvieira, I'll be waiting for your input

[adelosa]

I have opened up issue reporting for cardutil.. see https://bitbucket.org/hoganman/cardutil/issues Please report there and make sure you include information on how to replicate the issue. Without reproduction details, will be very hard for me to identify the issue. Make sure that you have the right encoding and blocking options for the file you are processing.

[adelosa]

No updates.. closing..

[patricklangat]

I added code to process special characters and it worked.