search

adelosa / cardutil

Payment cards tools including ISO8583 parser and Mastercard IPM files processing
MIT License
24 stars 4 forks source link

mci_csv_to_ipm: DE48 fields dropped when PDS fields in file #5

Closed charisad closed 2 years ago

charisad commented 2 years ago

After using mci_csv_to_ipm --out-encoding cp500 to create ipm file, header FILE ID seems to be hidden, Can you suggest the data elements needed to create the file from the csv.

adelosa commented 2 years ago

The utility does not add headers and footers. You need to add these records to your csv file and add any columns for the header specific fields that you require to be present in the record. I can't advise on what fields are required in a header or footer - I would suggest you extract an existing IPM file to CSV to see what is present and use that as a template.

charisad commented 2 years ago

The utility does not add headers and footers. You need to add these records to your csv file and add any columns for the header specific fields that you require to be present in the record. I can't advise on what fields are required in a header or footer - I would suggest you extract an existing IPM file to CSV to see what is present and use that as a template.

Okay I have found the columns for the header specific fields yet the mci_csv_to_ipm reads the header record as a first presentment.

charisad commented 2 years ago

The utility does not add headers and footers. You need to add these records to your csv file and add any columns for the header specific fields that you require to be present in the record. I can't advise on what fields are required in a header or footer - I would suggest you extract an existing IPM file to CSV to see what is present and use that as a template.

Okay I have found the columns for the header specific fields yet the mci_csv_to_ipm reads the header record as a first presentment.

Hi Anthony, any help with this one?

adelosa commented 2 years ago

The MTI and Function code (DE24) makes it a header/trailer record from memory. The Mastercard IPM manuals will have the details you need to construct the header/footers for the file.

charisad commented 2 years ago

The MTI and Function code (DE24) makes it a header/trailer record from memory. The Mastercard IPM manuals will have the details you need to construct the header/footers for the file.

I get you alright, I have used the correct MTI(1644) and function code(697) and added the required DE48 to the csv but the utility still treats it as a first presentment. This is the csv sample i am using to create the ipm using mci_csv_ipm. adelosa

charisad commented 2 years ago

Hi Anthony, after using the mci_csv_to_ipm to convert the above .csv file to .ipm. This is what happens, the header is read as a first presentment hence the file id is lost. Can you advise on that? Thanks in advance

adelosa commented 2 years ago

Attach a test CSV file that causes the issue and details of the expected behaviour. Then I can advise.

charisad commented 2 years ago

MCCreditAcq_20211207124011107_outgoing.csv Above is the attached csv file. The expected ipm file should be of the first format I have sent the sample ipm file to your email adelosa@gmail.com The wrong format is attached to the second email I have sent. Thanks in advance

charisad commented 2 years ago

Attach a test CSV file that causes the issue and details of the expected behaviour. Then I can advise.

Hi Anthony, I have attached the sample csv. Sent you the format I want to generate and the wrong format it generated in your email adelosa@gmail.com

adelosa commented 2 years ago

I think the issue is caused by the combination of DE48 (field that contains PDS fields) and specification of a PDS field in your CSV file (PDS0148). Either would be OK, but not both. The specification of a PDS field will likely overwrite any details you provide in DE48. You can either add PDS0148 to DE48 or remove DE48 and respecify the header fields as PDSxxxx type fields. Let me know how it goes. MCCreditAcq_MCCMDFSnk_211207_00005_outgoing.ipm.csv.ipm.txt MCCreditAcq_MCCMDFSnk_211207_00005_outgoing.ipm.txt

adelosa commented 2 years ago

Any progress on this one?

charisad commented 2 years ago

Any progress on this one?

Hi Anthony, I tried it but it seem not to be working. The DE48 is overwritten regardless. When I write all the PDS records in the DE48 and ignore the PDS columns, I get the error string.startswith[PDS] not found.

charisad commented 2 years ago

With the sample csv file I sent you, how should the csv look for me to get my desired output. My header and trailer are always taken as first presentment.

adelosa commented 2 years ago

Can you attach the amended input csv files so I can check the output on my side?

charisad commented 2 years ago

MCCreditAcq_20220607084815224_outgoing.csv Attached is the amended csv

adelosa commented 2 years ago

The DE48 in this file is not formatted correctly in PDS format.

This is what is in the file: 01050250022206070000001599500014011002522060700000015995000140122001T01910012

Which when converted to PDS fields is: field:0105 Len:025 Data:0022206070000001599500014 field:0110 Len:025 Data:2206070000001599500014012 field:2001 Len:T01 <<--- ERROR, not a length! 910012

I have created some new tests which I will add to the code base but it demonstrates the following behaviour in regards to CSV to IPM and PDS fields.

DE fields that receive PDS fields are marked in the config with a field processor of "PDS". Currently PDS DE fields are configured as 48, 62, 123, 124 and 125.

As I said, its either/or.. You can't use both. I may in the future look at enhancing the function to deal with both being present somehow, but for now, thats how it works.

I have now confirmed how PDS fields work and they do load to the resulting IPM file according to the rules above.

charisad commented 2 years ago

MCCreditAcq_20220607113622385_outgoing.csv Updated csv

adelosa commented 2 years ago

This will generate an IPM file as provided.. I just processed the first (header) record. I have omitted the extract padding at end of file.

00000000: 00 00 00 8B 31 36 34 34  80 00 01 00 00 01 00 00  ....1644........
00000010: 02 00 00 00 00 00 00 00  36 39 37 31 30 35 30 31  ........69710501
00000020: 30 35 30 32 35 30 30 32  32 32 30 36 30 37 30 30  0502500222060700
00000030: 30 30 30 30 31 35 39 39  35 30 30 30 31 31 30 31  0000159950001101
00000040: 31 30 30 32 35 30 30 32  32 32 30 36 30 37 30 30  1002500222060700
00000050: 30 30 30 30 31 35 39 39  35 30 30 30 31 31 30 31  0000159950001101
00000060: 32 32 30 30 31 30 30 32  32 32 30 36 30 37 30 30  2200100222060700
00000070: 30 30 30 30 31 35 39 39  35 30 30 30 31 31 54 30  00001599500011T0
00000080: 31 39 31 30 30 31 32 30  30 30 30 30 30 30 31 00  191001200000001.
00000090: 00 00 00 40 40 40 40 40  40 40 40 40 40 40 40 40  ...@@@@@@@@@@@@@

The values from the header extracted from the generated file: {'MTI': '1644', 'DE24': '697', 'DE48': '010502500222060700000015995000110110025002220607000000159950001101220010022206070000001599500011T01910012', 'PDS0105': '0022206070000001599500011', 'PDS0110': '0022206070000001599500011', 'PDS0122': '0', 'PDS0222': '70000001599500011T01910012', 'DE71': 1}

The tool is converting the file from CSV to IPM correctly. If you are still having issues having the header recognised, then its not an issue with this tool but your data or the tool you are feeding the file to.

charisad commented 2 years ago

I also generated the ipm fine and it worked fine but please try encoding from the cp500 to latin1. It will throw an error, can you check that part for me.

adelosa commented 2 years ago

Sounds like a different issue. Can you please raise as new item in GitHub with file and commands used to replicate issue and I will have a look. Will close off issue 5.

On Wed, 8 Jun 2022, 18:20 Collins Adom Baffour, @.***> wrote:

I also generated the ipm fine and it worked fine but please try encoding from the cp500 to latin1. It will throw an error, can you check that part for me.

— Reply to this email directly, view it on GitHub https://github.com/adelosa/cardutil/issues/5#issuecomment-1149610198, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBUU24FY7YEA6UBZDZHP4TVOBJUTANCNFSM5WWKQSMQ . You are receiving this because you commented.Message ID: @.***>

charisad commented 2 years ago

Thanks Anthony

charisad commented 2 years ago

Sounds like a different issue. Can you please raise as new item in GitHub with file and commands used to replicate issue and I will have a look. Will close off issue 5.

On Wed, 8 Jun 2022, 18:20 Collins Adom Baffour, @.***> wrote:

I also generated the ipm fine and it worked fine but please try encoding from the cp500 to latin1. It will throw an error, can you check that part for me.

— Reply to this email directly, view it on GitHub https://github.com/adelosa/cardutil/issues/5#issuecomment-1149610198, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACBUU24FY7YEA6UBZDZHP4TVOBJUTANCNFSM5WWKQSMQ . You are receiving this because you commented.Message ID: @.***>

Thanks Anthony