search

tilln / jmeter-iso8583

ISO8583 Plugin for JMeter
MIT License
62 stars 32 forks source link

Not receiving response data (body or header) in ISO 8583 using JMeter5.4 #69

Closed Rahulkhotre closed 1 year ago

Rahulkhotre commented 1 year ago

@tilln I am getting Blank response in view results tree. I can see request is received and response processed successfully at the destination server. I believe it might be the issue of MUX not matching DE 11 + 41. I have not added any fields for MUX Key configuration & MTI mapping. can you tell me what fields I need to add to match the MUX with the response. Note- I don't have field 41 in my request and in xml file too.

image image

ISO8583Issue_v1.txt

Packager- org.jpos.iso.packager.GenericPackager PFA JMeter Logs.

tilln commented 1 year ago
@Rahulkhotre As your message does not contain field 41 which is a default mapping field you may likely be able to fix this by configuring key fields in the Mux Key Configuration. You may want to try adding an entry with "Key Fields" 11 and perhaps 7 or another field that has a unique value for each handshake: MTI Key Fields
2804 7 11
Rahulkhotre commented 1 year ago

@tilln I have added below entry in Mux Key Configuration in JMeter . But still getting timeout. JMeter logs already attached while raising the issue. Do I need to provide any value for the MTI mapping in Mux Settings in JMeter and if yes then what value?

As my message does not contain field 41 which is a default mapping field, do I need to ask developer to add this field in isopackager xml file?

<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

MTI | Key Fields -- | -- 2804 | 7 11

tilln commented 1 year ago

@Rahulkhotre

JMeter logs already attached while raising the issue.

Well, firstly I can't seem to access them and secondly, they wouldn't be relevant any more after you changed the mux config. So please upload logs, with debug level and package logging enabled.

Do I need to provide any value for the MTI mapping in Mux Settings in JMeter and if yes then what value?

No, assuming your 280x request maps to a 281x response, so the default will work.

As my message does not contain field 41 which is a default mapping field, do I need to ask developer to add this field in isopackager xml file?

No, assuming you can match based on fields 7 and 11.

Rahulkhotre commented 1 year ago

@tilln Thanks for responding! Please find attached JMeter Logs with debug level (I am not sure how to do package logging enabled)

JMeter Logs.txt

tilln commented 1 year ago

Apologies, I just realised that the delimiter needs to be , so you want to enter:

MTI Key Fields
2804 7, 11

Mind the space after the comma.

Refer: https://github.com/jpos/jPOS/blob/v2_1_8/jpos/src/main/java/org/jpos/q2/iso/QMUX.java#L78-L85

Rahulkhotre commented 1 year ago

@tilln I have tried that also but still getting timeout error and response tab is blank in JMeter.

image
tilln commented 1 year ago

Sorry again, ignore the previous comment. The Mux uses StringTokenizer and either space or comma as delimiters.

Your logs show that you are not receiving a response so the timeout you're seeing is genuine.

Please verify that the request arrives at the server and is accepted and a response is sent.

Rahulkhotre commented 1 year ago

@tilln

On the destination logs, I can see payload request is receiving and processing by the server and response payload is being sent also.

tilln commented 1 year ago

Can you confirm that the response from the server contains the same values for fields 7 and 11 as the request?

Can you run Wireshark on the JMeter host to confirm that the response actually arrives there?

Rahulkhotre commented 1 year ago

I have checked servers logs and Yes, the response from the server contains the same values for fields 7 and 11 as the request.

only change I can see in the response is that in response I am getting "Type" : "2814" where as in request I am sending Type:2804.

Note- we have packet sender utility from which we are getting the response. Wireshark I have not checked but will check and confirm you

Rahulkhotre commented 1 year ago

@tilln

I have ran Wireshark on the JMeter host and response actually arrives there.

tilln commented 1 year ago

@Rahulkhotre That's great. What does it look like? Looking at the binary data may provide some clues about why the response is not received by the channel.

Rahulkhotre commented 1 year ago

28046220010000000006105818496700000000202338220438500438508310070029060012UCS00000000028146220010002000006105818496700000000202338220438500438508310000070029060012UCS000000000.

please let me know if you need any other details.

tilln commented 1 year ago

What about the actual bytes, including message lengths etc.? Wireshark screenshot perhaps?

How long does it take for the response to arrive? Possibly longer than your 10 seconds?

I'm afraid I won't really be able to help you if I can't reproduce this issue.

Rahulkhotre commented 1 year ago

Hi,

I have added details at the below URL. please check and let me know if you need more details https://stackoverflow.com/questions/76299539/not-receiving-response-data-body-or-header-for-iso-8583-in-jmeter5-4 Less than 10 secs it takes for the response to arrive

Rahulkhotre commented 1 year ago

@tilln Wireshark screenshots:

  1. image
  2. image
  3. image
  4. image
Rahulkhotre commented 1 year ago

I can do the screen sharing also if you need

chhil commented 1 year ago

Request 28046220010000000006105818496700000000202338220438500438508310070029060012UCS00000000028146220010002000006105818496700000000202338220438500438508310000070029060012UCS000000000 Response 28146220010002000006105818496700000000202348210348140348148310000070029060012UCS000000000

Provide the packager file you have configured so that we can unpack the message and see what the field data looks like.

Rahulkhotre commented 1 year ago

rdc-fields.txt

PFA file you requested.

Request 28046220010000000006105818496700000000202338220438500438508310070029060012UCS000000000

Response 28146220010002000006105818496700000000202348210348140348148310000070029060012UCS000000000

tilln commented 1 year ago

@Rahulkhotre Please upload a Wireshark screenshot with the line selected that shows the response, for example image

I want to see the message lengths please.

Can you also confirm you are using ASCIIChannel as per your logs and screenshots? I get different byte sequences with 134 bytes request and 137 bytes response.

What seems most conspicuous is the trailing . character in the response.

Rahulkhotre commented 1 year ago

@tilln I am using LogChannel. When I was using ASCIIChannel, I was getting error at the server side. It was prefixing the 4 digits to the request payload, due to this it was giving error at the server side and no response in JMeter. Hence I am using LogChannel now still no response in JMeter.

image image
chhil commented 1 year ago

Please determine the correct channel to use. If you use logchannel then make sure the response sender also uses the log channel format. You tried to fix the channel by changing AsciiChannel to LogChannel and it worked for the receiver, but is the receiver sending the response in the logchannel format? Align the channel correctly, read the spec, give it some thought.

Rahulkhotre commented 1 year ago

I have used log channel and its sending the correct request payload to the server and server also processing it correctly and generating the correct response payload. But the problem is JMeter giving timeout and blank response.

chhil commented 1 year ago

But the problem is JMeter giving timeout and blank response.

It is not jmeter plugin problem, its you selecting an incorrect channel.

LogChannel expects the message inside

`

message

` Your responder is not doing that hence LogChannel does not see the message.

Someone accepting a message needs to know what the start and end of your message is. Channels either prepend a length so that it knows when a message starts and when one ends (e.g. AsciiChannel), or have some start and end xml tags or whatever your system does to indicate the message started and ended (delimiters). Rememberr TCP is stream based and 100 messages could be lined up in the tcp stack and there needs to be a way to determine when one ends and another starts.

I suggest you first figure out whats the appropriate channel for your communication.

A simple test

Point your jmeter to a netcat server listening on the appropriate port. Run a single jmeter test. Let the netcat server receive the request. Paste the response of the message inside response and it should work.

Rahulkhotre commented 1 year ago

dev team confirmed that appropriate channel for communication is ASCIIChannel.

But when I use this channel in JMeter , I am getting below error at the server side. do we get response in JMeter in case of failures?

Message : Invocation of Method 'parseISOMessage(java.lang.String,java.lang.String)' from Class 'com.ld.rdc.iso.UnpackISOMessage' with arguments [java.lang.String productIndicator, java.lang.String isoMsg] resulted in an error. Expected arguments are [java.lang.String productIndicator, java.lang.String isoMsg]. Cause: java.lang.Exception - org.jpos.iso.ISOException: field packager '5' is null unpacking field=5, consumed=26

chhil commented 1 year ago

With asciichannel configured JMeter will send (0086 is length added by AsciChannel followed by data)

008628046220010000000006105818496700000000202338220438500438508310070029060012UCS000000000

Your application should send (0089 length followed by data) 008928146220010002000006105818496700000000202348210348140348148310000070029060012UCS000000000 Asciichannel will read the length as 0089 and parse the message data correctly.

Do the above and it should work. Capture wireshark data and share the text (not the image).

If you are getting an error at the server side make sure the packager configured in jmeter is the same on the server side or atleast uses the same field logic with whatever it uses to parse it. If it complains for field 5 means the bitmap is not being processed correctly and the parsed bitmap is indicating that DE 5 is there but its not actually there.

Rahulkhotre commented 1 year ago

In wireshark I can see below follow tcp stream- 008628046220010000000006105818496700000000202304231004391004398310070029060012UCS00000000028148200000002000000000000000000000020230523070439606

Frame 451: 107 bytes on wire (856 bits), 107 bytes captured (856 bits) on interface \Device\NPF{A98855A0-A795-4774-838F-B5F8F19409E8}, id 0 Section number: 1 Interface id: 0 (\Device\NPF{A98855A0-A795-4774-838F-B5F8F19409E8}) Encapsulation type: Ethernet (1) Arrival Time: May 23, 2023 10:04:39.951661000 Coordinated Universal Time [Time shift for this packet: 0.000000000 seconds] Epoch Time: 1684836279.951661000 seconds [Time delta from previous captured frame: 0.016626000 seconds] [Time delta from previous displayed frame: 0.029434000 seconds] [Time since reference or first frame: 6.644758000 seconds] Frame Number: 451 Frame Length: 107 bytes (856 bits) Capture Length: 107 bytes (856 bits) [Frame is marked: False] [Frame is ignored: False] [Protocols in frame: eth:ethertype:ip:tcp:data] [Coloring Rule Name: TCP] [Coloring Rule String: tcp] Ethernet II, Src: 02:d8:78:08:49:62 (02:d8:78:08:49:62), Dst: 02:58:54:d8:f2:d2 (02:58:54:d8:f2:d2) Destination: 02:58:54:d8:f2:d2 (02:58:54:d8:f2:d2) Address: 02:58:54:d8:f2:d2 (02:58:54:d8:f2:d2) .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Source: 02:d8:78:08:49:62 (02:d8:78:08:49:62) Address: 02:d8:78:08:49:62 (02:d8:78:08:49:62) .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default) .... ...0 .... .... .... .... = IG bit: Individual address (unicast) Type: IPv4 (0x0800) Internet Protocol Version 4, Src: , Dst: 192.172.10.150 0100 .... = Version: 4 .... 0101 = Header Length: 20 bytes (5) Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT) Total Length: 93 Identification: 0x3729 (14121)

  1. .... = Flags: 0x2, Don't fragment ...0 0000 0000 0000 = Fragment Offset: 0 Time to Live: 254 Protocol: TCP (6) Header Checksum: 0x35ad [validation disabled] [Header checksum status: Unverified] Source Address: Destination Address: 192.172.10.150 Transmission Control Protocol, Src Port: 8081, Dst Port: 8534, Seq: 1, Ack: 91, Len: 53 Source Port: 8081 Destination Port: 8534 [Stream index: 6] [Conversation completeness: Complete, WITH_DATA (31)] [TCP Segment Len: 53] Sequence Number: 1 (relative sequence number) Sequence Number (raw): 3287276264 [Next Sequence Number: 54 (relative sequence number)] Acknowledgment Number: 91 (relative ack number) Acknowledgment number (raw): 1926203599 0101 .... = Header Length: 20 bytes (5) Flags: 0x018 (PSH, ACK)
    1. .... .... = Reserved: Not set ...0 .... .... = Accurate ECN: Not set .... 0... .... = Congestion Window Reduced: Not set .... .0.. .... = ECN-Echo: Not set .... ..0. .... = Urgent: Not set .... ...1 .... = Acknowledgment: Set .... .... 1... = Push: Set .... .... .0.. = Reset: Not set .... .... ..0. = Syn: Not set .... .... ...0 = Fin: Not set [TCP Flags: ·······AP···] Window: 211 [Calculated window size: 27008] [Window size scaling factor: 128] Checksum: 0x8e8d [unverified] [Checksum Status: Unverified] Urgent Pointer: 0 [Timestamps] [Time since first frame in this TCP stream: 0.063294000 seconds] [Time since previous frame in this TCP stream: 0.029434000 seconds] [SEQ/ACK analysis] TCP payload (53 bytes) Data (53 bytes) Data: 3238313438323030303030303032303030303030303030303030303030303030303030303230323330353233303730343339363036 [Length: 53]
chhil commented 1 year ago

This is the request response stream

008628046220010000000006105818496700000000202304231004391004398310070029060012UCS00000000028148200000002000000000000000000000020230523070439606 Request Part 008628046220010000000006105818496700000000202304231004391004398310070029060012UCS000000000 Response Part 28148200000002000000000000000000000020230523070439606

As you can see request part has a 4 byte 0086 length preceding it, that was added by ascii channel.

There is no 4 byte length header in the response, so jmeter jpos plugin takes the first 4 bytes of message (mti) and treats it as length and everything gets parsed incorrectly after that.

For ascii channel to understand the message ,0053 (the length of the response) needs to prefix the data.

Length prepended response data 005328148200000002000000000000000000000020230523070439606

You need to understand when we say ascii channel, its what adds the 4 byte length header. The data in the message is governed by the packager which is set to ascii for all fields. So do not confuse ascii channel with packager that is ascii for all your fields.

How is your server understanding a request that has a 4 byte length header but not responding with a 4 byte length header?

If this is what your server sends, I do not think there is any channel that will understand this as all channels need some sort of header that has a length or some start and end delimiter. As I have mentioned TCP is stream based, consider a huge byte array that has 50 messages in it and arrive, without any length or delimiters, there is no way to determine where each message starts and ends.

Rahulkhotre commented 1 year ago

How is your server understanding a request that has a 4 byte length header but not responding with a 4 byte length header?

I think server is also not understanding a request that has a 4 byte length header because I can see error at the server side.

When I am sending request with LogChannel, server is not giving error at the server side.

what do you think what is the issue here is it Channel or Packager file?

chhil commented 1 year ago

When I am sending request with LogChannel, server is not giving error at the server side.

Like I mentioned earlier LogChannel needs a <isomsg> message </isomg> when it receives data. While sending it sends whatever data has been asked for it to send. LogChannel is more for reading log file dta which have xml data

e.g. The log would have something like this and the LogChannel will extract the iso message from it. Its not something you would use.

<log realm="/127.0.0.1:4444" at="2023-05-23T14:05:26.236" lifespan="13940ms">
  <receive>
    <isomsg direction="incoming">
      <!-- org.jpos.iso.packager.GenericPackager[/Volumes/ZDrive/temp/jposmailinglist/jpos-2.1.1-SNAPSHOT/src/packager.xml] -->
      <field id="0" value="2814"/>
      <field id="2" value="5818496700"/>
      <field id="3" value="000000"/>
      <field id="7" value="20234821034814"/>
      <field id="11" value="034814"/>
      <field id="24" value="831"/>
      <field id="39" value="000"/>
      <field id="62" value="0029060"/>
      <field id="63" value="UCS000000000"/>
    </isomsg>
  </receive>
</log>

As I mentioned earlier if your server is not prepending the message, you will not be able to use this jmeter plugin that use opensource jpos for iso message processing. If the underlying jpos cannot handle it, chances are no entity that your server talks to in a prod environment will be able to. I would suggest the server developers look atthe spec they are implementing , as it surely feels like they have missed the prefixing of length.

what do you think what is the issue here is it Channel or Packager file?

The server is the issue , iso message part is correct but in all likelihood not implemented the transportation (jmeter plugin channel) headers that are needed.

Rahulkhotre commented 1 year ago

ok thanks. I will check why at server side its failing due to prefix that is coming in the request and why its not sending prefix in the response payload as well.

Rahulkhotre commented 1 year ago

As per dev team, they would get request payload without any length prefix and server will respond with response body without any length prefix but it will end with the postfix \03

Also, The debug log should contain Channel output similar to the following:

2022-02-22 12:34:56,789 DEBUG n.c.b.j.i.Q2: (channel/HOSTNAME:PORT) [send] Out: 2022-02-22 12:34:56,987 DEBUG n.c.b.j.i.Q2: (channel/HOSTNAME:PORT) [receive] In: But in my case I am not getting the second line in logs. Does it mean my request/packager itself is having some issue?

tilln commented 1 year ago

@Rahulkhotre You could try to use BASE24Channel (if your server accepts a trailing 0x03 byte) or alternatively write your own Channel class.

Extending BaseChannel would be the easiest way, and you would want to override streamReceive, similar to what BASE24Channel does: https://github.com/jpos/jPOS/blob/v2_1_8/jpos/src/main/java/org/jpos/iso/channel/BASE24Channel.java#L97

tilln commented 1 year ago

But in my case I am not getting the second line in logs. Does it mean my request/packager itself is having some issue?

That may also be the case, but as we already established, it is the incorrect Channel Class that prevents the message from being received correctly. (I have updated the FAQ section to hopefully make that clearer.)

Rahulkhotre commented 1 year ago

I have received new server where it will accept the request with length prefix. But how do I send below value (externalMessageHeader) in JMeter. "MTI": "0200", "externalMessageHeader": "ISO015000010", "productIndicator": "atm",

when I have added externalMessageHeader in Message Header (Hex), JMeter its not sending that value in JMeter request at all. Please check below screenshot. Request payload should have started with the externalMessageHeader value and then MTI value.

image
tilln commented 1 year ago

If the header was in fact not sent, a separate issue should be opened for it. However, I believe the header is in fact sent, as you will be able to verify in Wireshark, but it is not displayed in the hexdump.

I suggest you set the header on the Channel instead (string not hex though) as this will enable the Channel to deal with the response header.

Rahulkhotre commented 1 year ago

yes header is sent now. But JMeter adding length prefix and due to this its giving Invalid ISO message at the server.

Issue is at the server only because they have not implanted system as per international standard and due to this its not receiving message with the length prefix and also not sending the response with the length prefix.

tilln commented 1 year ago

@Rahulkhotre

Issue is at the server only

If there is no issue with this plugin, can you please close this issue.

Rahulkhotre commented 1 year ago

ok...is it possible to instruct JMeter/plugin to not add length prefix while sending the request to the server?

chhil commented 1 year ago

If you are using BASE24Channel in the plugin channel configuration, it does not send length header or expect one, it sends the 0x03 trailer and expects the server to send a 0x03 trailer.

If you cannot find a channel that meets your needs, ask your developer to write a custom channel that you can use in the plugin. Already mentioned in https://github.com/tilln/jmeter-iso8583/issues/69#issuecomment-1561877640

Rahulkhotre commented 1 year ago

Hi, thanks for the help. its working now after changing the channel class to base24channel.