no bit 41 causing Upstream error: ISO host sent data with no client connected! Ignoring...

[niko95kus]

Hi igor,

i'm trying to implement socket queue. I run the socket queue using this command node socketQueue.js --upstreamHost=192.168.1.35 --upstreamPort=5036 --listenHttpPort=8080 --noAutoReversal --vv --useLengthHeader --lengthHeaderType=hex --hostConfig=alto host config file alto.txt

when I try purchase operation (0200) it's working fine, but when I try to echo / sign on (0800), it shows Upstream error: ISO host sent data with no client connected! Ignoring... Here are my json request {"0":"0800", "2":"6048200000002731", "7":"0109102806","11":"1", "70":"301"}

json response {"result": "","errors": ["Gateway timeout"]}

socket queue stack trace 2018-01-11 14:56:52 - info: Client http:::1:52649 connected 2018-01-11 14:56:52 - info: Client http:::1:52649 sent data 2018-01-11 14:56:52 - verbose:

http:::1:52649

 [Echo Request]

 Message Type Indicator [0].......................0800
 Bitmap [1].......................................0400000000000000
 Primary Account Number [2].......................604820******2731
 Transmission Date and Time [7]...................0109102806
 System Trace Audit Number [11]...................000001
 Network Management Information Code [70].........301

================================================================================================

2018-01-11 14:56:52 - info: Writing to queue http:::1:52649 [0] 2018-01-11 14:56:52 - info: New queue item 2 2018-01-11 14:56:52 - info: Processing queue [pending 1 / total 1] 2018-01-11 14:56:52 - verbose: Queue keys dump: 2 2018-01-11 14:56:52 - info: Upstreaming data for http:::1:52649 2018-01-11 14:56:52 - verbose: [.I0800c2...] 2018-01-11 14:56:52 - info: Got data from ISO-host (77b) 2018-01-11 14:56:52 - verbose: [K.0810C2...] 2018-01-11 14:56:52 - info: Upstream error: ISO host sent data with no client connected! Ignoring...

ISO host says

 [Echo Response]

 Message Type Indicator [0].......................0810
 Bitmap [1].......................................0400000000000000
 Primary Account Number [2].......................604820******2731
 Transmission Date and Time [7]...................0109102806
 System Trace Audit Number [11]...................000001
 Response Code [39]...............................00 (Approve Transaction)
 Network Management Information Code [70].........301

================================================================================================

2018-01-11 14:57:27 - info: Queue notice: http:::1:52649 reached queue timeout 2018-01-11 14:57:27 - info: Dropping the client http:::1:52649 2018-01-11 14:57:27 - info: Client http:::1:52649 gateway timeout 2018-01-11 14:57:32 - info: Client http:::1:52649 event 2018-01-11 14:57:32 - verbose: Releasing terminal null with the key 2 2018-01-11 14:57:32 - info: Warning: client http:::1:52649 [queue id 2] terminated activity

based on your explanation, is it because I don't send bit 41 in the json request? or did I miss something?

I tried to send bit 41 in the echo and sign on request, but the ISO host didn't response my request

Please help Thanks

[juks]

Hi!

Yes, it expects the field 41 to identify the connected terminal. Could you please provide the same case where field 41 is present?

[niko95kus]

json request {"0":"0800", "2":"6048200000002731", "7":"0109102806","11":"1","41":"02700301", "70":"301"}

json response {"result": "","errors": ["Gateway timeout"]}

socket queue stack trace

2018-01-11 15:33:23 - info: Client http:::1:60056 connected 2018-01-11 15:33:23 - info: Client http:::1:60056 sent data 2018-01-11 15:33:23 - verbose:

http:::1:60056

 [Echo Request]

 Message Type Indicator [0].......................0800
 Bitmap [1].......................................0400000000000000
 Primary Account Number [2].......................6048200000002731
 Transmission Date and Time [7]...................0109102806
 System Trace Audit Number [11]...................000001
 Card Acceptor Terminal Identification [41].......02700301
 Network Management Information Code [70].........301

================================================================================================

2018-01-11 15:33:23 - info: Writing to queue http:::1:60056 [0] 2018-01-11 15:33:23 - info: New queue item 1 2018-01-11 15:33:23 - info: Processing queue [pending 1 / total 1] 2018-01-11 15:33:23 - verbose: Queue keys dump: 1 2018-01-11 15:33:23 - info: Upstreaming data for http:::1:60056 2018-01-11 15:33:23 - verbose: [.Q0800c2200000008000000400000000000000166048200000002731010910280600000 102700301301] 2018-01-11 15:33:23 - verbose: [00 51 30 38 30 30 63 32 32 30 30 30 30 30 30 30 38 30 30 30 30 30 30 34 30 30 30 30 30 30 30 30 30 30 30 30 30 30 31 36 36 30 34 38 32 30 30 30 30 30 30 30 32 37 33 31 30 31 30 39 31 30 32 38 30 36 30 30 30 30 30 31 30 32 37 30 30 33 30 31 33 30 31] 2018-01-11 15:33:23 - info: Got data from ISO-host (24b) 2018-01-11 15:33:23 - verbose: [..0810000000000200000000] 2018-01-11 15:33:23 - verbose: [16 00 30 38 31 30 30 30 30 30 30 30 30 30 30 32 30 30 30 30 30 30 30 30 ] 2018-01-11 15:33:23 - info: Upstream error: ISO host sent data with no client connected! Ignoring...

ISO host says

 [Echo Response]

 Message Type Indicator [0].......................0810
 Bitmap [1].......................................0000000200000000
 Response Code [39]...............................

================================================================================================

2018-01-11 15:33:58 - info: Queue notice: http:::1:60056 reached queue timeout 2018-01-11 15:33:58 - info: Dropping the client http:::1:60056 2018-01-11 15:33:58 - info: Client http:::1:60056 gateway timeout 2018-01-11 15:34:03 - info: Client http:::1:60056 event 2018-01-11 15:34:03 - verbose: Releasing terminal 02700301 with the key 1 2018-01-11 15:34:03 - info: Warning: client http:::1:60056 [queue id 1] terminated activity

the echo response didn't meet the expectation as seen above

Is there any way to get the response without the present of field 41?

[juks]

This is easy to imagine: you have many connections on one hand, sending you messages that you pass through the single connection on the other hand to the ISO upstream. You need some way to dispatch the responses coming via one connection to the clients, each waiting in it's own connection.

Like NAT server: each connection is proxied via corresponding source port number, where port number acts in a way as terminal id in the current terminology.

Also according to your output the field 39 format sent by the ISO server is no 'ans' but i guess 'hn'. You can see the code 0x00, so this is no alphabetic zero.

[juks]

What kind of ISO host you try to communicate with?

[niko95kus]

Based on your terminology, it means that every request to the socket queue should have different terminal id? Example if there are several requests with the same terminal id, the socket queue can send the wrong response data to the client?

Did you mean field 39 type should be 'hn' (currently 'an' in my config)? I tried to change it to 'hn', but the response parsed wrongly

The ISO host is using JPOS and i'm trying to communicate from php

Actually there is a case where the ISO host will send echo to socket queue and I need to send the echo reply to the ISO host. Can this be done with socket queue?

[juks]

Not every request but every client should have it's terminal id (TID). Is is 8583 terminology. Every request sent by the terminal is supplied with system trace audit number (STAN). If there will be two clients with one TID SocketQueue will queue the transactions on order of appearance.

When SocketQueue receives the response from the upstream server is looks for the key pair - TID+STAN in the connections pool to dispatch the answer.

I have not seen any system that does not provide stan+tid for echo messages.

'Hn' should work but mind the length is 1, not 2. If not I will hace a chance to reproduce it later.

[niko95kus]

I've tried changing field 39 to 'hn' with length 1 and it's not working either. '39': { length: 1, name: 'Response Code', type: 'hn', alias: '' }, For field 39, I think I will keep it with 'an' and length 2. Thanks for your suggestion

Actually the echo and sign on message have the stan, but there is no TID. These are the fields required for the 0800/0810 message in my case. field 1, 7, 11, 39, and 70 So, for echo and sign on there is a stan on field 11. Is there any workaround that I can use to make the socket queue able to send the received message to the client, based on the stan only (without TID)? In my case, there will be only 1 TID in 1 socket queue. So, actually the message's identifier is the stan, because the TID always be the same. Is this possible with socket queue?

[juks]

I could consider this kind of improvement, but not right now.

You can play with the source yourself if you like:

• Remove terminal id check here https://github.com/juks/SocketQueue/blob/4badee10f03764edfd4efcf71b44692146bccb86/lib/iso8583-queue/lib/iso8583Queue.js#L232

• Add check for null value here https://github.com/juks/SocketQueue/blob/4badee10f03764edfd4efcf71b44692146bccb86/lib/iso8583-queue/lib/iso8583Queue.js#L143 https://github.com/juks/SocketQueue/blob/4badee10f03764edfd4efcf71b44692146bccb86/lib/iso8583-queue/lib/iso8583Queue.js#L132

I might miss something but this is all in this file.

[niko95kus]

Done. Thank you for your instruction.

Besides, I want to enhance the socket queue with auto reply for network message (such as echo and cut off). The idea is when the ISO host send echo (0800 message), the socket queue will automatically reply with 0810 message. Can I add some custom logic just like the auto reversal logic?

Then, can I make a daily log file and maximum daily log file? So, when the maximum daily log file reached, the old log file will automatically deleted.

I think this would be great. Please advise. Thanks

[juks]

There is EchoServer inside SocketQueue: https://github.com/juks/SocketQueue/blob/master/lib/testSuite/lib/echoServer.js. You can add your own response logic for any sort of message. The actions should be implemented in "SocketBank" module: https://github.com/juks/SocketQueue/blob/master/lib/testSuite/lib/socketBank.js. You can see what is already implemented.

As for log rotating. It is fine to use the standard means like logrotate to rotate the logs on daily or size basis. After each rotation the HUP signal should be sent to SocketQueue by it's PID. You can add --pidFile option, so that logrotate will be able to read the process id.

[juks]

Added the "queueMessageKeyFields" configuration parementer. Default value is "11,41" (STAN, TID).

[juks]

Probably, will work on another option that disables terminal lock. So TID value becomes no longer obligatory.

[niko95kus]

Hi Igor, thanks for the update, waiting for the next update to make the TID value become optional :)

In the meantime, I still can't figure it out how to make auto reply for network message (0800) just like your EchoServer did.

Take this example: A = Bank POS ISO HOST B = Socket Queue C = JSON HTTP Client

As you mention in the docs, the connection will looks like this A - B - C

Here's the simulation C send echo / sign on (0800), the request will go like this C -> B -> A Then A, will reply with 0810 C <- B <- A The example above works fine.

But, sometimes A will send the echo (0800) and I have to reply it with 0810 message. What I'm trying to achieve is like this. A send echo, the request will go like this A -> B Then B automatically reply with 0810 based on the logic I set B -> A I want to achieve this, because as mention above, A can initiate the echo (0800), even when there is no client (C) connected. Can this be done with socket queue? Please advise.

Thank you

[juks]

What you mean is kind of routing logic right?

[niko95kus]

Yes, it's kind of routing logic. So, the EchoServer only responding to the network message request from the Bank POS ISO HOST using the established connection (existing upstreamHost and upstreamPort, not opening new echoServerPort)

[juks]

The complexity of the idea of dealing with multiple upstream connections and routing between them is too high to put it in the current implementation without major refactoring.

So what I can suggest is you break in into the source again. For example, see this method:

https://github.com/juks/SocketQueue/blob/be60e90a685eded685173201c64c1d6583358526/lib/socketServer/lib/upstream.js#L72

Here, right in the start, you can place your own "if". This condition, based on "data" with decide whether to use local call to socket bank (for instance) to provide the answer for source message. If so, no need to queue the message. You can just call sender.reply(someData), then return true.

Not much elegant but better then nothing.

[niko95kus]

the sendData method is for sending data to the ISO host. is it true?

Anyway, I've managed for the auto reply. I added socket bank test suite inside upsteam.js and modified the logic inside this condition https://github.com/juks/SocketQueue/blob/be60e90a685eded685173201c64c1d6583358526/lib/socketServer/lib/upstream.js#L253-L259 become this

if(p.getField(0) == '0800'){
  dd('Auto reply network code ' + p.getField(70) + ' to ISO host');

  rp = parent.bank.replyPacket(p);
  if (rp) this.write(rp.getMessage({
                                      staticHeader: global.c['useStaticHeader'],
                                      lengthHeader: global.c['useLengthHeader']
                                  }));
  dd('Echo server sent auto reply response');
}
else{
  message = 'Upstream error: ISO host sent data with no client connected! Ignoring...';
  message += '\n\n' + 'ISO host says' + p.pretty();
  dd(message.red);
}

return false;

Do you think this will make a mess?

Besides, in what condition this condition executed? You said that "Found item with dead sender", but I don't get it. https://github.com/juks/SocketQueue/blob/be60e90a685eded685173201c64c1d6583358526/lib/socketServer/lib/upstream.js#L260-L266

Then, I add redis client too in upsteam.js to save the ISO response. Now, I want to close the redis connection when the socket queue is killed or restarted. Any idea in where should I close the redis connection? Please advise.

Thank you

[juks]

the sendData method is for sending data to the ISO host. is it true? Got it. This was misunderstanding. I supposed you consider the client application to send echo message not the host. But you got the direction. Can't say whether it is messy or not: you are the captain of your implementation :-)

Found item with dead sender Means a message in the queue with the Socket instance that is already closed. So if the application will try to send data where the will be a sort of "Broken pipe" error.

Now, I want to close the redis connection when the socket queue is killed or restarted. Any idea in where should I close the redis connection? Feel free to place it here https://github.com/juks/SocketQueue/blob/a5399b5bdd8b82497a6214a3bb2e56348cfb1c93/socketQueue.js#L285

[niko95kus]

Thanks for your guide. Will try my implementation in the live system. Waiting for your next feature update :)

[juks]

Added boolean queueLockTerminal option. So terminal id can now be optional field thanks to queueMessageKeyFields option, implemented before.

[niko95kus]

The queueLockTerminal update doesn't work for me. I've set the queueLockTerminal to false and tried to echo with this JSON {"0":"0800", "2":"6048200000008123", "7":"0109102806","11":"1", "70":"301"} the result remain the same "Upstream error: ISO host sent data with no client connected! Ignoring..."

Is there any changes should be made in this function regarding the queueLockTerminal implementation? I checked that I still get messageIdentified variable false when sending ISO message with no TID. https://github.com/juks/SocketQueue/blob/2f7b6519fce1c615be84ffdd672dae8cdbb35878/lib/iso8583-queue/lib/iso8583Queue.js#L206-L232

[juks]

As I said above, this option together with queueMessageKeyFields makes it possible to work without field 41 (TID).

Please see this discussion: https://github.com/juks/SocketQueue/issues/22

[niko95kus]

Got it. Thanks, now it's working.

I see that you reduce the queue size from 500 to 100. Is there any performance issue regarding this? https://github.com/juks/SocketQueue/blob/2f7b6519fce1c615be84ffdd672dae8cdbb35878/lib/iso8583-queue/lib/iso8583Queue.js#L5

[juks]

No issues, just a default. Tested at 1000 with no visible problems.

[juks]

Also: this is not perfomance related: in case there is an upstream connection failure you may not want to accept new message before it gets solved. This is regulated by the queue size in a way.

To utilize the queue of 100 items completely with transactions flow, this flow should be quite intensive.

[niko95kus]

All right then. Thanks for your help :)

[juks]

Welcome :-)

[shoaibjdev]

In the latest version have to comment below line in file SocketQueue/lib/iso8583-queue/lib/iso8583Queue.js to override terminal id check. For some messages specially network messages (800 or 1800) terminal Id is usually not returned in response from upstream and is really not required to be validated.

/if (!p.getField(id) || p.getField(id) != qe.packet.getField(id)) { messageIdentified = false; break; }/

[juks]

Hi! Did you set the queueMessageKeyFields parameter without field 41?

[rustamin]

Hallo @niko95kus

May I know how you implement response echo request from ISO host? Thank you.

[niko95kus]

Hi @rustamin I've managed to make the auto reply by modifying the code. Take a look here

Hope it helps

[juks]

Just implemented upstream echo response as it was requested in #28