search

instaclick / PDI-Plugin-Step-AMQP

AMQP Plugin for PDI
MIT License
10 stars 14 forks source link

Blocking During Wait Timeout #30

Open jwatt-cognito opened 8 years ago

jwatt-cognito commented 8 years ago

Hi,

We want to use this plugin as part of our event processing system which writes event data to MySQL after it arrives in the AMQP queue. We have enabled the Wait for data option. We want to specify a very long WaitTimeout so the transformation will effectively wait indefinitely for messages to arrive.

1) We are seeing some behaviour we are not expecting. The step processes messages but does not ACK them off the queue until the timeout has expired. Obviously with a very long timeout the messages will remain unacknowledged for a long time.

2) We believe database transactions are not being committed until the timeout has expired. Again, with a very long timeout the transactions will remain uncommitted for a long time.

3) When running the transformation through the Spoon IDE we see the AMQP connections are left open. We have checked that this does not happy when running the transformation on the command line but this issue does impact our developers ability to work as they must constantly kill orphaned connections.

Any help would be much appreciated.

James Watt Cognito iQ

dshvedchenko commented 8 years ago

Hi, It is by design. in Transactional mode all ACK/NACK tags collects and processed at the end of mapping. 3) which PDI version do you use ? plugin were compiled for 5.3. And also tested in commad line mode. So this bug can be overseen.

FabioBatSilva commented 8 years ago

1 We are seeing some behaviour we are not expecting. The step processes messages but does not ACK them off the queue until the timeout has expired. Obviously with a very long timeout the messages will remain unacknowledged for a long time.

That is usually not a good idea, (see 2).

I suggest you take different approach instead of running it indefinitely i think you should operate over a bach of messages.

So have your transformation run with a message limit of x at the and of the transformation all transactions will get committed and messages acknowledged.

You can have the whole job running every x minutes or have the transformation that deals with the message included by another job that triggers it.

This way every bach of x message has transactional semantics.

2 We believe database transactions are not being committed until the timeout has expired. Again, with a very long timeout the transactions will remain uncommitted for a long time.

That is the expected behaviour in a transactional transformation, database transactions get committed/rollback only at the end of a transformation the same is true for this plugin when running in transactional mode.

ajay7779 commented 6 years ago

Hi, I am using this plugin for the first time so how do we set Wait for Messages,Wait Timeout and delay the Ack Msg Step DeliveryTag Field in the plugin as those options are not available in the plugin frontend

Could someone help?

Thanks