UnderstandLingBV / Tuktu

Big Data Science Swiss Army Knife - http://www.tuktu.io --
http://www.understandling.com
Other
60 stars 16 forks source link

enrich the processor API with a cleanUp method #29

Closed dmassart closed 8 years ago

dmassart commented 8 years ago

Currently there is a "def initialize(config: JsObject)" method that allows to initialize a processor before the first data packet is processed but there is no "def cleanUp()" (not sure what the antonym of 'initialize' is) that allows to execute code after all the data packets have been completely processed. This would enable the possibility to better manage resources such as connections to services and databases.

Note that this is different than "compose Enumeratee.onEOF" that is triggered after the processing of the last data packet has started but does not ensure that the processing of all these data packets has completed.

ErikTromp commented 8 years ago

So how would that work then? And for what use case is this relevant?

For example, if you have a flow that kicks off another flow (embedded in a meta processor for example), how would you go by determining if something has finished?

Also: if you are using this for managing connection pools, you are probably better off improving the connection pool to keep track of leases to a single connection, release the lease on cleanUp() and close the connection in the pool if there are no more leases.