search

Informatievlaanderen / VSDS-Linked-Data-Interactions

https://informatievlaanderen.github.io/VSDS-Linked-Data-Interactions/
European Union Public License 1.2
4 stars 7 forks source link

Use database connection pool (DBCPConnectionPool) controller service in LDESClientProcessor #677

Open SBylemans opened 3 months ago

SBylemans commented 3 months ago

Is your feature request related to a problem? Please describe. Currently the LDESClientProcessor defines it's own entityManagers and handles those connections to the DB, using StatePersistenceFactory. This introduces a whole heap of problems:

  1. Transactions not being closed properly https://github.com/Informatievlaanderen/VSDS-Linked-Data-Interactions/issues/678
  2. Using all of the database connections
  3. Reusing a database connection of a processor with the same name, for which the connection parameters may not be the same. For https://github.com/Informatievlaanderen/VSDS-Linked-Data-Interactions/issues/676, I created a processor with the same name as another LDESClientProcessor. Trying to start that processor resulted in the same error as in the ticket while having other connection details.
  4. ...

Describe the solution you'd like The LDESClientProcessor should use the DBCPConnectionPool controller service of NiFi in order to obtain a database connection. Inspiration of use can be taken from the PutSQL processor of NiFi itself.

Additionally, to query and handle transactions and such, one could use additional libraries such as jOOQ which is more flexible than Hibernate.

jobulcke commented 1 month ago

This will be verified when #663 has been solved