search

powa-team / powa-archivist

powa-archivist: the powa PostgreSQL extension
http://powa.readthedocs.io/
PostgreSQL License
51 stars 20 forks source link

Add capture functions #13

Closed frost242 closed 5 years ago

frost242 commented 5 years ago

Add data capture functions to help implement a remote mode to PoWA.

All new functions are named after get_MODULE_capture_data() and give back the data used in the corresponding snapshot functions.

The functions are also registered in a new column named query_source from the powa_functions table. This way, a remote worker knows which function to call to get all data.

Per idea from Julien Rouhaud.

rjuju commented 5 years ago

@frost242 I looked again at this PR. I fixed some minor issues and used it as a basis for implementing a remote mode. You can find everything in the branch v4 (https://github.com/powa-team/powa-archivist/commits/v4).

FTR I kept the function written in plpgsql, and moved the sql one to plpgsql too, since the further commit made the query_src functions need to do more than a simple query. I also made this branch v4.0.0, as it's a huge change.

If you want to try v4, you can use the UI available in the remote_worker branch of powa-web (https://github.com/powa-team/powa-web/commits/remote_worker). The UI and new daemon for the remote snapshot will need some cleanup, but it should already be entirely functional. The new server API in powa-archivist will also probably require more thought, but what I added should be enough to test it.

rjuju commented 5 years ago

@marco44 I agree, but sql functions are not an option anymore for the remote-mode, see https://github.com/powa-team/powa-archivist/blob/v4/powa--4.0.0dev.sql#L1525 for a simple example.

This PR is outdated anyway, so @frost242 if you're ok I think we should close it and have review on the v4 branch, since I had to fix many things for the remote mode. There's a bunch of documentation changes that are needed (especially for 3rd part extension integration), but I'd prefer to do that after making sure that we have a design we all agree about.

frost242 commented 5 years ago

@rjuju: I'm OK to close thise PR. It's already obsolete and I'm glad that you found some time to work further.