search

dathere / datapusher-plus

A standalone web service that pushes data into the CKAN Datastore fast & reliably. It pushes real good!
GNU Affero General Public License v3.0
27 stars 21 forks source link

remove ckanserviceprovider dependency #66

Closed jqnatividad closed 1 year ago

jqnatividad commented 1 year ago

As per @amercader 's suggestion during the Feb 7 2023 tech team discussion, it's better to remove the dependency to simplify the architecture of Datapusher+, and make it more robust.

Perhaps, we can take another page from XLoader, and use its simpler queuing tech and get rid of the need to use a separate web server.

jqnatividad commented 1 year ago

Most of the value add of DP+ is concentrated in changes in jobs.py.

Instead of getting rid of ckanserviceprovider, it may be easier to port the changes to a fork of xloader instead.

cc @tino097 @TomeCirun

tino097 commented 1 year ago

@jqnatividad so you are suggesting to use xloader instaed of datapusher-plus ?

jqnatividad commented 1 year ago

I'm thinking forking xloader and adding the qsv-powered stuff in jobs.py might be easier than removing the ckanserviceprovider dependency from DP+ and copying the xloader queueing stuff into DP+.

It'd still be DP+, but forked from xloader...

jqnatividad commented 1 year ago

Nothing is that easy, but an initial inspection of xloader's jobs.py looks like a lot of the DP+ qsv stuff in its jobs.py can be ported over...

It will also make the task of making the Datastore Mgmt Interface easier, don't you think?

I'm also thinking of making it require Python3, so it will only support 2.9 and above...

jqnatividad commented 1 year ago

Closing this as DP+ 1.x will work as a CKAN extension and will no longer need ckanserviceprovider.