Doesn't work with Elasticsearch v6 without a tweak

[murphyke]

For posterity, I was able to get this to "work" with PostgreSQL 10 and Elasticsearch 6 by pinning the version of the elasticsearch package dependency (in setup.py, and requirements.txt for consistency). Without pinning the version, elasticsearch v7 is installed, which only works with Elasticsearch 7+.

The Dockerfile I used:

FROM postgres:10-alpine

RUN apk add --no-cache \
        python2 \
        python2-dev \
        py2-pip

RUN apk add --no-cache build-base

RUN pip install pgxnclient
RUN pgxn install multicorn

RUN pip install pg_es_fdw

ENTRYPOINT ["docker-entrypoint.sh"]

EXPOSE 5432
CMD ["postgres"]

I haven't run the full tests yet, but simple insert, delete, and select are working for me.

The documentation is pretty good, but I think it could be tweaked to make it even more clear up-front that quite likely you will have to accommodate your particular version Elasticsearch in addition to PostgreSQL and the most recent Multicorn that goes with your version of PostgreSQL.

Along with this, it would be better if the repo was configured to work with a particular set of versions, and those versions were clearly spelled out.

BTW, the current version of Multicorn (1.3.5 from 2018) is compatible with PostgreSQL 10 but not PostgreSQL 11.

[matthewfranglen]

Hi,

Thanks for writing up your experiences. I'm on holiday at the moment and will address your recommendations when I return.

On 13 August 2019 22:44:29 CEST, Kevin Murphy notifications@github.com wrote:

For posterity, I was able to get this to "work" with PostgreSQL 10 and Elasticsearch 6 by pinning the version of the elasticsearch package dependency (in setup.py, and requirements.txt for consistency). Without pinning the version, elasticsearch v7 is installed, which only works with Elasticsearch 7+.

The Dockerfile I used:

FROM postgres:10-alpine

RUN apk add --no-cache \
       python2 \
       python2-dev \
       py2-pip

RUN apk add --no-cache build-base

RUN pip install pgxnclient
RUN pgxn install multicorn

RUN pip install pg_es_fdw

ENTRYPOINT ["docker-entrypoint.sh"]

EXPOSE 5432
CMD ["postgres"]

I haven't run the full tests yet, but simple insert, delete, and select are working for me.

The documentation is pretty good, but I think it could be tweaked to make it even more clear up-front that quite likely you will have to accommodate your particular version Elasticsearch in addition to PostgreSQL and the most recent Multicorn that goes with your version of PostgreSQL.

Along with this, it would be better if the repo was configured to work with a particular set of versions, and those versions were clearly spelled out.

BTW, the current version of Multicorn (1.3.5 from 2018) is compatible with PostgreSQL 10 but not PostgreSQL 11.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/matthewfranglen/postgres-elasticsearch-fdw/issues/3

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[matthewfranglen]

I've worked on this a bit tonight.

• I've reproduced the issue with the Elasticsearch dependency.
• I was able to get this to work with Postgres 11.

I think that the version of multicorn installed by postgresql-11-python-multicorn has been updated to work with Postgres 11. I'm not sure where you got the 1.3.5 version number from though, as the debian package lists 1.3.4, which is the last release from 2017.

I'm going to look into improving the test setup as it is quite manual.

[murphyke]

@matthewfranglen Hey, thanks for your work on this. The Multicorn version I got from pgxn.org, in particular https://pgxn.org/dist/multicorn/. I admit it is strange since the GitHub repo only goes up to 1.3.4.

[matthewfranglen]

That's interesting! When I look at the github page I can see that there is a badge for 1.3.5 as well. Perhaps they are just not using tags to track that information anymore.

I have worked on this a bit more and the tests now run against different versions of Elastic Search (5, 6 and 7). I've documented this in the README near the top, with instructions to install the elasticsearch client before.

I've released a new version as the client API changes slightly for Elastic Search 7.

If you have any further suggestions please let me know, otherwise I feel this issue can be closed.

[murphyke]

If you have any further suggestions please let me know, otherwise I feel this issue can be closed.

Fantastic; I agree it can be closed now. These little improvements will have a big impact on the usability of this extension. There is a PG wiki entry on FDWs that points to a different and "archived" Elasticsearch FDW repo on GitHub. I would like to change it tomorrow to point to yours.

[matthewfranglen]

Sounds great! Thanks for opening this ticket :smile_cat:

[murphyke]

I would like to change [the PG wiki Elasticsearch FDW entry]

Done. https://wiki.postgresql.org/wiki/Foreign_data_wrappers#Big_Data_Wrappers