adding elasticsearch as a data source

[amrnablus]

I'm building a project that uses elasticsearch as the main data source for some data, while jhipster only allows elasticsearch to be on top of an RDBMS, it would be nice to generate the elasticsearch entities directly using jhipster.

Below is the structure I have in mind:

1. Support data source per entity
   • The data sources become a checklist instead of a radio button; So, the project can talk to say both mongodb and mysql
   • Create a default data source for each project/microservice, this will be used for any new entity unless otherwise specified
   • When the entity gets created, the user will be asked if they need to select a different data source than the default, and select it
2. Add elasticsearch as a data source
   • This should be straightforward, the only concern I have is that we might have to allow the user to enter the data type manually as elasticsearch supports a very wide variety of types

I doubt this can be done as a module, I'm open for feedback and suggestions.

[jdubois]

It's a bad idea to use Elasticsearch as your main datasource, and I don't recommend you to do this, but that's another topic (you are free to do whatever you want, of course!). Then, I have several clients that have some Elasticsearch-only backends (this is not their main datasource, it's a "searchable copy" of their main datasource, which is usually a specific database or ERP), so I totally see your point here.

To implement this, there is another option: we already have a "no database" option, but it's only for microservices. So "no database + Elasticsearch" is typically what you would like to do.

But there is an issue for people building monoliths and gateways: we still need a database for Spring Security... So this would only work for microservices.

[pascalgrimaud]

I agree with Julien.

About module, it can be done, but it will be very complex... Same for our main generator-jhipster, it will add complexity and too much use cases to test, specially:

When the entity gets created, the user will be asked if they need to select a different data source than the default, and select it

So personnaly, I'm against this.

[atomfrede]

From my experience trying to have a search engine as primary datastore is not a good idea and makes a lot of problems, so I'm also against this.

[deepu105]

Yes its is not something that adds a lot of value in the generator. I'm against it as well. You would have to do this change manually. I'm using a similar setup where we have all of the data synced to ES from an external job and we read from it. But we have a posgres DB for user auth and stuff

[gzsombor]

Adding yet another database to support, in my opinion, probably not worth it. As several other already explained why. But I think, extending the current server generator, to be a bit more decoupled, could be a worthy goal. I mean, if someone would want to write a module, to support HBase/ElasticSearch/ArangoDB/RethinkDB/OrientDB/..., he could easily write it. In this way, we could even move the internal MongoDB/Cassandra support to a module. What do you think ? Or should I open a new ticket about this? :)

[deepu105]

@gzsombor Thats a good idea indeed. The more decoupled things are the easier for maintenance. That was my main goal when I decoupled the client and server into separate sub generator. It helped us to write NG2 and now React module. But the problem is its a lot of work and I'm not sure if any of us have time. But if you are willing to take the lead on that I'm all for it. Lets see what others think

[amrnablus]

Hello All and sorry for the late reply (my internet was disconnected, yay middle east!)

your input is appreciated, I do agree that normally elasticsearch shouldn't be a primary database, but i have a particular case where data gets written via a tool and it only gets read/searched by the end user (pretty much the same setup @deepu105 mentioned in his previous comment).

That said, maybe it's not worth having ES as a primary database after all since I'm also deleting all the write operations generated by jhipster, it also seems that no one is using it as a "Real?" primary data source so i'm gonna agree with everyone here.

What @gzsombor suggested though sounds very interesting, we could decouple the database generation and then someone (me) can write an elasticsearch module. In fact, this way i'd be able to write a plugin that would only generate a RS (Read and Search) methods.

I guess this ticket can be closed now and I will file a new one regarding the decoupling, @gzsombor if you're up for that i can lend a hand

[jdubois]

@amrnablus in that case you are doing like all of us, so in fact we are all agreeing here :-)

I'm still interested in have an "Elasticsearch without database" microservice, that looks like a good use case. But I don't think this is exactly what you want/need, and we just don't have the time to do this at the moment.

-> I'm closing this as we all agree, and as we don't have anyone who is going to code this, but if someone is interested, I'll be very happy to have an "Elasticsearch without database" microservice