generated ELK logstash container fails with error in windows when the project is not in users home folder

deepu105 commented 8 years ago

Overview of the issue

This is migrated from https://github.com/jhipster/generator-jhipster-docker-compose/issues/3

when doing docker-compose up -d the elk_logstash_1 container fails with below error

Error: No config files found: /config-dir/logstash.conf
Can you make sure this path is a logstash config file?
You may be interested in the '--configtest' flag which you can
use to validate logstash's configuration before you choose
to restart a running system.

JHipster Version(s)

Master

Browsers and Operating System

Windows 10

Reproduce the error

generate a new docker-compose setting with the docker compose sub generator and run the containers

deepu105 commented 8 years ago

as @pascalgrimaud suggested Im gonna try with placing the projects under the user folder

cc @PierreBesson @wmarques

deepu105 commented 8 years ago

@pascalgrimaud that works. So I moved the test apps into C:\Users\Deepu\jhipster-docker-compose-demo and logstash starts up without any errors. But this is not ideal IMO, we should either find a solution or tell windows users that this is a limitation in windows

When I went to http://192.168.99.100:5601/ to view the kibana dashboard its didnt load there was some error, but if I click on Dashboard link and then come back to the discover link it seems to work. @cbornet mentioned the same behavior as well.

Is there suppose to be any dashboard loaded by default? coz I dont see any. but it works when load them manually

Timelion tab also doesnt seem to work im getting a blank page. does it work for you guys

the pre configured visualizations look very good great job guys

pascalgrimaud commented 8 years ago

It's a limitation in Windows + Docker, because we use a mount volume here

jdubois commented 8 years ago

On Mac I have also an issue with a mounted volume, this time for the Elasticsearch container:

Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.data' (/usr/share/elasticsearch/data/elasticsearch)
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/elasticsearch

Is this related? I don't understand why we need those? If it's not related I'll create another ticket, of course.

pascalgrimaud commented 8 years ago

Under OSX, you have to put your project under /Users to be able to mount volume

If it doesn't work, try to comment these 2 lines: https://github.com/jhipster/generator-jhipster/blob/master/generators/docker-compose/templates/elk.yml#L8-L9

jdubois commented 8 years ago

Thanks Pascal I already tried it:

My dashboards were empty, so it looks like you need the volume
I don't understand what it's used for Le 19 mars 2016 8:27 PM, "Pascal Grimaud" notifications@github.com a écrit :

Under OSX, you have to put your project under /Users to be able to mount volume

If it doesn't work, try to comment these 2 lines:

https://github.com/jhipster/generator-jhipster/blob/master/generators/docker-compose/templates/elk.yml#L8-L9

— You are receiving this because you commented. Reply to this email directly or view it on GitHub https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-198774448

deepu105 commented 8 years ago

Its not elegent to have projects under the user folder especially in windows where it looks very strange :(

Hope we can find some workaround

pascalgrimaud commented 8 years ago

From official Docker doc

If you are using Docker Machine on Mac or Windows, your Docker daemon has only limited access to your OS X or Windows filesystem. Docker Machine tries to auto-share your /Users (OS X) or C:\Users (Windows) directory. So, you can mount files or directories on OS X using.

You don't have to put project under User folder, only the shared volume. So you have to change the path on your own docker-compose file, if your projects are in another folder. There are 2 differents uses of volume:

to persist data => _mysql.yml
configuration => elk.yml

I have to think about configuration, and how we can avoid or limit these uses.

deepu105 commented 8 years ago

cant we make the current project folder as the shared volume for that docker file or does it have to be global?

Thanks & Regards, Deepu

On Sun, Mar 20, 2016 at 3:11 PM, Pascal Grimaud notifications@github.com wrote:

From official Docker doc https://docs.docker.com/engine/userguide/containers/dockervolumes/

If you are using Docker Machine on Mac or Windows, your Docker daemon has only limited access to your OS X or Windows filesystem. Docker Machine tries to auto-share your /Users (OS X) or C:\Users (Windows) directory. So, you can mount files or directories on OS X using.

You don't have to put project under User folder, only the shared volume. So you have to change the path on your own docker-compose file, if your projects are in another folder. There are 2 differents uses of volume:

to persist data => _mysql.yml https://github.com/jhipster/generator-jhipster/blob/master/generators/server/templates/src/main/docker/_mysql.yml#L6-L7

configuration => elk.yml https://github.com/jhipster/generator-jhipster/blob/master/generators/docker-compose/templates/elk.yml#L12-L13

I have to think about configuration, and how we can avoid or limit these uses.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-198862523

pascalgrimaud commented 8 years ago

@deepu105 : I think you can, but you have to configure your VM. Maybe this link can help you (I didn't test it)

What I tried is (under Windows) 1) git clone jhipster-console into D:\jhipster\jhipster-console 2) create a folder at C:\Users\Ibu\projets\volume -> this folder will contain all my volumes 3) copy all necessary into this folder. For example copy D:\jhipster\jhipster-console\kibana\dashboards -> C:\Users\Ibu\projets\volume\kibana\dashboards 4) edit the docker-compose.yml file

    jhipster-console:
        build: kibana/
        ports:
            - "5601:5601"
        volumes:
            # - ./kibana/dashboards:/tmp/dashboards
            - /c/Users/Ibu/projets/volume/kibana/dashboards:/tmp/dashboards

The step 3) is the most important, if you decide to go this way

jdubois commented 8 years ago

Concerning my issue with Elasticsearch: yes the solution is to remove the "volume" for the elk-elasticsearch image. I don't think it causes any issue, excepted that your data is kept inside the image, which isn't an issue as far as I know. @PierreBesson is it good for you?

PierreBesson commented 8 years ago

@jdubois if the volume is causing some issues on windows & Mac os then you can comment it out. I just thought that it was really useful for preventing loss of data.

pascalgrimaud commented 8 years ago

@PierreBesson :

I think we can comment "volume" for data -> so the user can uncomment to store data
but currently, we can't comment "volume" for configuration, right?

PierreBesson commented 8 years ago

@pascalgrimaud, yes we need volumes for configuration as they are very convenient.

jdubois commented 8 years ago

OK, so we have volume issues only on Windows now (that's solved for me on Mac).

If that's a limitation on Windows, can we just document it?

deepu105 commented 8 years ago

You mean the default generated config works for you? Irrespective of where you have your project?

For me the default config works only if i keep my project in the users/username folder which would be the weirdest place to keep stuff in windows :P On 22 Mar 2016 07:04, "Julien Dubois" notifications@github.com wrote:

OK, so we have volume issues only on Windows now (that's solved for me on Mac).

If that's a limitation on Windows, can we just document it?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-199526676

jdubois commented 8 years ago

Sorry, I'm using my "user" folder, but I don't find that strange... I have Linux background and that's what I have always done. Can we just document it as a Docker requirement (or limitation!) and be done with it? Le 22 mars 2016 1:48 AM, "Deepu K Sasidharan" notifications@github.com a écrit :

You mean the default generated config works for you? Irrespective of where you have your project?

For me the default config works only if i keep my project in the users/username folder which would be the weirdest place to keep stuff in windows :P On 22 Mar 2016 07:04, "Julien Dubois" notifications@github.com wrote:

OK, so we have volume issues only on Windows now (that's solved for me on Mac).

If that's a limitation on Windows, can we just document it?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub < https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-199526676

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-199564149

deepu105 commented 8 years ago

Yes I know in linux/mac people are used to that, but in windows since there are multiple drives people rarely keep stuff under os drive and I have never seen anyone actually using the User folder at all. Anyway Im ok if its documented(may be even highlighted), anyway people deploying this using windows/mac should be very less, prod deployments will mostly be linux. mac/windows will be mostly just for dev

Thanks & Regards, Deepu

On Tue, Mar 22, 2016 at 4:01 PM, Julien Dubois notifications@github.com wrote:

Sorry, I'm using my "user" folder, but I don't find that strange... I have Linux background and that's what I have always done. Can we just document it as a Docker requirement (or limitation!) and be done with it? Le 22 mars 2016 1:48 AM, "Deepu K Sasidharan" notifications@github.com a écrit :

You mean the default generated config works for you? Irrespective of where you have your project?

For me the default config works only if i keep my project in the users/username folder which would be the weirdest place to keep stuff in windows :P On 22 Mar 2016 07:04, "Julien Dubois" notifications@github.com wrote:

OK, so we have volume issues only on Windows now (that's solved for me on Mac).

If that's a limitation on Windows, can we just document it?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub <

https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-199526676

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub < https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-199564149

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-199686879

jdubois commented 8 years ago

And we should push the use of the DevBox -> no problem with that if you are with the DevBox as it's running on Linux.

deepu105 commented 8 years ago

@pascalgrimaud could you document this or let me know on what is the correct alert to add?

pascalgrimaud commented 8 years ago

I already did a warning about volume in installation page. Maybe do the same for ELK/jhipster-console?

deepu105 commented 8 years ago

ok let me see

Thanks & Regards, Deepu

On Wed, Mar 23, 2016 at 1:10 PM, Pascal Grimaud notifications@github.com wrote:

I already did a warning about volume in installation page. Maybe do the same for ELK/jhipster-console?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/jhipster/generator-jhipster/issues/3221#issuecomment-200184813

deepu105 commented 8 years ago

via https://github.com/jhipster/jhipster.github.io/pull/231

jhipster / generator-jhipster