search

artefactual / atom

Open-source, web application for archival description and public access.
http://www.accesstomemory.org
GNU Affero General Public License v3.0
249 stars 127 forks source link

Elasticsearch crashing when following docker guide #1255

Open karatekaneen opened 3 years ago

karatekaneen commented 3 years ago

OS: debian-10-buster-v20210122 Version: 2.6.2

On a fresh VM where I only installed docker & docker-compose something is not working with elasticsearch when following this guide.

When running the sudo docker-compose -f docker/docker-compose.dev.yml exec atom php symfony tools:purge --demo there is an error where it cannot resolve some host. Full output below:

robin@atom:~/atom$ sudo docker-compose -f docker/docker-compose.dev.yml up -d --force-recreate
Building with native build. Learn about native build in Compose here: https://docs.docker.com/go/compose-native-build/
Creating network "docker_default" with the default driver
Creating docker_gearmand_1      ... done
Creating docker_memcached_1     ... done
Creating docker_nginx_1         ... done
Creating docker_atom_1          ... done
Creating docker_elasticsearch_1 ... done
Creating docker_percona_1       ... done
Creating docker_atom_worker_1   ... done
robin@atom:~/atom$ sudo docker-compose -f docker/docker-compose.dev.yml exec atom php symfony tools:purge --demo
>> schema    converting "/atom/src/config/schema.yml" to XML
>> schema    putting /atom/src/config/generated-schema.xml
>> schema    converting "/atom/src/plugins/qbAclPlugin/config/schema.yml" to XML
>> schema    putting /atom/src/plugins/qbAclPlugin/config/generated-qbAclPlugin-schema.xml
>> schema    converting "/atom/src/plugins/qtAccessionPlugin/config/schema.yml" to XML
>> schema    putting /atom/src/plugins/qtAccessionPlugin/config/generated-qtAccessionPlugin-schema.xml
>> file+     config/generated-qbAclPlugin-schema.xml
>> file-     /atom/src/plugins/qbAclPlugin/config/generated-qbAclPlugin-schema.xml
>> file+     config/generated-qtAccessionPlugin-schema.xml
>> file-     /atom/src/plugins/qtAccessionPlugin/config/generated-qtAccessionPlugin-schema.xml
>> propel    Running "insert-sql" phing task
>> file-     /atom/src/config/generated-schema.xml
>> file-     /atom/src/config/generated-qbAclPlugin-schema.xml
>> file-     /atom/src/config/generated-qtAccessionPlugin-schema.xml

  Couldn't resolve host                          

ERROR: 1
robin@atom:~/atom$ sudo docker-compose -f docker/docker-compose.dev.yml exec atom make -C plugins/arDominionPlugin
make: Entering directory '/atom/src/plugins/arDominionPlugin'
Running LESS compiler...
lessc --compress --relative-urls css/main.less > css/main.css
make: Leaving directory '/atom/src/plugins/arDominionPlugin'

And when checking what's running we can see that Elasticsearch has exited (it never even starts from what i can see from the logs):

robin@atom:~/atom$ sudo docker-compose -f docker/docker-compose.dev.yml ps
         Name                       Command               State                   Ports                
-------------------------------------------------------------------------------------------------------
docker_atom_1            /atom/src/docker/entrypoin ...   Up       9000/tcp                            
docker_atom_worker_1     /atom/src/docker/entrypoin ...   Up       9000/tcp                            
docker_elasticsearch_1   /bin/bash bin/es-docker          Exit 1                                       
docker_gearmand_1        docker-entrypoint.sh gearmand    Up       127.0.0.1:63005->4730/tcp           
docker_memcached_1       docker-entrypoint.sh -p 11 ...   Up       127.0.0.1:63004->11211/tcp          
docker_nginx_1           /docker-entrypoint.sh ngin ...   Up       0.0.0.0:63001->80/tcp               
docker_percona_1         /docker-entrypoint.sh mysqld     Up       127.0.0.1:63003->3306/tcp, 33060/tcp

Attaching full log output below.

Is there something I can do to get this working? I have no clue of where to start looking

atom_logs.txt

karatekaneen commented 3 years ago

Did a new test on an Ubuntu 18.04 VM and it does not encounter the same problems with elasticsearch crashing. But it still receives the error with Couldn't resolve host when populating the db.

jraddaoui commented 3 years ago

Hi @karatekaneen,

I'm not sure I understand your environment ... are you running Docker within a local VM or in a virtual server? We also provide Vagrant boxes that may suit better your needs if you can't run Docker locally:

https://app.vagrantup.com/artefactual/boxes/atom

Otherwise, I wonder if the error is now related to ES' virtual memory requirement, you may need to change it in the host, and in the VM if that's the issue ...

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/vm-max-map-count.html

Also, you may get more information about the starting error, restarting the ES container and checking its logs:

docker-compose restart elasticsearch
docker-compose logs -f elasticsearch
karatekaneen commented 3 years ago

@jraddaoui For this I created a virtual machine, a compute engine as Google calls it. It is a virtual machine that runs in the cloud that comes with nothing more than an operating system on which we installed docker, docker-compose etc.

This is usually a good way to make sure that there is no conflicts between already installed software, older/newer versions ans stuff like that.

I know I saw some error on the map count in some of the environments I experimented with which I think created an exit 78 for the elasticsearch container which I solved by, as you said, increasing the map count.

jraddaoui commented 3 years ago

I see, @karatekaneen,

So this instance is facing a different issue? Could you check the logs restarting the service? Maybe there is some conflict in the ES data volume, if you create a backup of the database (or don't care about it) you could fully recreate the environment, removing the volumes:

docker-compose down -v
docker-compose up -d
karatekaneen commented 3 years ago

I tried that as well as tearing down multiple instances and start from scratch only to face the same error. The logs that I attached in the first post gives the same errors as all following attempts, with our without new volumes

jraddaoui commented 3 years ago

Oh, I see, so are you running multiple AtoM Docker Compose environments in the same server? If that's the case, you may need to change the data volumes of those environments to avoid the collision in the Docker engine:

https://github.com/artefactual/atom/blob/qa/2.x/docker/docker-compose.dev.yml#L5-L7

Try using different names for those volumes on each environment.

karatekaneen commented 3 years ago

No, it was a dedicated server just for atom, nothing else at all. A clean slate so to say, just to avoid conflicts as those you mention.