search

EGA-archive / LocalEGA

A federated storage for sensitive data
http://localega.readthedocs.io
Apache License 2.0
7 stars 16 forks source link

RabbitMQ fails upon start #131

Closed scimerman closed 1 year ago

scimerman commented 1 year ago

Current Behavior

I have tested the LocalEGA on our system. Unfortunately following a Quick install steps did not end up with a running environment. The problem is that the MQ instances are getting killed soon after startup.

Steps to Reproduce

$ git clone https://github.com/EGA-archive/LocalEGA.git $ cd LocalEGA/deploy $ make -C bootstrap $ make -j 4 images $ make up $ make ps => error

Additional information / debug

$ make up
[+] Running 13/13
 ⠿ Container save2db             Started             6.6s
 ⠿ Container backup2             Started             6.6s
 ⠿ Container dispatcher          Started             3.2s
 ⠿ Container mq                  Started             3.8s
 ⠿ Container ingest              Started             3.8s
 ⠿ Container inbox               Running             0.0s
 ⠿ Container archive-db          Running             0.0s
 ⠿ Container cega-accession      Started             2.4s
 ⠿ Container backup1             Started             3.8s
 ⠿ Container cega-users          Running             0.0s
 ⠿ Container db                  Running             0.0s
 ⠿ Container cleanup             Started             4.0s
 ⠿ Container cega-mq             Started             3.8s
$  make ps
NAME                COMMAND                  SERVICE             STATUS              PORTS
archive-db          "/usr/local/bin/entr…"   archive-db          running             0.0.0.0:15432->5432/tcp, :::15432->5432/tcp
backup1             "lega-entrypoint.sh …"   backup1             running             
backup2             "lega-entrypoint.sh …"   backup2             running             
cega-accession      "python accession.py"    cega-accession      running             
cega-mq             "/usr/local/bin/cega…"   cega-mq             exited (0)          <<<< !!
cega-users          "python /cega/server…"   cega-users          running             
cleanup             "lega-entrypoint.sh …"   cleanup             running             
db                  "/usr/local/bin/entr…"   db                  running             5432/tcp
dispatcher          "lega-entrypoint.sh …"   dispatcher          running             
inbox               "entrypoint.sh"          inbox               running             0.0.0.0:2222->9000/tcp, :::2222->9000/tcp
ingest              "lega-entrypoint.sh …"   ingest              running             
mq                  "/usr/local/bin/ega-…"   mq                  exited (0)          <<<< !!
save2db             "lega-entrypoint.sh …"   save2db             running             
$

Trying to restart these two services

NAME                COMMAND                  SERVICE             STATUS              PORTS

$ docker-compose up -d mq cega-mq
[+] Running 2/2
 ⠿ Container mq       Started                                                                                                                                             1.2s
 ⠿ Container cega-mq  Started                                                                                                                                             1.0s
$ docker-compose ps
NAME                COMMAND                  SERVICE             STATUS              PORTS
archive-db          "/usr/local/bin/entr…"   archive-db          running             0.0.0.0:15432->5432/tcp, :::15432->5432/tcp
backup1             "lega-entrypoint.sh …"   backup1             running             
backup2             "lega-entrypoint.sh …"   backup2             running             
cega-accession      "python accession.py"    cega-accession      running             
cega-mq             "/usr/local/bin/cega…"   cega-mq             running             4369/tcp, 5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:5670->5671/tcp, :::5670->5671/tcp, 0.0.0.0:15670->15672/tcp, :::15670->15672/tcp
cega-users          "python /cega/server…"   cega-users          running             
cleanup             "lega-entrypoint.sh …"   cleanup             running             
db                  "/usr/local/bin/entr…"   db                  running             5432/tcp
dispatcher          "lega-entrypoint.sh …"   dispatcher          running             
inbox               "entrypoint.sh"          inbox               running             0.0.0.0:2222->9000/tcp, :::2222->9000/tcp
ingest              "lega-entrypoint.sh …"   ingest              running             
mq                  "/usr/local/bin/ega-…"   mq                  running             4369/tcp, 5671-5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp, :::15672->15672/tcp
save2db             "lega-entrypoint.sh …"   save2db             running       
$

^ all started. After a few seconds, they are again failed, with little to nothing written in the logs

$ docker-compose ps
NAME                COMMAND                  SERVICE             STATUS              PORTS
archive-db          "/usr/local/bin/entr…"   archive-db          running             0.0.0.0:15432->5432/tcp, :::15432->5432/tcp
backup1             "lega-entrypoint.sh …"   backup1             running             
backup2             "lega-entrypoint.sh …"   backup2             running             
cega-accession      "python accession.py"    cega-accession      running             
cega-mq             "/usr/local/bin/cega…"   cega-mq             exited (0)          <<<< !!
cega-users          "python /cega/server…"   cega-users          running             
cleanup             "lega-entrypoint.sh …"   cleanup             running             
db                  "/usr/local/bin/entr…"   db                  running             5432/tcp
dispatcher          "lega-entrypoint.sh …"   dispatcher          running             
inbox               "entrypoint.sh"          inbox               running             0.0.0.0:2222->9000/tcp, :::2222->9000/tcp
ingest              "lega-entrypoint.sh …"   ingest              running             
mq                  "/usr/local/bin/ega-…"   mq                  exited (137)           <<<< !!
save2db             "lega-entrypoint.sh …"   save2db             running             
$ docker-compose logs cega-mq
cega-mq  | Killed
cega-mq  | Killed
$ docker-compose logs mq
mq  | Killed
$

The hosting system

scimerman commented 1 year ago

It is resolved. The issue was the hosting machine consumed entire available memory and crashed. Had just redeployed again on the hosting server with more available resources.

Would be helpful to have listed somewhere system's hardware requirements.