search

fititnt / uwazi-docker

Dockerized version of Uwazi (“openness" in Swahili). HURIDOCS designed Uwazi to make human rights information more open and accessible to the defenders who need it.
The Unlicense
11 stars 4 forks source link

FileNotFoundError: [Errno 2] No such file or directory: '/home/jr/uwazi-docker/data/mongo/WiredTiger.turtle.set' #14

Closed vasyugan closed 5 years ago

vasyugan commented 5 years ago

After cloning the code,

docker-compose run -e IS_FIRST_RUN=true --rm uwazi

ends with the following error:

Status: Downloaded newer image for mongo:3.4
Creating uwazi-docker_mongo_1         ... done
Creating uwazi-docker_elasticsearch_1 ... done
Building uwazi
Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 71, in main
  File "compose/cli/main.py", line 127, in perform_command
  File "compose/cli/main.py", line 845, in run
  File "compose/cli/main.py", line 1297, in run_one_off_container
  File "compose/service.py", line 316, in create_container
  File "compose/service.py", line 352, in ensure_image_exists
  File "compose/service.py", line 1067, in build
  File "site-packages/docker/api/build.py", line 154, in build
  File "site-packages/docker/utils/build.py", line 31, in tar
  File "site-packages/docker/utils/build.py", line 79, in create_archive
  File "tarfile.py", line 1803, in gettarinfo
FileNotFoundError: [Errno 2] No such file or directory: '/home/jr/uwazi-docker/data/mongo/WiredTiger.turtle.set'
[4393] Failed to execute script docker-compose
jr@erwin:~/uwazi-docker$ ls data/mongo/WiredTiger.
WiredTiger.lock    WiredTiger.turtle  WiredTiger.wt
fititnt commented 5 years ago

Thanks for the report.

If I did not commit or anwser this in 48 hours, can ping me back with a new comment.

fititnt commented 5 years ago

Just to give a fast feedback for now:

Here a full log. The main difference is that I had to run docker-compose run -e IS_FIRST_RUN=true --rm uwazi twice (probably because the elastic search was not ready for connections fast enough).

Also the message "WARNING: Image for service uwazi was built because it did not already exist. To rebuild this image you must use docker-compose build or docker-compose up --build." maybe could means some additional difference.

# fititnt at bravo in ~/tmp [20:42:53]
$ git clone https://github.com/fititnt/uwazi-docker.git
Cloning into 'uwazi-docker'...
remote: Enumerating objects: 145, done.
remote: Total 145 (delta 0), reused 0 (delta 0), pack-reused 145
Receiving objects: 100% (145/145), 48.99 KiB | 0 bytes/s, done.
Resolving deltas: 100% (78/78), done.
Checking connectivity... done.

# fititnt at bravo in ~/tmp [20:43:43]
$ cd uwazi-docker 

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:43:47]
$ docker-compose run -e IS_FIRST_RUN=true --rm uwazi
Creating network "uwazi-docker_default" with the default driver
Creating uwazi-docker_elasticsearch_1 ... done
Creating uwazi-docker_mongo_1         ... done
Building uwazi
Step 1/9 : FROM node:6-slim
 ---> e7745d9e5093
Step 2/9 : LABEL maintainer="Emerson Rocha <rocha@ieee.org>"
 ---> Using cache
 ---> 8d315a326fe4
Step 3/9 : RUN DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y   bzip2   dh-autoreconf   git   libpng-dev
 ---> Using cache
 ---> 82ab5966ecb0
Step 4/9 : RUN git clone -b 1.1 --single-branch --depth=1 https://github.com/huridocs/uwazi.git /home/node/uwazi/   && chown node:node -R /home/node/uwazi/   && cd /home/node/uwazi/   && yarn install   && yarn production-build
 ---> Using cache
 ---> 964f21c84b91
Step 5/9 : COPY --chown=node:node ./scripts/patch/uwazi/database/reindex_elastic.js /home/node/uwazi/database/reindex_elastic.js
 ---> Using cache
 ---> 6a4614ebfbca
Step 6/9 : RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2930ADAE8CAF5059EE73BB4B58712A2291FA4AD5   && echo "deb http://repo.mongodb.org/apt/debian jessie/mongodb-org/3.6 main" | tee /etc/apt/sources.list.d/mongodb-org-3.6.list   && apt-get update   && apt-get install -y mongodb-org-shell mongodb-org-tools
 ---> Using cache
 ---> 492e62752d45
Step 7/9 : WORKDIR /home/node/uwazi/
 ---> Using cache
 ---> d4ad8f33205f
Step 8/9 : COPY --chown=node:node docker-entrypoint.sh /docker-entrypoint.sh
 ---> b943b7ba41b1
Step 9/9 : ENTRYPOINT ["/docker-entrypoint.sh"]
 ---> Running in ef77676901a3
Removing intermediate container ef77676901a3
 ---> 2b6495a358c8
Successfully built 2b6495a358c8
Successfully tagged uwazi-docker_uwazi:latest
WARNING: Image for service uwazi was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
uwazi-docker: IS_FIRST_RUN: true
uwazi-docker: Enviroment variable IS_FIRST_RUN is true. Assuming need to install database from blank state

uwazi-docker: Deleting mongo uwazi_development MongoDB database
MongoDB shell version v3.6.4
connecting to: mongodb://mongo:27017/uwazi_development
MongoDB server version: 3.4.14
WARNING: shell and server versions do not match
{ "ok" : 1 }

uwazi-docker: Importing /home/node/uwazi/blank_state/uwazi_development to mongo uwazi_development MongoDB database
2018-10-12T23:44:40.642+0000    the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2018-10-12T23:44:40.685+0000    building a list of collections to restore from /home/node/uwazi/blank_state/uwazi_development dir
2018-10-12T23:44:40.726+0000    reading metadata for uwazi_development.translations from /home/node/uwazi/blank_state/uwazi_development/translations.metadata.json
2018-10-12T23:44:40.731+0000    reading metadata for uwazi_development.users from /home/node/uwazi/blank_state/uwazi_development/users.metadata.json
2018-10-12T23:44:40.732+0000    reading metadata for uwazi_development.sessions from /home/node/uwazi/blank_state/uwazi_development/sessions.metadata.json
2018-10-12T23:44:40.732+0000    reading metadata for uwazi_development.settings from /home/node/uwazi/blank_state/uwazi_development/settings.metadata.json
2018-10-12T23:44:40.997+0000    restoring uwazi_development.translations from /home/node/uwazi/blank_state/uwazi_development/translations.bson
2018-10-12T23:44:41.270+0000    restoring uwazi_development.settings from /home/node/uwazi/blank_state/uwazi_development/settings.bson
2018-10-12T23:44:41.443+0000    restoring uwazi_development.sessions from /home/node/uwazi/blank_state/uwazi_development/sessions.bson
2018-10-12T23:44:41.632+0000    restoring uwazi_development.users from /home/node/uwazi/blank_state/uwazi_development/users.bson
2018-10-12T23:44:41.633+0000    restoring indexes for collection uwazi_development.sessions from metadata
2018-10-12T23:44:41.634+0000    no indexes to restore
2018-10-12T23:44:41.634+0000    finished restoring uwazi_development.settings (1 document)
2018-10-12T23:44:41.634+0000    no indexes to restore
2018-10-12T23:44:41.634+0000    finished restoring uwazi_development.translations (1 document)
2018-10-12T23:44:41.826+0000    finished restoring uwazi_development.sessions (1 document)
2018-10-12T23:44:41.826+0000    restoring indexes for collection uwazi_development.users from metadata
2018-10-12T23:44:41.852+0000    reading metadata for uwazi_development.connections from /home/node/uwazi/blank_state/uwazi_development/connections.metadata.json
2018-10-12T23:44:41.852+0000    reading metadata for uwazi_development.relationtypes from /home/node/uwazi/blank_state/uwazi_development/relationtypes.metadata.json
2018-10-12T23:44:41.852+0000    reading metadata for uwazi_development.templates from /home/node/uwazi/blank_state/uwazi_development/templates.metadata.json
2018-10-12T23:44:42.330+0000    restoring uwazi_development.templates from /home/node/uwazi/blank_state/uwazi_development/templates.bson
2018-10-12T23:44:42.330+0000    no indexes to restore
2018-10-12T23:44:42.330+0000    finished restoring uwazi_development.templates (0 documents)
2018-10-12T23:44:42.331+0000    reading metadata for uwazi_development.dictionaries from /home/node/uwazi/blank_state/uwazi_development/dictionaries.metadata.json
2018-10-12T23:44:42.604+0000    restoring uwazi_development.connections from /home/node/uwazi/blank_state/uwazi_development/connections.bson
2018-10-12T23:44:42.604+0000    restoring indexes for collection uwazi_development.connections from metadata
2018-10-12T23:44:42.892+0000    restoring uwazi_development.relationtypes from /home/node/uwazi/blank_state/uwazi_development/relationtypes.bson
2018-10-12T23:44:42.893+0000    no indexes to restore
2018-10-12T23:44:42.893+0000    finished restoring uwazi_development.relationtypes (0 documents)
2018-10-12T23:44:42.893+0000    reading metadata for uwazi_development.entities from /home/node/uwazi/blank_state/uwazi_development/entities.metadata.json
2018-10-12T23:44:43.213+0000    restoring uwazi_development.dictionaries from /home/node/uwazi/blank_state/uwazi_development/dictionaries.bson
2018-10-12T23:44:43.337+0000    no indexes to restore
2018-10-12T23:44:43.337+0000    finished restoring uwazi_development.dictionaries (0 documents)
2018-10-12T23:44:44.459+0000    finished restoring uwazi_development.users (2 documents)
2018-10-12T23:44:44.703+0000    restoring uwazi_development.entities from /home/node/uwazi/blank_state/uwazi_development/entities.bson
2018-10-12T23:44:44.703+0000    finished restoring uwazi_development.connections (0 documents)
2018-10-12T23:44:44.782+0000    restoring indexes for collection uwazi_development.entities from metadata
2018-10-12T23:44:45.439+0000    finished restoring uwazi_development.entities (0 documents)
2018-10-12T23:44:45.439+0000    done
uwazi-docker: Applyng yarn reindex. This will use data from MongoDB to feed Elastic Search
yarn run v1.5.1
$ node run.js ./database/reindex_elastic.js
Deleting index... uwazi_development
{ FetchError: request to http://elasticsearch:9200/uwazi_development failed, reason: connect ECONNREFUSED 172.20.0.3:9200
    at ClientRequest.<anonymous> (/home/node/uwazi/node_modules/node-fetch/index.js:133:11)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at Socket.socketErrorListener (_http_client.js:310:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1296:8)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
  name: 'FetchError',
  message: 'request to http://elasticsearch:9200/uwazi_development failed, reason: connect ECONNREFUSED 172.20.0.3:9200',
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED' }
Creating index... uwazi_development
{ FetchError: request to http://elasticsearch:9200/uwazi_development failed, reason: connect ECONNREFUSED 172.20.0.3:9200
    at ClientRequest.<anonymous> (/home/node/uwazi/node_modules/node-fetch/index.js:133:11)
    at emitOne (events.js:96:13)
    at ClientRequest.emit (events.js:188:7)
    at Socket.socketErrorListener (_http_client.js:310:9)
    at emitOne (events.js:96:13)
    at Socket.emit (events.js:188:7)
    at emitErrorNT (net.js:1296:8)
    at _combinedTickCallback (internal/process/next_tick.js:80:11)
    at process._tickCallback (internal/process/next_tick.js:104:9)
  name: 'FetchError',
  message: 'request to http://elasticsearch:9200/uwazi_development failed, reason: connect ECONNREFUSED 172.20.0.3:9200',
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED' }
Indexing documents and entities... - 0 indexed
Done, took 0.04 seconds
Done in 14.20s.
uwazi-docker: If no fatal errors occurred, you will not need to use this command again

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:45:05]
$ code .

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:47:16]
$ code .

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:47:57]
$ docker-compose run -e IS_FIRST_RUN=true --rm uwazi
Starting uwazi-docker_elasticsearch_1 ... done
Starting uwazi-docker_mongo_1         ... done
uwazi-docker: IS_FIRST_RUN: true
uwazi-docker: Enviroment variable IS_FIRST_RUN is true. Assuming need to install database from blank state

uwazi-docker: Deleting mongo uwazi_development MongoDB database
MongoDB shell version v3.6.4
connecting to: mongodb://mongo:27017/uwazi_development
MongoDB server version: 3.4.14
WARNING: shell and server versions do not match
{ "dropped" : "uwazi_development", "ok" : 1 }

uwazi-docker: Importing /home/node/uwazi/blank_state/uwazi_development to mongo uwazi_development MongoDB database
2018-10-12T23:48:07.376+0000    the --db and --collection args should only be used when restoring from a BSON file. Other uses are deprecated and will not exist in the future; use --nsInclude instead
2018-10-12T23:48:07.376+0000    building a list of collections to restore from /home/node/uwazi/blank_state/uwazi_development dir
2018-10-12T23:48:07.377+0000    reading metadata for uwazi_development.translations from /home/node/uwazi/blank_state/uwazi_development/translations.metadata.json
2018-10-12T23:48:07.551+0000    restoring uwazi_development.translations from /home/node/uwazi/blank_state/uwazi_development/translations.bson
2018-10-12T23:48:07.552+0000    reading metadata for uwazi_development.users from /home/node/uwazi/blank_state/uwazi_development/users.metadata.json
2018-10-12T23:48:07.552+0000    reading metadata for uwazi_development.sessions from /home/node/uwazi/blank_state/uwazi_development/sessions.metadata.json
2018-10-12T23:48:07.552+0000    reading metadata for uwazi_development.settings from /home/node/uwazi/blank_state/uwazi_development/settings.metadata.json
2018-10-12T23:48:07.553+0000    no indexes to restore
2018-10-12T23:48:07.553+0000    finished restoring uwazi_development.translations (1 document)
2018-10-12T23:48:07.553+0000    reading metadata for uwazi_development.connections from /home/node/uwazi/blank_state/uwazi_development/connections.metadata.json
2018-10-12T23:48:07.749+0000    restoring uwazi_development.users from /home/node/uwazi/blank_state/uwazi_development/users.bson
2018-10-12T23:48:08.111+0000    restoring uwazi_development.connections from /home/node/uwazi/blank_state/uwazi_development/connections.bson
2018-10-12T23:48:08.161+0000    restoring indexes for collection uwazi_development.connections from metadata
2018-10-12T23:48:08.358+0000    restoring uwazi_development.settings from /home/node/uwazi/blank_state/uwazi_development/settings.bson
2018-10-12T23:48:08.556+0000    restoring uwazi_development.sessions from /home/node/uwazi/blank_state/uwazi_development/sessions.bson
2018-10-12T23:48:08.583+0000    restoring indexes for collection uwazi_development.users from metadata
2018-10-12T23:48:09.058+0000    no indexes to restore
2018-10-12T23:48:09.058+0000    finished restoring uwazi_development.settings (1 document)
2018-10-12T23:48:09.058+0000    restoring indexes for collection uwazi_development.sessions from metadata
2018-10-12T23:48:09.059+0000    reading metadata for uwazi_development.relationtypes from /home/node/uwazi/blank_state/uwazi_development/relationtypes.metadata.json
2018-10-12T23:48:09.256+0000    finished restoring uwazi_development.sessions (1 document)
2018-10-12T23:48:09.256+0000    reading metadata for uwazi_development.templates from /home/node/uwazi/blank_state/uwazi_development/templates.metadata.json
2018-10-12T23:48:09.421+0000    restoring uwazi_development.relationtypes from /home/node/uwazi/blank_state/uwazi_development/relationtypes.bson
2018-10-12T23:48:09.421+0000    no indexes to restore
2018-10-12T23:48:09.421+0000    finished restoring uwazi_development.relationtypes (0 documents)
2018-10-12T23:48:09.421+0000    finished restoring uwazi_development.connections (0 documents)
2018-10-12T23:48:09.421+0000    reading metadata for uwazi_development.dictionaries from /home/node/uwazi/blank_state/uwazi_development/dictionaries.metadata.json
2018-10-12T23:48:09.421+0000    reading metadata for uwazi_development.entities from /home/node/uwazi/blank_state/uwazi_development/entities.metadata.json
2018-10-12T23:48:09.569+0000    finished restoring uwazi_development.users (2 documents)
2018-10-12T23:48:09.569+0000    restoring uwazi_development.templates from /home/node/uwazi/blank_state/uwazi_development/templates.bson
2018-10-12T23:48:09.570+0000    no indexes to restore
2018-10-12T23:48:09.570+0000    finished restoring uwazi_development.templates (0 documents)
2018-10-12T23:48:09.730+0000    restoring uwazi_development.dictionaries from /home/node/uwazi/blank_state/uwazi_development/dictionaries.bson
2018-10-12T23:48:09.730+0000    no indexes to restore
2018-10-12T23:48:09.730+0000    finished restoring uwazi_development.dictionaries (0 documents)
2018-10-12T23:48:09.904+0000    restoring uwazi_development.entities from /home/node/uwazi/blank_state/uwazi_development/entities.bson
2018-10-12T23:48:09.907+0000    restoring indexes for collection uwazi_development.entities from metadata
2018-10-12T23:48:10.195+0000    finished restoring uwazi_development.entities (0 documents)
2018-10-12T23:48:10.195+0000    done
uwazi-docker: Applyng yarn reindex. This will use data from MongoDB to feed Elastic Search
yarn run v1.5.1
$ node run.js ./database/reindex_elastic.js
Deleting index... uwazi_development
Creating index... uwazi_development
Indexing documents and entities... - 0 indexed
Done, took 1.776 seconds
Done in 6.04s.
uwazi-docker: If no fatal errors occurred, you will not need to use this command again

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:48:17]
$ docker-compose up -d uwazi
uwazi-docker_mongo_1 is up-to-date
uwazi-docker_elasticsearch_1 is up-to-date
Creating uwazi-docker_uwazi_1 ... done

# fititnt at bravo in ~/tmp/uwazi-docker on git:master o [20:49:06]
$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                              NAMES
940816c3853c        uwazi-docker_uwazi                                    "/docker-entrypoint.…"   4 minutes ago       Up 4 minutes        0.0.0.0:3000->3000/tcp             uwazi-docker_uwazi_1
8cefdf983acb        mongo:3.4                                             "docker-entrypoint.s…"   9 minutes ago       Up 9 minutes        0.0.0.0:27017->27017/tcp           uwazi-docker_mongo_1
88b856a30df6        docker.elastic.co/elasticsearch/elasticsearch:5.5.3   "elasticsearch -Expa…"   9 minutes ago       Up 9 minutes        0.0.0.0:9200->9200/tcp, 9300/tcp   uwazi-docker_elasticsearch_1

captura de tela de 2018-10-12 20-50-22

fititnt commented 5 years ago

Ahh, hey @vasyugan, maybe this specific issue is more simple, even if I cannot reproduce.

Look that at the creation of the script, instead of docker volumes, the uwazi-docker store the data of the mongo at a folder ./data the cloned repository, so is a very good possibility that it is just a filesystem permission issue.

Reference image:

captura de tela de 2018-10-12 21-14-49

If you could debug a bit more maybe I could document it if is not something very, very specific to where you are running uwazi-docker.

vasyugan commented 5 years ago

@fitint Thanks, but I am running the docker-compose command through sudo, so it has root permissions. Therefore it is difficult to see how this could hinge on permissions. Also, couldn't reproduce this issue, got other instead...

fititnt commented 5 years ago

Ah ok.

But just to mention, at least on my system (Docker version 18.03.1-ce, build 9ee9f40 / Ubuntu 16.04.4 LTS) and I guess in other Ubuntu 16 on a remote VPS, I was able to do the full testing with a non-root user that is allowed to manage docker (this depends of your setup), but since the docker itself runs as root, the files created by the docker on ./data folder are with root permission, so I only need root permission to full unninstall that clean up all data (see https://github.com/fititnt/uwazi-docker/blob/master/uninstall.md).

There are several ways to run docker containers, and was choosen to store the important data on this folder and not as docker volume to give a bit more chance for someone do not delete all data by accident and also because can be a bit more simple to make a backup and (if someone start with docker and later want to make full change to a production system) make a bit easier. That is why, by default, the data is not stored as docker volumes, but is possible to do it. I could pass links or document more on that.

vasyugan commented 5 years ago

well, for now this is how I got around it: rm -rf the data directory, added myself to the docker group and restarted the whole process as non-root. Not sure what got me into this mess in the first place, but I'd tend to close the issue for now.

fititnt commented 5 years ago

No problem. Docker is a nice tool to know. What you learn here will serve you for other projects. Also, I will help you a but more testing and with how to make pull requests, so you also lean how to contribute.

The most hard part tend to be the time to discover and test, so if you already know how to make pull requests, will make it easier. We can start with the issue #15.