[BUG] database not found

HeloiseS commented 2 months ago

Main Problem

I have followed the install instructions (I am on Debian 12) and when running:

sudo docker run -it -p 8080:8080 --network otter-net astro-otter

i get the following error:

sudo docker run -it -p 8080:8080 --network otter-net astro-otter
Traceback (most recent call last):
  File "/home/default-user/otter-web/start.py", line 2, in <module>
    from otter_web.client import *
  File "/usr/local/lib/python3.10/dist-packages/otter_web/client/__init__.py", line 1, in <module>
    from .home import *
  File "/usr/local/lib/python3.10/dist-packages/otter_web/client/home.py", line 21, in <module>
    db = Otter(url=API_URL)
  File "/usr/local/lib/python3.10/dist-packages/otter/io/otter.py", line 77, in __init__
    super().__init__(connection, "otter", **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pyArango/database.py", line 32, in __init__
    self.reload()
  File "/usr/local/lib/python3.10/dist-packages/pyArango/database.py", line 96, in reload
    self.reloadCollections()
  File "/usr/local/lib/python3.10/dist-packages/pyArango/database.py", line 78, in reloadCollections
    raise UpdateError(data["errorMessage"], data)
pyArango.theExceptions.UpdateError: database not found. Errors: {'error': True, 'errorNum': 1228, 'errorMessage': 'database not found', 'code': 404}

Installation steps followed and trace

I followed all instructions in the read me - to help track the issue I am pasting here the commands from my terminal history and the messages I received:

First I did

sudo docker network create otter-net

returning

0efefeb56cd6b32d786a43b8846745d35dd9d66f6a821a8360d53285383a5569

then I moved to a subdirectory called /software and ran:

git clone https://github.com/astro-otter/otterdb.git
cd otterdb
sudo ./build.sh

which gave:

WARNING! Removing the otterdb-persist and otterdb containers if they already exist!
Unable to find image 'arangodb:latest' locally
latest: Pulling from library/arangodb
a0f4cbe03b6d: Pull complete 
49a1f452304f: Pull complete 
6cf357aa9fa4: Pull complete 
8cbf4ac60298: Pull complete 
Digest: sha256:86900a367435d28e68c5ea023f074887f446a2a0c31d4eae3d7812ef9bc4f6aa
Status: Downloaded newer image for arangodb:latest
bd404b4f53ae872077d8d15ca1bdde22a7fc64bb04a69db2fc4a0c715d1ea158
6148ec22b6a22dfcde26682258c8d352f13e0a0b89b61ff113728777ad174ea7
arango server is now running in the background
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

The not on the externally managed environment I assumed was because I ran this in my (base) anaconda environment which is where I operate by default to not mess with my system python on accident. I assumed this was not an issue but that may be incorrect. So i proceeded with the next set of instructions

sudo docker build https://github.com/astro-otter/otter-docker.git#main -t astro-otter

which gave:

[+] Building 654.4s (21/21) FINISHED                          docker:default
 => [internal] load git source https://github.com/astro-otter/otter-d  19.1s
 => resolve image config for docker.io/docker/dockerfile:1              2.4s
 => docker-image://docker.io/docker/dockerfile:1@sha256:865e5dd094beca  7.9s 
 => => resolve docker.io/docker/dockerfile:1@sha256:865e5dd094beca432e  0.0s 
 => => sha256:865e5dd094beca432e8c0a1d5e1c465db5f998dc 8.40kB / 8.40kB  0.0s 
 => => sha256:e3171ba32039ebf6fef09a7cbe2578e43a0ca25e087c 850B / 850B  0.0s 
 => => sha256:a1637aae1ee9435ad8372f51bdf6f05ed49e4a3b 1.26kB / 1.26kB  0.0s 
 => => sha256:1e45ed8b8be3fcf5baec105c530196be8d0b85 12.49MB / 12.49MB  7.5s
 => => extracting sha256:1e45ed8b8be3fcf5baec105c530196be8d0b853893e20  0.2s
 => [internal] load metadata for docker.io/library/ubuntu:22.04         1.5s
 => [ 1/16] FROM docker.io/library/ubuntu:22.04@sha256:58b87898e82351  20.9s
 => => resolve docker.io/library/ubuntu:22.04@sha256:58b87898e82351c6c  0.0s
 => => sha256:58b87898e82351c6cf9cf5b9f3c20257bb9e2dcf 1.34kB / 1.34kB  0.0s
 => => sha256:3d1556a8a18cf5307b121e0a98e93f1ddf1f3f8e092f 424B / 424B  0.0s
 => => sha256:97271d29cb7956f0908cfb1449610a2cd9cb46b0 2.30kB / 2.30kB  0.0s
 => => sha256:6414378b647780fee8fd903ddb9541d134a19 29.54MB / 29.54MB  19.7s
 => => extracting sha256:6414378b647780fee8fd903ddb9541d134a1947ce092d  1.1s
 => [ 2/16] RUN mkdir /home/default-user/                               0.3s
 => [ 3/16] RUN mkdir /home/default-user/astro-otter                    0.4s
 => [ 4/16] RUN mkdir /home/default-user/otterdb                        0.4s
 => [ 5/16] ADD otter /home/default-user/astro-otter                    0.1s
 => [ 6/16] ADD examples /home/default-user/                            0.1s
 => [ 7/16] ADD otterdb /home/default-user/otterdb                      0.1s
 => [ 8/16] RUN apt-get update && apt-get install -y     python3      145.3s
 => [ 9/16] RUN pip install --upgrade pip                               1.9s
 => [10/16] RUN pip install --upgrade setuptools setuptools_scm wheel   3.1s
 => [11/16] WORKDIR /home/default-user/astro-otter/                     0.1s
 => [12/16] RUN pip install --no-cache-dir $(pwd)                     417.1s
 => [13/16] WORKDIR /home/default-user/                                 0.2s
 => [14/16] RUN git clone -b docker https://github.com/astro-otter/ott  3.9s
 => [15/16] WORKDIR /home/default-user/otter-web                        0.1s
 => [16/16] RUN pip install --no-cache-dir $(pwd)                      15.1s
 => exporting to image                                                 13.8s
 => => exporting layers                                                13.8s
 => => writing image sha256:a8bc9db3773f4efc406b6cdd75145e96f089233a00  0.0s
 => => naming to docker.io/library/astro-otter                          0.0s

 1 warning found (use docker --debug to expand):
 - JSONArgsRecommended: JSON arguments recommended for CMD to prevent unintended behavior related to OS signals (line 37)

which looked fine so I then finally:

sudo docker run -it -p 8080:8080 --network otter-net astro-otter

which gave the error mentioned at the start

noahfranz13 commented 2 months ago

I think there are a few things that could be going on here. Could you start by seeing if the otterdb docker container is created and running by pasting the outputs of docker ps, docker image list, and docker container list here?

I'm guessing it is not but the website docker container seems to have been created successfully (this is good as that is the step that seemed to break the most for a while!). This would explain why the website can't seem to find it. My suspicion is that the command python3 -m pip install pyArango in the ./build.sh step. I found a fix here that I pushed a commit implementing it so hopefully that works if you try to reinstall things too!

kdalexander commented 2 months ago

I just tried reinstalling everything and I run into the same error message at the same step. I'm running macOS Ventura 13.6.1. Here are my outputs for the troubleshooting:

bash-3.2$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 891ef4830c1a arangodb "/entrypoint.sh aranâ¦" About an hour ago Up About an hour 0.0.0.0:8529->8529/tcp otterdb

bash-3.2$ docker image list REPOSITORY TAG IMAGE ID CREATED SIZE astro-otter latest 3c139d8e7520 51 minutes ago 3.71GB arangodb latest 1b065f7e6170 3 weeks ago 659MB

6a3cffb89c3c 2 months ago 3.8GB 47f26f6090f0 2 months ago 3.8GB odahub/welcome-saasfee2022 latest d32fba6a68ea 2 years ago 10MB odahub/saasfee22-offline-bootstrap latest 6678253c6606 2 years ago 179MB bash-3.2$ docker container list CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 891ef4830c1a arangodb "/entrypoint.sh aranâ¦" About an hour ago Up About an hour 0.0.0.0:8529->8529/tcp otterdb

noahfranz13 commented 2 months ago

@HeloiseS, I think everything should be fixed now with the installation and installation instructions in the README! @kdalexander tested the new instructions and was able to get it installed so I would suggest deleting the old partially created docker containers and then starting from new.

Let me know how things go so I can fix any other problems that occur! I think, in the end, what was happening was that the build script for the database backend was crashing before creating the database and importing the data which is why the website was never able to open and create the home page.

HeloiseS commented 2 months ago

Hello!

Okay I gave it a go and now I run into different errors.

Here's what I did: I pulled the repo to have the newest version, then I ran the requirements then tried to do the build.sh That didn't work because it said otterdb-persist was alreayd in use.

1) First attempt: I ignored that and proceeded with the rest of the install. It ran but then when I tried to run the container it said it was it couldn't find the port and establish a connection because of arango not running

==
Unable to establish connection, perhaps arango is not running.
===
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f2c13f38e80>: Failed to establish a new connection: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 830, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 830, in urlopen
    return self.urlopen(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 830, in urlopen
    return self.urlopen(
  [Previous line repeated 2 more times]
  File "/usr/local/lib/python3.10/dist-packages/urllib3/connectionpool.py", line 802, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.10/dist-packages/urllib3/util/retry.py", line 594, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='otterdb', port=8529): Max retries exceeded with url: /_api/user/root/database (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2c13f38e80>: Failed to establish a new connection: [Errno -2] Name or service not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/default-user/otter-web/start.py", line 7, in <module>
    from otter_web.client import *
  File "/usr/local/lib/python3.10/dist-packages/otter_web/client/__init__.py", line 1, in <module>
    from .home import *
  File "/usr/local/lib/python3.10/dist-packages/otter_web/client/home.py", line 21, in <module>
    db = Otter(url=API_URL)
  File "/usr/local/lib/python3.10/dist-packages/otter/io/otter.py", line 76, in __init__
    connection = Connection(username=username, password=password, arangoURL=url)
  File "/usr/local/lib/python3.10/dist-packages/pyArango/connection.py", line 259, in __init__
    self.reload()
  File "/usr/local/lib/python3.10/dist-packages/pyArango/connection.py", line 359, in reload
    r = self.session.get(self.getDatabasesURL())
  File "/usr/local/lib/python3.10/dist-packages/pyArango/connection.py", line 60, in __call__
    ret = self.fct(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/requests/adapters.py", line 700, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='otterdb', port=8529): Max retries exceeded with url: /_api/user/root/database (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f2c13f38e80>: Failed to establish a new connection: [Errno -2] Name or service not known'))

Then I figured maybe ignoring the build was a bad idea so I tried removing otterdb and otter-persist and run the build again (I'll note that accoridng to your build message I shouldn'tt have to do the deletion myself as your script gives me a warning that it will remove the containiers if they already exist, but clearly docker has a mind of its own and won't even let me run your code). Anyway here is what I get:


WARNING! Removing the otterdb-persist and otterdb containers if they already exist!
2b938bb3548dbd7e7c79aa3926fb9294ce56196adf02c46fa66f1b9018083f9a
cc2ea77a49e464453cf00083ee08f7ffec77ebd72cae83e3244fa2e2c7196447
arango server is now running in the background
Traceback (most recent call last):
File "/home/stevance/software/otterdb/import_jsons.py", line 8, in <module>
from pyArango.connection import *
ModuleNotFoundError: No module named 'pyArango'



Problem is pyArango exists - at least in my env - as I can import it in my python interpreter.

noahfranz13 commented 2 months ago

Sometimes it does seem like docker has a mind of its own :). You are correct that the message in the build file is incorrect. The first thing you should probably do is run

docker container rm -f otterdb
docker container rm -f otterdb-persist
docker container rm -f astro-otter

It's okay if some of those don't work, the container may just not exist. The next thing is to remove any leftover, lingering docker images that may be causing problems. To do this run docker image list and then run docker image rm -f IMAGEID where you replace IMAGEID with any image id's that you think need to be removed. Finally, please run

docker system prune

which will remove any images and containers that are not running that docker may use to try to build the new images faster (I've found without this command it causes problems sometimes).

You can then try following the instructions in the README again, although I suspect this will not solve the ModuleNotFoundError that you are getting. So a few questions/things to check:

Did you run python3 -m pip install -r requirements.txt from the otterdb repo? This command was new to the instructions to try to fix some other issues. And, did it throw any errors or not succeed?
Does the output of python3 -m pip list | grep pyArango say pyArango 2.0.2?
You said you are using a conda environment so if you have that environment activated and run conda list | grep pyArango does it show up? If not it might be installed in your base environment but not in the conda environment. Or, if it shows up in your conda environment but not in step 2 maybe make sure your conda environment is activated when you run the build script.

Sorry this is all so complicated and annoying and thank you so much for testing it all out for me! In the future, once we get it hosted somewhere, no one should ever have to deal with installing the backend and website dockers locally :)

HeloiseS commented 2 months ago

okay i tried running the build again and it complaine otter-net wasn';t found so i ran the first command again but now otter-persists exists again???

(base) stevance::aslxlap01 { ~/software/otterdb }-> sudo ./build.sh
WARNING! Removing the otterdb-persist and otterdb containers if they already exist!
964786fbd976a4f6102d1ed86f6b8569d25962a86a765e9d5822064fef851cf0
dc7ca5c365bc97ebc2ffc3ad11c36660b0038ca39da27430a354a54f11edbcf0
docker: Error response from daemon: network otter-net not found.
(base) stevance::aslxlap01 { ~/software/otterdb }-> sudo docker network create otter-net
6bf7929933555ee31e7a6586de27918c658243b41d424e73ed8e022098a64433
(base) stevance::aslxlap01 { ~/software/otterdb }-> sudo ./build.sh
WARNING! Removing the otterdb-persist and otterdb containers if they already exist!
Error response from daemon: Conflict. The container name "/otterdb-persist" is already in use by container "964786fbd976a4f6102d1ed86f6b8569d25962a86a765e9d5822064fef851cf0". You have to remove (or rename) that container to be able to reuse that name.

except it doesn't because when I do docker image list all I have is:

(base) stevance::aslxlap01 { ~/software/otterdb }-> sudo docker image list
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
arangodb     latest    1b065f7e6170   3 weeks ago   659MB

also about the hosting we can help - me and a couple of research engineers are building a Transient Arxiv (Tarxiv) to centralise all SNe, TDEs, CVs (anything that goes boom). We need it for astro-colibri but also I want to be able to query by lightcurve behaviour, not have to remember phone numbers and check 4 different catalogues. We're working on the database system and the cron jobs that will query TNS and the individual teslescopoe APIs so we update lighturves and data automatically without people having to submit. We're also fleshing out how science contributors get credit (e.g. through individual DOIs etc) - if you have ideas of how you'd want your work to be cited (or things you'd want to see implemented) let me know!

astro-otter / otter-docker

[BUG] database not found #1

Main Problem

Installation steps followed and trace