search

voxel51 / fiftyone

Refine high-quality datasets and visual AI models
https://fiftyone.ai
Apache License 2.0
8.81k stars 557 forks source link

[INSTALL] The mongodb binaries aren't downloaded and installed on Archlinux #3975

Open jakubhejhal opened 9 months ago

jakubhejhal commented 9 months ago

System information

Commands to reproduce

$ fiftyone

Describe the problem

The mongodb database binaries which are supposed to be shipped with fiftyone by the fiftyone-db auxilliary package aren't downloaded and installed:

$> ls .venv/lib/python3.10/site-packages/fiftyone/db/
__init__.py  __pycache__/

Thus running fiftyone fails:

$> fiftyone
Traceback (most recent call last):
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/core/odm/database.py", line 200, in establish_db_conn
    _db_service = fos.DatabaseService()
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/core/service.py", line 80, in __init__
    self.start()
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/core/service.py", line 209, in start
    super().start()
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/core/service.py", line 118, in start
    + self.command,
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/core/service.py", line 238, in command
    DatabaseService.find_mongod(),
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/core/service.py", line 286, in find_mongod
    raise ServiceExecutableNotFound("Could not find `mongod`")
fiftyone.core.service.ServiceExecutableNotFound: Could not find `mongod`

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/test/.venv/bin/fiftyone", line 5, in <module>
    from fiftyone.core.cli import main
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/__init__.py", line 25, in <module>
    from fiftyone.__public__ import *
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/__public__.py", line 16, in <module>
    _foo.establish_db_conn(config)
  File "/home/user/test/.venv/lib/python3.10/site-packages/fiftyone/core/odm/database.py", line 206, in establish_db_conn
    raise FiftyOneConfigError(
fiftyone.core.config.FiftyOneConfigError: MongoDB could not be installed on your system. Please define a `database_uri` in your `fiftyone.core.config.FiftyOneConfig` to connect to yourown MongoDB instance or cluster

I looked into the fiftyone-db setup.py and it's failing here.

$> cat /etc/os-release

NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="38;2;23;147;209"
HOME_URL="https://archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
PRIVACY_POLICY_URL="https://terms.archlinux.org/docs/privacy-policy/"
LOGO=archlinux-logo

The ID=arch isn't in the LINUX_DOWNLOADS dict.

But the installation of fiftyone-db fails silently.

benjaminpkane commented 9 months ago

Hi @jakubhejhal, if you can share which download url should be used for arch I can add the entry and publish a patch. Though because Arch Linux is a rolling release distribution it's hard to guarantee support

jakubhejhal commented 9 months ago

I downloaded the latest ubuntu binaries and copied them to site-packages/fiftyone/db:

tar -xzvf mongodb-linux-x86_64-ubuntu2204-7.0.4.tgz
cp -r bin ~/.venv/lib/python3.10/site-packages/fiftyone/db/

And that seems to fixed the issue. But as you said, arch is rolling release and I have no idea if arch may decide to upgrade some underlying mongodb dependencies.

I tried to list the linked libraries that the ubuntu mongodb uses:

$> objdump -p .venv/lib/python3.10/site-packages/fiftyone/db/bin/mongod

...

Dynamic Section:
  NEEDED               libcurl.so.4
  NEEDED               liblzma.so.5
  NEEDED               libresolv.so.2
  NEEDED               libcrypto.so.3
  NEEDED               libssl.so.3
  NEEDED               libm.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6
  NEEDED               ld-linux-x86-64.so.2

Which all seem to be very widely used libs, so I guess nothing should break?

zadobudak commented 9 months ago

Had the same issue and ubuntu binaries are working for me. As another solution (I don't know the consequences of it) falling back to the system mongod binaries can be used by checking the if it is in the $PATH. I am willing to contribute if you think it is logical.

benjaminpkane commented 9 months ago

Hi @zadobudak I believe that would be useful! A contribution would be great! That would be a more reliable approach.

Pardon the delay @jakubhejhal. I think @zadobudak's is a bit more reliable. I think the fiftyone-db should only attempt installing officially supported downloads.

Generally speaking, we recommend configuring a database_uri for the best experience if FiftyOne is used frequently as it avoids subprocess management of the db.

mvsjober commented 9 months ago

I had a similar problem on Red Hat. The way you are parsing the os-release file with csv.reader doesn't work if there are empty lines in the file. The file on Red Hat 8 seems to have an empty line after BUG_REPORT_URL causing the parsing to fail.

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

This leads to it failing to download the mongodb binaries.

benjaminpkane commented 9 months ago

I had a similar problem on Red Hat. The way you are parsing the os-release file with csv.reader doesn't work if there are empty lines in the file. The file on Red Hat 8 seems to have an empty line after BUG_REPORT_URL causing the parsing to fail.

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"

This leads to it failing to download the mongodb binaries.

Thanks for reporting this! Fixed in fiftyone-db==1.1.1

zadobudak commented 8 months ago

@benjaminpkane hello I am a little late but I found 2 solutions for the problem and implement it will send it here for contribution however I have one question in my mind which one is a better way?

  1. core service.py uses find_mongod and directing to the system mongod binary is possible from here. Since if there is no mongod binary in specified path we can add a fallback search in system path.
  2. db package setup.py in here if we cannot find the downloads in os release info it passes silently. If it is not present in the specified packages we can symlink or copy the system binary into the directory which fiftyone.constants tells. At this point im not really sure about which one is a better solution. Since this is a problem of db package changing different parts of the code seems not logical however I feel like 1st approach is better for changing system binaries paths etc. What do you think?
benjaminpkane commented 8 months ago

Hi @zadobudak. 2. sounds like a hard solution to maintain, I agree. 1. sounds like a reasonable enhancement. A contribution would be welcomed! Just want to make sure this is a backward compatible change for users