Installation and dependencies pitfalls

[iannesbitt]

Some issues I've run into when trying to replicate the mnlite service onto its own user account at mnlite@gmn2. Several overlapping dependency issues must be thought through and resolved for this package to become a plug-and-play installation candidate that we can easily reproduce across platforms/services.

1. poetry is required for dependency maintenance, but the current version (poetry==1.4.0) does not accept the formatting of poetry.lock/pyproject.toml dependency files created by this version (poetry==1.1.12)
2. Many of the dependencies are out of date and perhaps conflicting between sonormal and mnlite, but I'm not sure which ones should be pinned due to breaking changes and which can be bumped. Is a simple poetry update sufficient or are there other considerations?
3. We need to create an installation guide as the process is not obvious to me as someone with limited poetry experience.

[iannesbitt]

@datadavev can you add me as an editor on sonormal? I would like to be able to update the dependencies there.

[datadavev]

I transferred the sonormal repo to DataONEorg, will make updates and integrations a bit simpler

[iannesbitt]

Perfect, thank you!

[iannesbitt]

I finally got a working conda virtual environment. To do so I had to:

• Install uWSGI using conda since the wheel build fails at the linking step (complains about PEP 517 but won't build with pip install uwsgi==2.0.19 --no-use-pep517 either)
• Changed the following tag:

  -pyld = {git = "https://github.com/datadavev/pyld.git", tag = "2.0.4-dev_ddv-0.1"}
  +pyld = {git = "https://github.com/datadavev/pyld.git", tag = "ddv-2.1.0"}

• Changed click version

  -click = "7.1.2"
  +click = "8.1.3"

• poetry add the following dependencies which solved import errors:

  jinja2 = "<3.0"
  markupsafe = "^2.0.1,<2.1"
  itsdangerous = "^2.0.1,<2.1"
  werkzeug = "^2.0.3,<2.1"
  dataone-libclient = "^3.5.0"
  pyshacl = "^0.22.2"
  attrs = ">=21.3.0"

• Removed jupyter-book from [tool.poetry.dev-dependencies] as it had conflicting dependencies with attrs = ">=21.3.0"

[iannesbitt]

I will most likely end up redesigning the installation process like this as although it is unorthodox, it will significantly reduce the pain to reproducibly install mnlite in a new environment.

[iannesbitt]

Turns out uwsgi wasn't building because it wanted an older version of gcc (unbit/uwsgi#1770). I solved the build issue by doing the following:

apt-get update && apt-get install gcc-4.8 && rm /usr/bin/gcc && ln -s /usr/bin/gcc-4.8 /usr/bin/gcc
poetry add uwsgi # or: pip install uwsgi

and then re-linking the gcc version that was formerly linked (gcc-7):

rm /usr/bin/gcc && ln -s gcc-7 /usr/bin/gcc

[iannesbitt]

This can also be solved without root access by doing the following:

conda install -c free gcc

See that it's linked properly and is the correct version:

$ which gcc
#> /home/mnlite/miniconda3/envs/mnlite/bin/gcc
$ gcc --version
#> gcc (GCC) 4.8.5
#> Copyright (C) 2015 Free Software Foundation, Inc.
#> This is free software; see the source for copying conditions.  There is NO
#> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Then poetry add uwsgi