389-ds-bot commented 4 years ago

Cloned from Pagure issue: https://pagure.io/389-ds-base/issue/50037

Created at 2018-11-21 20:55:23 by vashirov (@vashirov)
Assigned to vashirov (@vashirov)

Issue Description

When I try to install lib389 in virtual environment under non-root user, it fails. Because it tries to install cli utils under /usr/sbin:

    running install_data
    copying cli/dsctl -> /usr/sbin/
    error: could not delete '/usr/sbin/dsctl': Permission denied

and man pages under /usr/share/man.

We should use data_files for man pages with relative path and scripts for cli tools.

389-ds-bot commented 4 years ago

Comment from vashirov (@vashirov) at 2018-12-12 17:09:34

3144

389-ds-bot commented 4 years ago

Comment from vashirov (@vashirov) at 2018-12-12 17:09:35

Metadata Update from @vashirov:

Custom field component adjusted to None
Custom field origin adjusted to None
Custom field reviewstatus adjusted to None
Custom field type adjusted to None
Custom field version adjusted to None

389-ds-bot commented 4 years ago

Comment from vashirov (@vashirov) at 2018-12-12 17:10:27

Using scripts for cli tools sadly doesn't work, as it supports only bin directory, but we're installing under sbin

389-ds-bot commented 4 years ago

Comment from mreynolds (@mreynolds389) at 2018-12-20 17:59:36

Metadata Update from @mreynolds389:

Issue set to the milestone: 1.4.0

389-ds-bot commented 4 years ago

Comment from vashirov (@vashirov) at 2019-05-23 11:46:44

Commit c4a2eb4a fixes this issue

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-13 15:00:40

Metadata Update from @Firstyear:

Issue status updated to: Open (was: Closed)

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-13 15:01:47

Hi, this causes a regression:

{william@leap 23:00} ~/development/389ds/ds I0> find /usr | grep -i dscreate
/usr/lib/python3.6/site-packages/lib389-1.4.0.1-py3.6.egg/sbin/dscreate
/usr/lib/python3.6/site-packages/lib389-1.4.0.1-py3.6.egg/share/man/man8/dscreate.8

Doing a make && make install, the sbin tools install into the egg, not into the sbin paths. It's likely this may also break rpm packaging too. So I think this needs to be done differently to avoid this problem ... :(

389-ds-bot commented 4 years ago

Comment from vashirov (@vashirov) at 2019-06-13 15:21:21

IMO, ds* tools should not be installed from lib389 in the first place. They should be part of 389-ds-base package, not python3-lib389.

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-13 15:23:23

An admin may want to install the admin tools on a machine (IE their workstation, or an admin tool container) without installing all of the 389-ds package. So being in python3-lib389 is very reasonable, and intentional. If they were not in python3-lib389 they would be in a 389-ds-cli which deps on python3-lib389.

Either way, that comment does not address the fact that make && make install is now unable to access the cli after this change, and it will probably break our rpms. So either we need to revert this setup.py change, or the cli tools have to be installed as part of autotools instead of setup.py which I don't really want to do given that's not "pythonic".

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-13 15:39:50

3502

389-ds-bot commented 4 years ago

Comment from vashirov (@vashirov) at 2019-06-13 15:46:39

An admin may want to install the admin tools on a machine (IE their workstation, or an admin tool container) without installing all of the 389-ds package. So being in python3-lib389 is very reasonable, and intentional. If they were not in python3-lib389 they would be in a 389-ds-cli which deps on python3-lib389.

lib389 is a library (it's in its name!). Shipping administration tools as part of library doesn't make sense to me. I think the split to 389-ds-cli is a very reasonable option.

Either way, that comment does not address the fact that make && make install is now unable to access the cli after this change, and it will probably break our rpms.

Key word is "probably". It didn't break Fedora rpms:

[root@server-f30 ds]# rpm -q 389-ds-base 
389-ds-base-1.4.1.3-1.fc30.x86_64
[root@server-f30 ds]# rpm -ql python3-lib389 | grep dsc
/usr/sbin/dsconf
/usr/sbin/dscontainer
/usr/sbin/dscreate
/usr/sbin/dsctl
/usr/share/man/man8/dsconf.8.gz
/usr/share/man/man8/dscreate.8.gz
/usr/share/man/man8/dsctl.8.gz

So either we need to revert this setup.py change, or the cli tools have to be installed as part of autotools instead of setup.py which I don't really want to do given that's not "pythonic".

They can be installed as part of spec file (this is how it's done currently): https://src.fedoraproject.org/rpms/389-ds-base/blob/master/f/389-ds-base.spec#_750

Again, IMO, the split is the most sensible option: have another python package with the cli tools that will use python3-lib389 as a dependency. Having lib389 installed in venv is a step in the right direction, since the administrator can install latest and the greatest ds* cli tools and lib389 in venv without root access to the system (as it was before).

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-13 15:49:40

Okay, so I can do the suse work to make it 389-ds-cli on my side - but here the thing is prefix builds still break with this change .... So those are used by myself and some others so we need to do something, not just virtualenv (because I don't use them).

389-ds-bot commented 4 years ago

Comment from vashirov (@vashirov) at 2019-06-13 16:32:02

Before we start doing changes, let's step back for a moment and discuss what exactly are the requirements. Perhaps we can come up with a solution that works for everyone.

I see the following, please correct me if I miss something:

lib389 should be installable in venv without root. This is useful for testing, having lib389 installed separately from the system one, ability to install from pypi, etc.
lib389 should be installable in prefix build and have a working ds* cli tools.
ds* cli tools should be installable without the main server as they can be used by the administrator from their workstation.
- note: dscreate and dsctl are useless without the server on the same host, dsconf and dsidm can be used from another host.

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-14 09:38:01

lib389 should be installable in venv without root. This is useful for testing, having lib389 installed separately from the system one, ability to install from pypi, etc. lib389 should be installable in prefix build and have a working ds* cli tools.

So these two are the concern of this ticket. By adding venv support, prefix installs are broken. IMO prefix builds matter more as it's how development occurs. So my suggestion:

We add the revert patch to allow developers to work again.
To support venv, we need to look at how setup.py in the rpm works, and extract that "magic" and make it part of the lib389-install/lib389 targets in Makefile.am
Then we can re-do the change from /usr/sbin/ -> sbin/ in setup.py to support venvs and prefix.

ds* cli tools should be installable without the main server as they can be used by the administrator from their workstation. note: dscreate and dsctl are useless without the server on the same host, dsconf and dsidm can be used from another host.

Yep, I'm aware. dscontainer also is useless unless it's on the host. Saying this, this is really a packaging problem for downstream, and I'm confused why it was brought up in this ticket when the problem here is about broken prefix builds and potential to break downstream, rather than saying the package names for downstream seem incorrect .....

So I think this is up to me for SUSE to do, and @mreynolds389 for RH/Fedora to do if we consider it worthwhile for those distros/platforms.

Lets focus on the prefix build issue.

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-14 09:51:35

Here are the suse build macros, I'll try and see if these work for this.

{william@leap 17:50} ~/development/389ds I0> rpm --eval '%python_build'

python_flavor=`[ -f _current_flavor ] && cat _current_flavor || true`
if [ -z "$python_flavor" ]; then python_flavor="tmp"; fi
if [ "$python_flavor" != "python2" ]; then
    if [ -d build ]; then mv build _build.$python_flavor; fi
    if [ -d _build.python2 ]; then mv _build.python2 build; fi
fi
echo python2 > _current_flavor

/usr/bin/python2 setup.py  build \
    --executable="/usr/bin/python2 -s"

python_flavor=`[ -f _current_flavor ] && cat _current_flavor || true`
if [ -z "$python_flavor" ]; then python_flavor="tmp"; fi
if [ "$python_flavor" != "python3" ]; then
    if [ -d build ]; then mv build _build.$python_flavor; fi
    if [ -d _build.python3 ]; then mv _build.python3 build; fi
fi
echo python3 > _current_flavor

/usr/bin/python3 setup.py  build \
    --executable="/usr/bin/python3 -s"
{william@leap 17:50} ~/development/389ds I0> rpm --eval '%python_install'

python_flavor=`[ -f _current_flavor ] && cat _current_flavor || true`
if [ -z "$python_flavor" ]; then python_flavor="tmp"; fi
if [ "$python_flavor" != "python2" ]; then
    if [ -d build ]; then mv build _build.$python_flavor; fi
    if [ -d _build.python2 ]; then mv _build.python2 build; fi
fi
echo python2 > _current_flavor

/usr/bin/python2 setup.py  install \
    -O1 --skip-build --force --root /home/william/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64 --prefix /usr

python_flavor=`[ -f _current_flavor ] && cat _current_flavor || true`
if [ -z "$python_flavor" ]; then python_flavor="tmp"; fi
if [ "$python_flavor" != "python3" ]; then
    if [ -d build ]; then mv build _build.$python_flavor; fi
    if [ -d _build.python3 ]; then mv _build.python3 build; fi
fi
echo python3 > _current_flavor

/usr/bin/python3 setup.py  install \
    -O1 --skip-build --force --root /home/william/rpmbuild/BUILDROOT/%{NAME}-%{VERSION}-%{RELEASE}.x86_64 --prefix /usr

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-14 10:22:58

And here are the fedora ones

sh-4.4# cat /usr/lib/rpm/macros.d/macros.python3
%__python3 /usr/bin/python3
%python3_sitelib %(%{__python3} -Ic "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
%python3_sitearch %(%{__python3} -Ic "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")
%python3_version %(%{__python3} -Ic "import sys; sys.stdout.write(sys.version[:3])")
%python3_version_nodots %(%{__python3} -Ic "import sys; sys.stdout.write(sys.version[:3].replace('.',''))")
%python3_platform %(%{__python3} -Ic "import sysconfig; print(sysconfig.get_platform())")
%py3dir %{_builddir}/python3-%{name}-%{version}-%{release}

%py3_shbang_opts -s

# Use the slashes after expand so that the command starts on the same line as
# the macro
%py3_build() %{expand:\\\
  CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}"\\\
  %{__python3} %{py_setup} %{?py_setup_args} build --executable="%{__python3} %{py3_shbang_opts}" %{?*}
  sleep 1
}

%py3_build_egg() %{expand:\\\
  CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}"\\\
  %{__python3} %{py_setup} %{?py_setup_args} bdist_egg %{?*}
  sleep 1
}

%py3_build_wheel() %{expand:\\\
  CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}"\\\
  %{__python3} %{py_setup} %{?py_setup_args} bdist_wheel %{?*}
  sleep 1
}

%py3_install() %{expand:\\\
  CFLAGS="${CFLAGS:-${RPM_OPT_FLAGS}}" LDFLAGS="${LDFLAGS:-${RPM_LD_FLAGS}}"\\\
  %{__python3} %{py_setup} %{?py_setup_args} install -O1 --skip-build --root %{buildroot} %{?*}
}

%py3_install_egg() %{expand:\\\
  mkdir -p %{buildroot}%{python3_sitelib}
  easy_install-%{python3_version} -m --prefix %{buildroot}%{_prefix} -Z dist/*-py%{python3_version}.egg %{?*}
}

%py3_install_wheel() %{expand:\\\
  pip%{python3_version} install -I dist/%{1} --root %{buildroot} --strip-file-prefix %{buildroot} --no-deps
}

After some testing I can not get ds* to install in /usr/sbin with --root or --prefix or --install-data, so I stand by my comment that we should revert this, and if venvs are needed, they should be done in a way that doesn't break prefix builds.

389-ds-bot commented 4 years ago

Comment from vashirov (@vashirov) at 2019-06-14 11:09:40

Obligatory xkcd comic: https://xkcd.com/1172/ Let's revert this change to unblock you. Once I have a proper solution, I'll CC you in PR to review it properly (you did review the last time, but apparently it was not a concern back then).

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-14 11:18:49

Well, I didn't realise it would cause this issue .... :( :( there are plenty of times we commit code and things go wrong, and we fix it later. It's the nature of the business I think ...

So I look forward to the next update where we manage to have both working!

389-ds-bot commented 4 years ago

Comment from firstyear (@Firstyear) at 2019-06-14 11:27:32

Commit 09ba2514 relates to this ticket

389-ds-bot commented 4 years ago

Comment from mreynolds (@mreynolds389) at 2020-02-12 16:42:57

Metadata Update from @mreynolds389:

Issue priority set to: minor
Issue set to the milestone: 1.4 backlog (was: 1.4.0)

389ds / 389-ds-base

lib389 fails to install in venv under non-root user #3096

Issue Description

3144

3502