Open f1-outsourcing opened 7 years ago
Hi,
It's quite likely that some part of CalendarServer, the python distribution plumbing, or the 'autoconf' style plumbing used by C deps is confused that you are putting the SVN checkout in /usr/local, given that this all ends with a 'permission denied' error. The ./bin/develop script is definitely intended to bootstrap a development sandbox, but not to 'install' anything. Get it working in the sandbox first (i.e. in a user's home directory, running as that user), then deal with role accounts and installation later.
-dre
On Feb 8, 2017, at 1:31 PM, F1 Outsourcing Development notifications@github.com wrote:
Obviously i would like to stay as close a possible to installing packages default supplied by rh / centos.
What am i doing wrong? Where is an install howto for rhel/centos? Why not an rpm repository?
Preparing Python requirements
Obtaining file:///usr/local/calendarserver (from -r /usr/local/calendarserver/requirements-cs.txt (line 1)) Obtaining twextpy from git+https://github.com/apple/ccs-twistedextensions.git@a8528bca42bd3529b5186bb6a979da4a764baf07#egg=twextpy https://github.com/apple/ccs-twistedextensions.git@a8528bca42bd3529b5186bb6a979da4a764baf07#egg=twextpy (from -r /usr/local/calendarserver/requirements-cs.txt (line 2)) Updating ./.develop/virtualenv/src/twextpy clone (to a8528bca42bd3529b5186bb6a979da4a764baf07) Could not find a tag or branch 'a8528bca42bd3529b5186bb6a979da4a764baf07', assuming commit. Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 20, in File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 331, in doSetup() File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 288, in doSetup version_string = version() File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 139, in version info = svn_info(source_root) File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 63, in svn_info stderr=subprocess.STDOUT, File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output process = Popen(stdout=PIPE, *popenargs, **kwargs) File "/usr/lib64/python2.7/subprocess.py", line 711, in init errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 13] Permission denied
Command "python setup.py egg_info" failed with error code 1 in /usr/local/calendarserver/.develop/virtualenv/src/twextpy
yum install python2-pip gcc bzip2 openssl-devel readline-devel zlib-devel flex python-devel python-ldap python2-pg8000 python-zope-interface python-pycparser python-cffi python-dateutil pytz python-psutil python-setproctitle python-virtualenv
pip install twextpy
useradd caldav
mkdir /usr/local/calendarserver/ chown caldav.caldav /usr/local/calendarserver/
cd /usr/local/ su caldav
git clone -b CalendarServer-9.0 https://github.com/apple/ccs-calendarserver.git calendarserver
cd /usr/local/calendarserver/
bin/develop — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apple/ccs-calendarserver/issues/471, or mute the thread https://github.com/notifications/unsubscribe-auth/AE_8oBr4333Y22_QzavsGqqnljCxUjuBks5rajQxgaJpZM4L7Y4i.
Of course I meant git, not svn :)
-dre
On Feb 9, 2017, at 11:31 AM, Andre LaBranche dre@mac.com wrote:
Hi,
It's quite likely that some part of CalendarServer, the python distribution plumbing, or the 'autoconf' style plumbing used by C deps is confused that you are putting the SVN checkout in /usr/local, given that this all ends with a 'permission denied' error. The ./bin/develop script is definitely intended to bootstrap a development sandbox, but not to 'install' anything. Get it working in the sandbox first (i.e. in a user's home directory, running as that user), then deal with role accounts and installation later.
-dre
On Feb 8, 2017, at 1:31 PM, F1 Outsourcing Development <notifications@github.com mailto:notifications@github.com> wrote:
Obviously i would like to stay as close a possible to installing packages default supplied by rh / centos.
What am i doing wrong? Where is an install howto for rhel/centos? Why not an rpm repository?
Preparing Python requirements
Obtaining file:///usr/local/calendarserver file:///usr/local/calendarserver (from -r /usr/local/calendarserver/requirements-cs.txt (line 1)) Obtaining twextpy from git+https://github.com/apple/ccs-twistedextensions.git@a8528bca42bd3529b5186bb6a979da4a764baf07#egg=twextpy https://github.com/apple/ccs-twistedextensions.git@a8528bca42bd3529b5186bb6a979da4a764baf07#egg=twextpy (from -r /usr/local/calendarserver/requirements-cs.txt (line 2)) Updating ./.develop/virtualenv/src/twextpy clone (to a8528bca42bd3529b5186bb6a979da4a764baf07) Could not find a tag or branch 'a8528bca42bd3529b5186bb6a979da4a764baf07', assuming commit. Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 20, in File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 331, in doSetup() File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 288, in doSetup version_string = version() File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 139, in version info = svn_info(source_root) File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 63, in svn_info stderr=subprocess.STDOUT, File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output process = Popen(stdout=PIPE, *popenargs, **kwargs) File "/usr/lib64/python2.7/subprocess.py", line 711, in init errread, errwrite) File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child raise child_exception OSError: [Errno 13] Permission denied
Command "python setup.py egg_info" failed with error code 1 in /usr/local/calendarserver/.develop/virtualenv/src/twextpy
yum install python2-pip gcc bzip2 openssl-devel readline-devel zlib-devel flex python-devel python-ldap python2-pg8000 python-zope-interface python-pycparser python-cffi python-dateutil pytz python-psutil python-setproctitle python-virtualenv
pip install twextpy
useradd caldav
mkdir /usr/local/calendarserver/ chown caldav.caldav /usr/local/calendarserver/
cd /usr/local/ su caldav
git clone -b CalendarServer-9.0 https://github.com/apple/ccs-calendarserver.git https://github.com/apple/ccs-calendarserver.git calendarserver
cd /usr/local/calendarserver/
bin/develop — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apple/ccs-calendarserver/issues/471, or mute the thread https://github.com/notifications/unsubscribe-auth/AE_8oBr4333Y22_QzavsGqqnljCxUjuBks5rajQxgaJpZM4L7Y4i.
this virtualenv / running under an unprivileged user installation is not working. If i run the install under root installs the python modules in the default location and then it is working.
I suggest to setup a vm with centos/rhel and investigate this.
Hello @f1-outsourcing
When running from a virtualenv, I found that something like this patch is needed to allow ./bin/develop to succeed. That change is part of the travis-ci integration PR that is not yet merged to master, although this change isn't specific to travis.
CalendarServer can run from any install location as long as PATH and PYTHONPATH are aware of that location. The same goes for any of CalendarServer's dependencies, with the addition of a small but platform-dependent cluster of environment vars: C_INCLUDE_PATH, LD_LIBRARY_PATH, CPPFLAGS, LDFLAGS, DYLD_LIBRARY_PATH - see https://github.com/apple/ccs-calendarserver/blob/master/bin/package#L154. These aren't all needed by any one platform, and some are used only at compile time, but the distinction between compile-time and run-time is often abstracted away, so we set them all in the environment.sh produced by ./bin/package.sh
The reason it worked for you in /usr/local is that this directory is typically included by default in all of the search paths configured by the above environment variables. The reason you needed to be root is likely that /usr/local and its descendants are only writable by root, by default. The ./bin/develop script doesn't expect to be run from anyplace where it doesn't have write access to the filesystem.
... and while trying to pin down exactly where it might be trying to write, other than /usr/local/calendarserver, I spotted a smoking gun of a different sort, from the traceback you posted:
File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 63, in svn_info
We switched from svn to git several months back, and the string "svn_info" does not appear in the current version of twextpy (aka 'ccs-twistedextensions') - the svn_info function was replaced with git_info in this change. I guess this indicates the version you got from pip is not the most recent version... Anyway, I'm not exactly sure why this would manifest as a 'permissions denied' error - perhaps something was trying to execute one of the http redirect responses returned by svn.calendarserver.org, or maybe it's that the 'editable' flag is present on twextpy's entry in the requirements file, which maybe implies writability to the current twextpy install location. In any case, you always need the version of twextpy referenced by CalendarServer's requirements files - in this case, requirements.cs - that's the copy on master, but for a release we reference a specific version.
Hmmmm, but if I do a
git clone -b CalendarServer-9.0 https://github.com/apple/ccs-calendarserver.git calendarserver
I expect to checkout a stable (working) version. That does not need patching or a lot of reading.
@f1-outsourcing that is a reasonable expectation. Unfortunately, even for our most supported platform targets (macOS and Oracle Linux), there is still at least some setup required prior to running ./bin/develop (such as installing pip / virtualenv if needed).
In this case, the ./bin/develop script will get twextpy for you if it's not already present on your system. You have manually installed a version of twextpy that's older than the one specified by CalendarServer, so remove that and re-run ./bin/develop.
Once ./bin/develop has completed successfully, you should be able to use ./bin/package to produce an installation dstroot. This dstroot includes an environment.sh file, which should be sourced by the shell of the user that will run caldavd.
Obviously i would like to stay as close a possible to installing packages default supplied by rh / centos.
What am i doing wrong? Where is an install howto for rhel/centos? Why not an rpm repository?
Preparing Python requirements
Obtaining file:///usr/local/calendarserver (from -r /usr/local/calendarserver/requirements-cs.txt (line 1)) Obtaining twextpy from git+https://github.com/apple/ccs-twistedextensions.git@a8528bca42bd3529b5186bb6a979da4a764baf07#egg=twextpy (from -r /usr/local/calendarserver/requirements-cs.txt (line 2)) Updating ./.develop/virtualenv/src/twextpy clone (to a8528bca42bd3529b5186bb6a979da4a764baf07) Could not find a tag or branch 'a8528bca42bd3529b5186bb6a979da4a764baf07', assuming commit. Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 20, in
File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 331, in
doSetup()
File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 288, in doSetup
version_string = version()
File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 139, in version
info = svn_info(source_root)
File "/usr/local/calendarserver/.develop/virtualenv/src/twextpy/setup.py", line 63, in svn_info
stderr=subprocess.STDOUT,
File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output
process = Popen(stdout=PIPE, *popenargs, **kwargs)
File "/usr/lib64/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
Command "python setup.py egg_info" failed with error code 1 in /usr/local/calendarserver/.develop/virtualenv/src/twextpy