Closed robarchibald closed 7 years ago
robarchibald
commented
7 years ago Just wanted to make a comment about this pull request. I really LOVE the ccs-calendarserver project. I'm using it on a Ubuntu Linux 16.04 server and it's working great so far. I would be willing to make several other pull requests of fixes I found while installing on Ubuntu if you'd be willing to consider them. Great job to the team though! I love this Calendar Server!
dreness
commented
7 years ago The travis error here can be ignored; I haven't yet merged the travis-ci integration, yet for some reason it's still trying to test pull requests...
dreness
commented
7 years ago @robarchibald Thanks for the kind comments :) I'd like to at least see the changes you've made for ubuntu, especially if they are working around a bug or incorrect assumption in our code. Historically our primary targets have been macOS and Oracle Linux, but we know of users on Ubuntu, FreeBSD, and Debian - in fact there are community-provided packages for FreeBSD and Debian. The dependency setup is somewhat complicated and platform-specific, so there's a lot that can go wrong, and generally we're ok with leaving that work for package maintainers. That said, if there are low-risk / low-maintenace changes to CS that make life on Ubuntu better, we'll definitely consider them!
If you haven't already done so, I'd recommend running the unit tests and CalDAVTester (./bin/test and ./bin/testserver, relative to the top of the git checkout) on your system, both with and without your changes. On the ubuntu baseline I'm using for travis-ci, the CalDAVTester suite passes all green, however there are a few unit tests failures (although at least some of those appear specific to the travis-ci environment).
robarchibald
commented
7 years ago Andre, Just getting back to this again. So, here are the things I've had to do to get the 9.0 release to work on Ubuntu.
To getbin/develop to work, I needed to change the dependency for cryptography in requirements-twisted-default.txt to be 1.5.2 instead of 1.1 which is what is there in the release. Looking at the value that is currently in the main branch, it now shows 1.6 so this looks to already be fixed. This was to resolve the following error on compile:
build/temp.linux-x86_64-2.7/_openssl.c:695:6: error: conflicting types for ‘BIO_new_mem_buf’
BIO *BIO_new_mem_buf(void *, int);
^
In file included from /usr/include/openssl/asn1.h:65:0,
from build/temp.linux-x86_64-2.7/_openssl.c:411:
/usr/include/openssl/bio.h:692:6: note: previous declaration of ‘BIO_new_mem_buf’ was here
BIO *BIO_new_mem_buf(const void *buf, int len);
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Rolling back uninstall of cryptography
Command "/home/robarch/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-Sq1fQi/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-OYeUV5-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/robarch/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/include/site/python2.7/cryptography" failed with error code 1 in /tmp/pip-build-Sq1fQi/cryptography2. Even after getting bin/develop to "work", it was still throwing spurious errors. Things such as bin/develop: 555: [: true: unexpected operator
bin/develop: 42: bin/develop: [[: not found bin/develop: 42: bin/develop: [[: not found bin/develop: 42: bin/develop: [[: not found bin/develop: 42: bin/develop: [[: not found
These looked suspiciously like shell scripting errors. Ubuntu will default to sh when you run a shell script without the `#!/bin/bash` at the top (or whatever your favorite shell is), Darwin's default shell is bash, so people probably don't run into this issue there. All I had to do to get rid of these errors was to run `/bin/bash bin/develop` and it worked without any errors. If it wouldn't cause issues on Darwin, it'd be great if we could add `#!/bin/bash` to the top of the script files and then it would work successfully for everyone no matter what.
3. Final error left was this one: .../ccs-calendarserver-CalendarServer-9.0/bin/_py.sh: line 104: [: # 1 "
Turns out the issue was with with the `c_macro` function inside bin/_build.sh. It gets called inside of the `find_header` function to check version numbers of various libraries it requires. In this particular case, it was checking ldap.h and inexplicably got back the value `# 1 "<stdin>"`. One I realized it was ldap.h that was the problem I was able to run `apt-get install libldap2-dev` and it completed without errors.
4. Not Ubuntu-specific, but to get this to work with an external Postgres database I knew I needed to manually install the DB schema to the database. According to https://github.com/apple/ccs-calendarserver/blob/master/doc/Admin/MultiServerDeployment.rst documentation, I needed to run `calendarserver_bootstrap_database`. Unfortunately, I couldn't find that tool anywhere.
5. Related to number 2, I thought that running `bin/calendarserver_check_database_schema` would check my schema and upgrade it if it wasn't up to date. I looked at that script because it sounded the closest to the `calendarserver_bootstrap_database` command based on its name. When running `bin/calendarserver_check_database_schema` as well as other commands in the bin folder, I noticed I kept seeing an error showing the command options and then the error: `option --config not recognized`. Since these scripts are generated at build time, I figure there is a build step that adds the --config option in the script so that it points to the right config file when running all the calendarserver_* commands. Unfortunately, I couldn't find where that was being added so I don't have a pull request for you on that item. In my case, I fixed my issue by deleting the --config option in the script I was using.
6. With the `bin/calendarserver_check_database_schema` script working, I tried to run it, but it immediately crashed. Turns out that now the python script failed when trying to run the PSQL command because it didn't check to ensure PSQL existed before running it. I just submitted one more pull request on this script to make it so that it doesn't crash if PSQL isn't accessible. If it were me, I would also change the default path for PSQL to be ".develop/roots/PostgreSQL/bin/psql" since that should be there for everyone as a part of the package as I understand it, but I don't know if there are other reasons that "../postgresql/_root/bin/psql" was chosen as the default.
7. Now that I had gotten this far, the script was finally working and I was excited to have it do its magic, but it was a complete fail. It didn't update my schema! It did check the schema for me so that was good. Apparently it would be great to get that `calendarserver_bootstrap_database` command back. Eventually I just added the schema from txdav/common/datastore/sql_schema/current.sql and that worked fine.
So, that's what I've uncovered so far. I'll run the bin/test like you've suggested and report my findings with that.Thanks for the detailed feedback. #2 should be fixed by https://github.com/apple/ccs-calendarserver/commit/c5c2c3050fe1c7d4050de9c826c36baad9d0b5af
robarchibald
commented
7 years ago Here are the results of my run of bin/test. From parsing my log, this appears to runs 1967 tests and I only had 13 fails. That's 99.34% successful on Ubuntu which seems like a great start. My biggest concern for now is actually numbers 13 which instead of reporting a FAIL, it reports an ERROR and looks to me like it might crash the test run (see below). My unscientific check for how many tests there are for CalendarServer total grep -r "def test_" . | wc -l revealed 19363 lines of tests so if that's right, I've only run 10% of the tests. Here are the fails:
Here's the full log of the errors:
ConfigTestCase
test_writeConfig ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/calendarserver/tools/test/test_config.py", line 153, in test_writeConfig
self.assertTrue(results["result"]["ServerHostName"].endswith(hostName))
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 400, in assertTrue
super(_Assertions, self).assertTrue(condition, msg)
File "/usr/lib/python2.7/unittest/case.py", line 422, in assertTrue
raise self.failureException(msg)
twisted.trial.unittest.FailTest: False is not true
[FAIL]
GatewayTestCase
test_createLocation ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/calendarserver/tools/test/test_gateway.py", line 299, in test_createLocation
u"Created Location 01 %s %s" % (unichr(208), u"\ud83d\udca3"))
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
super(_Assertions, self).assertEqual(first, second, msg)
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 924, in assertMultiLineEqual
self.fail(self._formatMessage(msg, standardMsg))
twisted.trial.unittest.FailTest: <FailTest instance at 0x7f739a9d5e60 with str error:
Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/python/reflect.py", line 443, in safe_str
return str(o)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xd0' in position 105: ordinal not in range(128)
[FAIL]
test_writeConfig... Traceback (most recent call last): File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/calendarserver/tools/test/test_gateway.py", line 497, in test_writeConfig self.assertTrue(results["result"]["ServerHostName"].endswith(hostName)) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 400, in assertTrue super(_Assertions, self).assertTrue(condition, msg) File "/usr/lib/python2.7/unittest/case.py", line 422, in assertTrue raise self.failureException(msg) twisted.trial.unittest.FailTest: False is not true [FAIL]test_createAddress... Traceback (most recent call last): File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/calendarserver/tools/test/test_gateway.py", line 260, in test_createAddress "Created Address 01 %s %s" % (unichr(208), u"\ud83d\udca3") File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib/python2.7/unittest/case.py", line 924, in assertMultiLineEqual self.fail(self._formatMessage(msg, standardMsg)) twisted.trial.unittest.FailTest: <FailTest instance at 0x7f739949d0f0 with str error: Traceback (most recent call last): File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/python/reflect.py", line 443, in safe_str return str(o) UnicodeEncodeError: 'ascii' codec can't encode character u'\xd0' in position 102: ordinal not in range(128)[FAIL]
test_getLocationAttributes... Traceback (most recent call last): File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks result = g.send(result) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/calendarserver/tools/test/test_gateway.py", line 219, in test_getLocationAttributes "Created Location 01 %s %s" % (unichr(208), u"\ud83d\udca3")) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual super(_Assertions, self).assertEqual(first, second, msg) File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib/python2.7/unittest/case.py", line 924, in assertMultiLineEqual self.fail(self._formatMessage(msg, standardMsg)) twisted.trial.unittest.FailTest: <FailTest instance at 0x7f739a4cc0a0 with str error: Traceback (most recent call last): File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/python/reflect.py", line 443, in safe_str return str(o) UnicodeEncodeError: 'ascii' codec can't encode character u'\xd0' in position 105: ordinal not in range(128)[FAIL]
CalendarSQLStorageTests
test_migrateHomeFromFile ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/caldav/datastore/test/test_sql.py", line 476, in test_migrateHomeFromFile
if self.requirements['home1'][k] is not None]))
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
super(_Assertions, self).assertEqual(first, second, msg)
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 796, in assertSetEqual
self.fail(self._formatMessage(msg, standardMsg))
twisted.trial.unittest.FailTest: Items in the second set but not the first:
'calendar_empty'
[FAIL]
CalendarObjectSplitting
test_calendarObjectSplit_clean_cache ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/caldav/datastore/test/test_sql.py", line 7833, in test_calendarObjectSplit_clean_cache
self.assertTrue(hcount <= 2) # Just user01 home left (maybe user05 too)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 400, in assertTrue
super(_Assertions, self).assertTrue(condition, msg)
File "/usr/lib/python2.7/unittest/case.py", line 422, in assertTrue
raise self.failureException(msg)
twisted.trial.unittest.FailTest: False is not true
[FAIL]
FileStorageTests
test_createCalendarWithName_exists ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_asynctest.py", line 67, in _cb
"did not catch an error, instead got %r" % (ignore,))
twisted.trial.unittest.FailTest: did not catch an error, instead got <Calendar: /etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/trial/txdav.caldav.datastore.test.test/FileStorageTests/test_createCalendarWithName_exis/Zia0uM/temp/store/calendars/uids/ho/me/home1/.SAGvuVQOspfOf1wrcalendar_empty>
[FAIL]
test_removeCalendarWithName_exists ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/caldav/datastore/test/common.py", line 764, in test_removeCalendarWithName_exists
self.assertNotIdentical((yield home.calendarWithName(name)), None)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 467, in assertIsNot
raise self.failureException(msg or '%r is %r' % (first, second))
twisted.trial.unittest.FailTest: None is None
[FAIL]
test_calendars ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/caldav/datastore/test/common.py", line 1312, in test_calendars
set(home1_calendarNames)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
super(_Assertions, self).assertEqual(first, second, msg)
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 796, in assertSetEqual
self.fail(self._formatMessage(msg, standardMsg))
twisted.trial.unittest.FailTest: Items in the second set but not the first:
'calendar_empty'
[FAIL]
test_calendarWithName_exists ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/caldav/datastore/test/common.py", line 675, in test_calendarWithName_exists
self.fail("calendar %r didn't exist" % (name,))
twisted.trial.unittest.FailTest: calendar 'calendar_empty' didn't exist
[FAIL]
test_loadAllCalendars ... Traceback (most recent call last):
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
result = g.send(result)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/caldav/datastore/test/common.py", line 1335, in test_loadAllCalendars
set(home1_calendarNames)
File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/twisted/trial/_synctest.py", line 437, in assertEqual
super(_Assertions, self).assertEqual(first, second, msg)
File "/usr/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python2.7/unittest/case.py", line 796, in assertSetEqual
self.fail(self._formatMessage(msg, standardMsg))
twisted.trial.unittest.FailTest: Items in the second set but not the first:
'calendar_empty'
[FAIL]
SubprocessStartup
test_startService_Socket ... Traceback (most recent call last):
Failure: twisted.trial.util.DirtyReactorAggregateError: Reactor was unclean.
DelayedCalls: (set twisted.internet.base.DelayedCall.debug = True to debug)
<DelayedCall 0x7f7366ccde18 [113.31892395s] called=0 cancelled=0 onTimeout(<Deferred at 0x7f7366ccddd0>)
traceback at creation:
File "
[ERROR]
Traceback (most recent call last): File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/base/datastore/subpostgres.py", line 527, in gotStatus self.ready(createConnection()) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/base/datastore/subpostgres.py", line 437, in createConnection "schema creation", "postgres" File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/base/datastore/subpostgres.py", line 359, in produceConnection return self._connectorFor(databaseName).connect(label) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/txdav/base/datastore/dbapiclient.py", line 252, in connect connection = self.dbModule.connect(self.connectArgs, **self.connectKw) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/pg8000/init.py", line 103, in connect user, host, unix_sock, port, database, password, ssl, timeout) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/pg8000/core.py", line 1616, in init self.message_types[code](self._read(data_len - 4), None) File "/etc/endfirst/calendarserver/ccs-calendarserver-CalendarServer-9.0/.develop/virtualenv/local/lib/python2.7/site-packages/pg8000/core.py", line 1840, in handle_AUTHENTICATION_REQUEST "server requesting MD5 password authentication, but no " pg8000.core.InterfaceError: server requesting MD5 password authentication, but no password was provided [ERROR]
Remove confusing text stating that username is a required argument to run this tool. Add connection URI to enable non-standard configurations of database names, users, passwords, etc.
By submitting a request, you represent that you have the right to license your contribution to Apple and the community, and agree that your contributions are licensed under the Apache License Version 2.0.
For existing files modified by your request, you represent that you have retained any existing copyright notices and licensing terms. For each new file in your request, you represent that you have added to the file a copyright notice (including the year and the copyright owner's name) and the Calendar and Contacts Server's licensing terms.
Before submitting the request, please make sure that your request follows the Calendar and Contacts Server's guidelines for contributing code.