search

ryansb / ofCourse

Python courseware leveraging Flask and OpenShift
Apache License 2.0
12 stars 14 forks source link

A different dulwich error using Fedora 23 #75

Open ghost opened 8 years ago

ghost commented 8 years ago

I've generated a 64-bit Fedora 23 workstation virtual machine to try (among other reasons) to get closer to what other ofCourse developers may be using in their own builds.

I end up with this traceback:

Traceback (most recent call last):
  File "/home/joe/hfoss-S16/ofcourse_environment/bin/ofcourse", line 10, in 
    sys.exit(cli())
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/__init__.py", line 204, in openshift
    app_url = push(appname, api, domain)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 119, in push
    map(os.remove, openshift_files.keys())
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 79, in __exit__
    six.reraise(exc_type, value, tb)
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/ofcourse/cli/openshift_utils.py", line 101, in push
    repo.do_commit("Commit openshift files")
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/repo.py", line 565, in do_commit
    committer = self._get_user_identity()
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/repo.py", line 496, in _get_user_identity
    config.get(("user", ), "name"),
  File "/home/joe/hfoss-S16/ofcourse_environment/lib/python2.7/site-packages/dulwich/config.py", line 380, in get
    raise KeyError(name)
KeyError: 'name'

To better document and make consistent what I'm doing, I made the attached script to run through the whole process of generating a test course, and captured a run of it in the second attached file using 'script'.

script.txt output.txt

ryansb commented 8 years ago

It seems like your gitconfig lacks a user/name. Try git config --global user.name "Dee Joe"

ryansb commented 8 years ago

This is due to a missing git username. Reopen if that doesn't fix it.

ghost commented 8 years ago

I'm getting this stack trace

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 159, in openshift
    if is_dirty():
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 130, in is_dirty
    s = git.status(repo)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 549, in status
    tracked_changes = get_tree_changes(r)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 564, in get_tree_changes
    index = r.open_index()
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/repo.py", line 725, in open_index
    return Index(self.index_path())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/index.py", line 203, in __init__
    self.read()
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/index.py", line 223, in read
    f = SHA1Reader(f)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/pack.py", line 1360, in __init__
    self.sha1 = sha1('')
TypeError: Unicode-objects must be encoded before hashing

when running any one of several variants of this script:

GHACCT=ritjoe
COURSE_NAME=testofcourse
SEMESTER=F16
COURSEDIR=${COURSE_NAME}-${SEMESTER}
GHURL=https://github.com/$GHACCT/$COURSE_NAME
#USERNAME="D Joe"
#EMAIL="deejoe+artemisia@etrumeus.com"

mkdir $COURSEDIR
cd $COURSEDIR

virtualenv --no-site-packages -p python3 ofcourse_environment
source ofcourse_environment/bin/activate
pip install ofcourse
mkdir $COURSE_NAME

cd $COURSE_NAME
ofcourse new
git init
git config --replace-all --global user.name "Joe"
git config --replace-all --global user.email "ritjoe@etrumeus.com"
git add .
git commit -m "initial commit"
ofcourse openshift --app $COURSE_NAME --domain ritjoe

It's working up through the openshift deployment.

As per https://github.com/ryansb/ofCourse/issues/75#issuecomment-212119815 I've played around with how I quote user.name and user.email but it seems not to matter.

ghost commented 8 years ago

I'm not sure I can re-open this myself:

http://stackoverflow.com/a/21333938

ryansb commented 8 years ago

That appears to be a dulwich bug - it's not resulting from things ofcourse passes in, but rather choking on decoding the sha (inside plumbing).

ghost commented 8 years ago

pip freeze output, as requested via IRC

$ pip freeze
ansible==2.1.1.0
Babel==2.3.4
ecdsa==0.13
httplib2==0.9.2
iniparse==0.4
Jinja2==2.8
MarkupSafe==0.23
paramiko==1.16.0
pyasn1==0.1.9
pycrypto==2.6.1
pygpgme==0.3
pyliblzma==0.5.3
python-keyczar==0.71rc0
pytz==2016.6.1
PyYAML==3.11
rpm-python==4.13.0rc1
six==1.10.0
virtualenv==14.0.6
ESC[33mYou are using pip version 8.0.2, however version 8.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.ESC[0m
ryansb commented 8 years ago

dulwich isn't in there - is it living outside your virtualenv or something?

ghost commented 8 years ago

Not sure when the script bombs out of the venv, but making sure I'm in the venv and trying the ofcourse openshift step again interactively I get what looks like the same trace, but with this pip freeze:

ESCkjoe@localhost:~/scripted-ofcourse/testofcourse-F16/testofcourseESC\(ofcourse_environment) [joe@localhost testofcourse]$ pip freeze
click==6.6
dulwich==0.10.0
feedparser==5.2.1
Flask==0.11.1
Flask-Mako==0.4
Frozen-Flask==0.12
icalendar==3.10
itsdangerous==0.24
Jinja2==2.8
Mako==1.0.4
MarkupSafe==0.23
ofcourse==0.2.5
oshift==0.2.3
python-dateutil==2.5.3
pytz==2016.6.1
PyYAML==3.11
requests==2.11.1
six==1.10.0
tornado==4.4.1
validator.py==1.2.5
Werkzeug==0.11.10
ghost commented 8 years ago

Confirmed that it was just that the subshell from my script that was in the venv and that therefor left the venv as the subshell exited on finishing the script.

So, that pip freeze is representative of what is in the venv

ghost commented 8 years ago

issuing

pip install --upgrade dulwich

from within the venv takes dulwich from 0.10.0 to 0.14.1 which moves us past the error above, but then generates 

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 159, in openshift
    if is_dirty():
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 131, in is_dirty
    return any(s.staged.values() + [s.unstaged])
TypeError: unsupported operand type(s) for +: 'dict_values' and 'list'

That can then be ameliorated by making the line in question:

    return any([s.staged.values()] + [s.unstaged])

Allows things to move a bit further, before running into this error:

Traceback (most recent call last):
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/bin/ofcourse", line 11, in <module>
    sys.exit(cli())
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/__init__.py", line 204, in openshift
    app_url = push(appname, api, domain)
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/ofcourse/cli/openshift_utils.py", line 93, in push
    git.reset(repo, "hard")
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/porcelain.py", line 558, in reset
    tree = r[committish].tree
  File "/home/joe/scripted-ofcourse/testofcourse-F16/ofcourse_environment/lib/python3.5/site-packages/dulwich/repo.py", line 449, in __getitem__
    type(name).__name__)
TypeError: 'name' must be bytestring, not str

and that's as far as I've gotten.

It looks like they've been working on this in dulwich, see this latest commit: Fix python3 compatibility

But I think the most recent release (from early July) does not yet include the python3 fix

ghost commented 8 years ago

More from our IRC conversation, with regard to forcing an even more recent version of Dulwich with python3 fixes (see above), pending its release:

You can use the git URL in your requirements file
to get the dulwich devel version with Py3 fixes
locally you can do 
pip install -U --force  'git+https://github.com/jelmer/dulwich.git#egg=dulwich'
ryansb commented 8 years ago

Great, that looks like the best solution for now. Once a new release is out, I'll pin the requirements.txt to the newer version for Python 3 installations.

ghost commented 7 years ago

OK, this saw me through last year.

That said, looks like we should put this to bed one way or another now. With the new year starting and some changes in my own environment, I'm taking a look at this again.

On the one hand, so far as I know, no one is using ofcourse this semester so we have some leisure to spend time looking at this instead of OMG PLEASE JUST MAKE IT WORK.

On the other hand, pressures off and easy to forget in light of other priorities.