Broken on python3.4 - Githubissues

whitequark commented 8 years ago

An exception (below) happens if I try to upload anything using PhotoBackup 0.1.2 and Bottle v0.12.9:

Traceback (most recent call last):
  File "/usr/local/bin/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/usr/local/bin/bottle.py", line 1732, in wrapper
    rv = callback(*a, **ka)
  File "/usr/local/lib/python3.4/dist-packages/photobackup_bottle/photobackup.py", line 102, in save_image
    validate_password(request)
  File "/usr/local/lib/python3.4/dist-packages/photobackup_bottle/photobackup.py", line 78, in validate_password
    password = request.forms.get('password').encode('utf-8')
  File "/usr/local/bin/bottle.py", line 166, in __get__
    if key not in storage: storage[key] = self.getter(obj)
  File "/usr/local/bin/bottle.py", line 1084, in forms
    for name, item in self.POST.allitems():
  File "/usr/local/bin/bottle.py", line 166, in __get__
    if key not in storage: storage[key] = self.getter(obj)
  File "/usr/local/bin/bottle.py", line 1232, in POST
    data = cgi.FieldStorage(**args)
  File "/usr/lib/python3.4/cgi.py", line 559, in __init__
    self.read_multi(environ, keep_blank_values, strict_parsing)
  File "/usr/lib/python3.4/cgi.py", line 714, in read_multi
    self.encoding, self.errors)
  File "/usr/lib/python3.4/cgi.py", line 561, in __init__
    self.read_single()
  File "/usr/lib/python3.4/cgi.py", line 724, in read_single
    self.read_binary()
  File "/usr/lib/python3.4/cgi.py", line 746, in read_binary
    self.file.write(data)
TypeError: must be str, not bytes

stephanepechard commented 8 years ago

It's a bug on Python itself, fixed for lastest version of Python 3.3, 3.4 and 3.5. My own server runs successfully on Python 3.4.3, is yours up to date?

whitequark commented 8 years ago

Mine is 3.4.2 from Debian stable (jessie). Is there a workaround?

stephanepechard commented 8 years ago

What you could do:

use pyenv to get the version of Python you want on your Debian Jessie (that's what I do on my own) ;
downgrade the Android PhotoBackup client to the 0.8 version, before the use of OkHttp but I can understand this is not very comfortable ;
use another PhotoBackup server implementation like NodeJS, PHP or Go ;

What I could do:

reimplement AAHC (used from the start till PhotoBackup 0.8 and which didn't have the problem, inexplicably), in addition of OkHttp, for the case when it does not work properly (the switch was motivated by the support of redirection by OkHttp) ;
try Flask or Django to see if one might handle this more correctly ;
propose a Docker image that would contain a fully functional PhotoBackup server implementation ; this is something I planned to do anyway.

whitequark commented 8 years ago

Thanks for looking into it. That all seems an awful lot of work to work around Debian's slowness... I was hesitant to change the implementation or use virtualenv since I'm trying to keep my VPS low-maintenance, and having to remember to keep virtualenv up to date is contrary to that.

In any case, I've used the Android app more and found it seriously lacking in usability. I should be able to contribute fixes for that but I won't have time for it very soon, so let's hope that by the time I get back to it, this particular problem goes away on its own without any additional work by either of us.

stephanepechard commented 8 years ago

I understand your will of simplicity for your server, I can only praise on that. Does a Docker image would ease your work a bit? It's quite easy to install and use, even if you need to trust the image builder (and it needs a lot of space). Concerning the Android app, could you be more specific on its "serious lack of usability"? Please, feel free to open issues about it, and I'll try to do my best to make it better.

whitequark commented 8 years ago

Does a Docker image would ease your work a bit? It's quite easy to install and use, even if you need to trust the image builder (and it needs a lot of space).

My issue with Docker is much the same as with virtualenv (except it's even worse in Docker); in short, I want one vulnerable copy of openssl updated from debian-security by cron, and not ten different versions of it in different Docker images, updated... basically never. Repeat for all other dependencies. I don't really even consider Docker for anything exposed to the network.

Concerning the Android app, could you be more specific on its "serious lack of usability"?

The notifications don't have a progress bar (I have ADSL with ~300kbps of upload bandwidth; hard to tell if stuck or still uploading); the notifications don't render properly if I try to upload something after an upload has errored; the initial state of 'upload journal' filtering buttons does not reflect the actual state of the list; while scrolling the list, photos get assigned to completely wrong filenames, and, worse, the sequence of thumbnails shifts across the sequence of filenames as more thumbnails are loaded; the "upload all" option doesn't seem to do anything at all; there is no "(re)upload all pending/errored" button; the "service active" switch gets turned off if the app dies (can be reproduced by force quitting it), which means that it is likely to turn itself off unnoticed at some point if it encounters some unexpected network error.

Um. That's the gist of it. There were a few other minor papercuts but I don't remember them offhand.

stephanepechard commented 8 years ago

My issue with Docker is much the same as with virtualenv

I get your concern. Ultimately, sorry for the inconvenience.

The notifications don't have a progress bar

I tried to put one at a moment, but I couldn't make it, because of AAHC if I remember correctly. I could try again, except if I do #47, which I might ;

the notifications don't render properly if I try to upload something after an upload has errored;

very possible, sometimes had some issues with the notification, but hadn't the time to check on it ; it would be kind of fixed if I do #47 ;

the initial state of 'upload journal' filtering buttons does not reflect the actual state of the list;

it was a bug at some time, it should be corrected now ;

while scrolling the list, photos get assigned to completely wrong filenames ;

that shouldn't happen of course, I'm very surprised by this, except if it is correlated with the following:

and, worse, the sequence of thumbnails shifts across the sequence of filenames as more thumbnails are loaded;

loading is slower than displaying and cells are recycled ; all of this produce the effect you describe, but at the end, it should stabilize on the right images ; still, I recently get this a bit better, but not perfect ; the way to render it (almost) perfectly would be to use cache, but it is a lot of more work...

the "upload all" option doesn't seem to do anything at all ;

where do you see an "upload all" option? AFAIK, there is none ;

there is no "(re)upload all pending/errored" button

this one is on my TODO list :-)

the "service active" switch gets turned off if the app dies (can be reproduced by force quitting it), which means that it is likely to turn itself off unnoticed at some point if it encounters some unexpected network error

this one really surprises me, as it never occurred to me. The service works well after I force quit it (I suppose you don't kill the service in the settings, but even if I do it, it restarts and works as expected), what device do you own and which version of Android do you use?

Thanks for all these inputs!

whitequark commented 8 years ago

I'm using the version 0.10.4 installed from F-Droid; the phone is Xperia Z2 with CM 12.1-20160710 (Android 5.1.1).

loading is slower than displaying and cells are recycled ; all of this produce the effect you describe, but at the end, it should stabilize on the right image

The problem is that, I think, due to Xperia taking pretty high resolution photos (25 MP IIRC) this is taking a lot of time and I only have five hundred or so, so the list becomes kind of useless unless I'm willing to wait a lot.

where do you see an "upload all" option? AFAIK, there is none ;

I mean Pictures to upload → All pictures of the device. (By the way that should probably be "on" and not "of"...)

this one really surprises me, as it never occurred to me.

Just reproduced this again--note that I'm force killing the app itself and not the service. (I think it hung, or maybe I thought it was hung when it really wasn't, and I force-killed it, which is how I discovered this.)

stephanepechard commented 8 years ago

The high resolution of your device should not (in theory) be a problem, as the thumbnails I use are those created and distributed by the operating system. But if you scroll a lot on the screen, it may be painful...

The "Pictures to upload → All pictures of the device" option does not restart an upload, it just alters the preferences to be taken in account at the next pictures taken. Maybe it should restart though.

The restart/kill problem puzzles me more. It will be hard for me to investigate on it as I should use the exact same environment. Still, I can try to use your CM version on the emulator (don't know if I can do this). I have a question : did it hang during upload or while idle?

whitequark commented 8 years ago

The high resolution of your device should not (in theory) be a problem, as the thumbnails I use are those created and distributed by the operating system.

I suspect there is some other problem because the builtin gallery has essentially zero delay loading thumbnails even with rapid scrolling, and moreover, opening the upload journal twice doesn't reduce the delay at all. Some missing opt-in caching mechanism?..

But if you scroll a lot on the screen, it may be painful...

The use case here was to find particular photos I cared most about, since, see above, 300kbps upload.

The "Pictures to upload → All pictures of the device" option does not restart an upload, it just alters the preferences to be taken in account at the next pictures taken. Maybe it should restart though.

But how does it differ from the other setting then? All pictures taken after I've installed the app are "newly taken", unless I misunderstand something.

The restart/kill problem puzzles me more. It will be hard for me to investigate on it as I should use the exact same environment.

Assuming other stuff gets fixed, I can simply debug that myself, I'm familiar with Android app development.

I have a question : did it hang during upload or while idle?

I don't recall, sorry.

stephanepechard commented 8 years ago

Some missing opt-in caching mechanism?..

Yes, as I said, I didn't implement any cache mechanism

But how does it differ from the other setting then? All pictures taken after I've installed the app are "newly taken", unless I misunderstand something.

well, it may be unclear, but if all pictures are already backed up, both settings are identical: as soon as a picture is taken, it is uploaded ; the "newly taken" option is mainly to avoid back up everything again in case of reinstall for example. Writing it just settle how unclear it is though...

PhotoBackup / server-bottle

Broken on python3.4 #17