implement filepicker-django with less custom code

[btbonval]

"The filepicker django library defines the FPFileField model field so you can get all the benefits of using Filepicker.io as a drop-in replacement for the standard django FileField. No need to change any of your view logic." https://github.com/Ink/django-filepicker/wiki#models

And yet here's part of our view logic that we've changed: https://github.com/FinalsClub/karmaworld/blob/6688df3805a24b7996d04649b4443413dce83552/karmaworld/apps/document_upload/views.py#L12

which is called from this AJAX url: https://github.com/FinalsClub/karmaworld/blob/675f7b061f2a96f3ce87ac92bda5c2f8554a0578/karmaworld/urls.py#L76-L77

and that url is called by a submit for the form in this template... https://github.com/FinalsClub/karmaworld/blob/497aab5d45513b0d8d39b928a28a6f69d3d9a8b6/karmaworld/templates/partial/filepicker.html#L53

... the form of which contains a pretty major change to view logic: https://github.com/FinalsClub/karmaworld/blob/497aab5d45513b0d8d39b928a28a6f69d3d9a8b6/karmaworld/templates/partial/filepicker.html#L7-L22

The above highlighted code is in stark contrast to step 5 here: https://github.com/Ink/django-filepicker#installation

illustrated in the demo's template and view code as such:

• https://github.com/Ink/django-filepicker/blob/master/demo/templates/home.html#L9-L13
• https://github.com/Ink/django-filepicker/blob/master/demo/filepicker_demo/views.py#L8-L28

More to the point, because of our custom code in the template, all of the following code is unused (the latter two should indeed be used, while former two might not be necessary):

• https://github.com/FinalsClub/karmaworld/blob/8d2b5c043fb74e4e1275807071017dc60a64fd07/karmaworld/apps/notes/models.py#L52-59
• https://github.com/FinalsClub/karmaworld/blob/8d2b5c043fb74e4e1275807071017dc60a64fd07/karmaworld/apps/notes/models.py#L89-L92
• https://github.com/FinalsClub/karmaworld/blob/e79f61fa27aebe5deadfbea750985ac5e04c5171/karmaworld/secret/filepicker.py.example#L9
• https://github.com/FinalsClub/karmaworld/blob/b3e9fe27f2b0de7adeacab1c9db847e7123c1dd0/karmaworld/settings/common.py#L13-15

[btbonval]

agggg. this code: https://github.com/FinalsClub/karmaworld/blob/9b8fd07613313a6f6ad2c97c21a1c03db768c970/karmaworld/apps/notes/models.py#L125-L137

turned out to be copy pasta! and I replaced it with the original from here: https://github.com/btbonval/django-filepicker/blob/693eedf55789d852f11a45a97685a16806e309b1/django_filepicker/utils.py#L44-L56

yet the former loaded a file that could be read while the latter loads a closed file.

[btbonval]

oh here's the other part of the original code. also copy pasta. https://github.com/FinalsClub/karmaworld/blob/9b8fd07613313a6f6ad2c97c21a1c03db768c970/karmaworld/apps/notes/models.py#L114-L123

original is here, being called in the same order and all: https://github.com/btbonval/django-filepicker/blob/693eedf55789d852f11a45a97685a16806e309b1/django_filepicker/utils.py#L62-L71

[btbonval]

gets closed as soon as its retrieved.

In [1]: from karmaworld.apps.document_upload.models import RawDocument
In [2]: f = RawDocument.objects.all()[0]
In [3]: f
Out[3]: <RawDocument: RawDocument at https://www.filepicker.io/api/file/Yt7TWmagQOOwRDqwdVFU (from None)>
In [4]: fd = f.get_file()
In [5]: dir(fd)
...
In [6]: fd.file
Out[6]: <closed file '/tmp/tmpdHXDTB', mode 'r' at 0x9669e38>

[btbonval]

Good news: file comes back with proper data if I forget to submit permissions. The data? Tells me to include permission stuff.

In [7]: import django_filepicker
In [8]: fpf = django_filepicker.utils.FilepickerFile(f.fp_file.name)
In [9]: dir(fpf)
...
In [10]: fpf.url
Out[10]: u'https://www.filepicker.io/api/file/Yt7TWmagQOOwRDqwdVFU'
In [11]: test = fpf.get_file()
In [12]: test
Out[12]: <File: tmp3O2rVn>
In [13]: test.file
Out[13]: <open file '/tmp/tmp3O2rVn', mode 'r' at 0x96ee020>
In [14]: test.file.read()
Out[14]: '[uuid=60CC6DC745514780] This action has been secured by the developer of this website. Error: Proper credentials not provided. Signature: None\nPolicy: None'

[btbonval]

In [16]: test = fpf.get_file(f.fp_file.field.additional_params)
In [17]: test.file
Out[17]: <open file '/tmp/tmpzADNzW', mode 'r' at 0x97af498>
In [18]: test.read()
(yup, that's the data)

Okay, so calling it above does the right thing, even though this is the same:

    def get_file(self):
        """ Downloads the file from filepicker.io and returns a
        Django File wrapper object """
        fpf = django_filepicker.utils.FilepickerFile(self.fp_file.name)
        return fpf.get_file(self.fp_file.field.additional_params)

[btbonval]

Just to exactly copy all two lines of get_file in Document, one more time, just to see:

In [29]: fpf = django_filepicker.utils.FilepickerFile(f.fp_file.name)
In [30]: fpf.get_file(f.fp_file.field.additional_params).file
Out[30]: <open file '/tmp/tmpFrS1Cv', mode 'r' at 0x97af968>

I'm lost. The world has left me behind, and now I am surely dead. For logic and functional processes, repeatable subsets of call stacks, all for nought.

[btbonval]

The file is opened in Document.get_file(), which has a printout added to it, but is closed after it returns from Document.get_file(). So far as I know, there are no other steps.

In [5]: f.get_file().file
<open file '/tmp/tmp9ZXblx', mode 'r' at 0x96e5e38>
inside model: <open file '/tmp/tmp9ZXblx', mode 'r' at 0x96e5e38>
Out[5]: <closed file '/tmp/tmp9ZXblx', mode 'r' at 0x96e5e38>

[btbonval]

No other steps. Inside f.get_file shown below, file is open. when file returns up to shell? closed.

In [8]: f.get_file.im_func
Out[8]: <function karmaworld.apps.notes.models.get_file>
In [9]: from karmaworld.apps.notes.models import Document
In [10]: Document.get_file.im_func
Out[10]: <function karmaworld.apps.notes.models.get_file>

[btbonval]

I've been debugging the junk out of this. I subclassed Django's File, intercepted File.close with a pdb, and it never gets called.

Yet the file inside the File ends up closed.

I can't do the same trick with file, it's a primitive type in Python.

Seriously I have no idea why this file gets shut from being returned by a method! In debug, I can run the same command, read from the file, all kinds of stuff. When I return out of the method, all file descriptors I created in this fashion are closed.

I think it's time for a break because nothing makes sense.

[btbonval]

plan for tomorrow: copy contents of the file into StringIO and trying passing that into File. At least if StringIO gets closed mysteriously along the way, I can debug that. I can't drop debug into the file type to see when it gets closed.

[AndrewMagliozzi]

Whoa! Byran. Maybe Charles can help if he has a moment.

[charlesconnell]

@btbonval, when you get a chance, could you summarize what the problem is with django-filepicker and what you've done with it? Also, what are we trying to solve in the first place?

[btbonval]

@charlesconnell "less custom code" means we make use of code django-filepicker wrote, rather than code we have customized. Although "customized" turns out to be a lot of code copied from django-filepicker soure code and pasted into our project. For example, Document.get_file was code copy/pasted right out of a django-filepicker function in FPFileField. Why?! We could just call Document.fp_file.get_file! That violates Do not Repeat Yourself, and in this case we're talking about only partial repetition of upstream code. That is problematic on many levels. Quick example: django-filepicker decide to change how they handle something related to get_file. They update their code and it is self consistent, but our copied version of get_file is out of date with new assumptions and breaks. If we simply called their functions in the first place, we'd get their assumptions built in with updates.

Since django-filepicker is sort of the core of our project, @AndrewMagliozzi and I decided we should make proper use of it instead of all this copy/paste repetition. That's what I'm doing in this ticket.

Additionally, without a revamp like this, we couldn't make proper use of a private API secret for signing permissions to Filepicker. For example, we haven't been using secret.filepicker for uploading files! I think it might be used by gdrive, but absolutely not for Filepicker uploads. That was a big problem when it came time to integrate signatures and permissions. I was working on signatures and permissions in this ticket as well. That's all done and aces.

I haven't pushed any of my work in progress because as soon as I do, I lose git rebase master functionality. It's a bad habit, but you convinced me that merges are ugly. So there's not much to look at and test right now.

This ticket was approximately 7 other problems rolled together. I could push all my changes, close this ticket, and open a new ticket for the last remaining problem which is basically that the file primitive (what comes from open(file, 'r')) that gdrive.py needs is being closed for no apparent reason. I'm probably just going to move the content into a StringIO as a workaround to get this ticket done.

[btbonval]

I only left things hanging because 1) I was pissed off and 2) it was 7:30am and I wanted to sleep some hours before.

[btbonval]

Figured out the problem. No wonder it was hard to find. garbagecollection was responsible for closing the file, which is transparent.

FilepickerFile from django-filepicker downloads the file from FP and drops the data in a temp file. We only need the temp file, not the FilepickerFile, so the temp file is returned but the FilepickerFile is dropped.

When FilepickerFile gets garbage collected (FilepickerFile.__del__), it closes the temp file it had opened.

[sethwoodworth]

Because the forms look like balls and can't be styled easily

On Fri, Jan 31, 2014 at 12:21 AM, Bryan Bonvallet notifications@github.comwrote:

It looks like we do not presently use forms directly in our templates. Who decided to use Django only to ignore everything it does? Anyway, here's how to do it:

https://docs.djangoproject.com/en/1.5/ref/forms/api/#outputting-forms-as-html

Reply to this email directly or view it on GitHubhttps://github.com/FinalsClub/karmaworld/issues/308#issuecomment-33762557 .

[btbonval]

@sethwoodworth previously:

        <input id="filepicker-file-upload"
          type="filepicker-dragdrop"
          data-fp-apikey="A5pg98pDjQk6k3lBZ8VDVz"
          data-fp-multiple="true"
          data-fp-folders="true"
          data-fp-button-class="add-note-btn small-10 columns large-4"
          data-fp-button-text="<i class='fa fa-arrow-circle-o-up'></i> add notes"
          data-fp-drag-text="Drop Some Knowledge"
          data-fp-drag-class="dragdrop show-for-medium-up large-7 columns"
          data-fp-extensions=".pdf,.doc,.docx,.txt,.html,.rtf,.odt,.png,.jpg,.jpeg,.ppt,.pptx"
          data-fp-store-path="{{ course.school.slug }}/ {{ course.slug }}/"
          data-fp-store-location="S3"
          data-fp-services="COMPUTER,DROPBOX,URL,GOOGLE_DRIVE,EVERNOTE,GMAIL,BOX,FACEBOOK,FLICKR,PICASA,IMAGE_SEARCH,WEBCAM,FTP"
          />

now:

    fp_file = django_filepicker.models.FPFileField(
                # FPFileField settings
                apikey=FILEPICKER_API_KEY,
                services='COMPUTER,DROPBOX,URL,GOOGLE_DRIVE,EVERNOTE,GMAIL,BOX,FACEBOOK,FLICKR,PICASA,IMAGE_SEARCH,WEBCAM,FTP',
                additional_params={
                    'data-fp-multiple': 'true',
                    'data-fp-folders': 'true',
                    'data-fp-button-class':
                      'add-note-btn small-10 columns large-4',
                    'data-fp-button-text':
                      mark_safe("<i class='fa fa-arrow-circle-o-up'></i> add notes"),
                    'data-fp-drag-class':
                      'dragdrop show-for-medium-up large-7 columns',
                    'data-fp-drag-text': 'Drop Some Knowledge',
                    'data-fp-extensions':
                      '.pdf,.doc,.docx,.txt,.html,.rtf,.odt,.png,.jpg,.jpeg,.ppt,.pptx',
                    'data-fp-store-location': 'S3',
                    'data-fp-policy': fp_policy,
                    'data-fp-signature': fp_signature,
                    'onchange': "got_file(event)",
                },

moved from template file full tag attributes to a dictionary. Same code, different place, follows django-filepicker more closely and adds more dynamic capability such as not hard coding the API key.

[btbonval]

@sethwoodworth To be fair, django-filepicker doesn't document a damned thing. I figured this out by reading every line of source code.

I also had 2 or 3 commits that will be going to a pull request for django-filepicker because it doesn't handle Filepicker security a la signature and policy. That's a new thing though.

[btbonval]

@sethwoodworth Oh yeah, I also had to muck about with the Django form stuff a bit:

class FileUploadForm(ModelForm):
    auto_id = False
    class Meta:
        model = Note
        fields = ('fp_file',)
        widgets = {
          'fp_file': FPFileWidget(attrs={
                       'id': 'filepicker-file-upload',
                     }),
          }

I mean, it kinda sucks because instead of all being in one spot (the template), it's now in the model, form, and view. Seemingly arbitrary where any piece goes. I don't like that, but it's doing things the way Django wants them.

[btbonval]

Submitted a ticket to django-filepicker about deleting the context out of bounds.

As a workaround, I read the data out of the FilepickerFile into an os.tmpfile (1 MiB buffered loop, gross), mangle the Django File to use that os.tmpfile, and return that. Appears to work on the interactive shell.

[btbonval]

Stop being paranoid, s3boto!! although the real problem looks like multiple mappings, this url thing is getting annoying. Need to sort that.

[2014-01-31 19:30:45,414: ERROR/MainProcess] karmaworld.apps.document_upload.tasks.process_raw_document[40917530-2a87-44da-97a0-cd893584c38c]: Traceback (most recent call last):
  File "/home/vagrant/karmaworld/karmaworld/apps/document_upload/tasks.py", line 16, in process_raw_document
    convert_raw_document(raw_document, user=user)
  File "/home/vagrant/karmaworld/karmaworld/apps/notes/gdrive.py", line 254, in convert_raw_document
    logger.info("Zero or multiple mappings found with fp_file " + raw_document.fp_file.url)
...
  File "/var/www/karmaworld/venv/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 64, in _get_url
    return self.storage.url(self.name)
  File "/var/www/karmaworld/venv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 257, in url
    name = self._normalize_name(self._clean_name(name))
  File "/var/www/karmaworld/venv/local/lib/python2.7/site-packages/storages/backends/s3boto.py", line 156, in _normalize_name
    raise SuspiciousOperation("Attempted access to '%s' denied." % name)
SuspiciousOperation: Attempted access to 'https:/www.filepicker.io/api/file/vcT2j4JITelO7V7xuiPq' denied

[btbonval]

Oh that zero or multiple mappings is fine. It's talking about no Notes with that URL, but that's okay because we haven't got around to making the RawDocument into a Note.

This seems relevant to the s3boto paranoia: http://stackoverflow.com/questions/10390244/how-to-set-up-a-django-project-with-django-storages-and-amazon-s3-but-with-diff

[btbonval]

Nope. Tried above fix and S3 is still paranoid, crashing the whole system because it doesn't like a URL.

[btbonval]

path of least resistance: last time I had a problem with FPFileField.url, I noticed the original code used FPFileField.name and switched back to that. No problems. So I switched this printout from FPFileField.url to FPFileField.name as well.

End to end test completed with revamped Filepicker junk AND security. I'd be more excited about finally gettin this done if I hadn't expended so much energy hating every step of the process.

[btbonval]

okay. nuke VM, restart, make sure it all still works. if so, will push commits.

[btbonval]

fresh VM. added course, uploaded note using Filepicker auth, deleted course (which cascaded to note, rawdocument, and 3 tags).

[btbonval]

(FFVII triumph song)

[AndrewMagliozzi]

nice work Bryan. Thanks for the yeoman effort

On Fri, Jan 31, 2014 at 9:09 PM, Bryan Bonvallet notifications@github.comwrote:

<>

Reply to this email directly or view it on GitHubhttps://github.com/FinalsClub/karmaworld/issues/308#issuecomment-33860415 .