search

new-day-international / reddit

New Day fork of the code that powers reddit.com
Other
3 stars 1 forks source link

File Upload Feature #22

Closed memetic007 closed 11 years ago

memetic007 commented 11 years ago

The reddit engine currently doesn't have the ability for users to upload files, other than thru a very arcane interface in the Moderator tools.

This will be a very early requirement for Lightnet.

here is a proposed early version:

  1. Create a new add an item option - "Upload a File" in addition to current "Submit a new Link" and "Submit a new text post". Like the two current new item options, "Upload a File" should be enabled/disabled in the Moderator tools. Default should be "enabled". done
  2. Clicking the "Upload a File" button should bring up a form with fields like those for Link Item, but with URL replaced by "file to upload". There should be "browse my computer", an "Upload File" button, and a "Clear" button. The "Upload File" and "Clear" buttons should be initially greyed out. done

There will be a "check box" for whether the Item will "appear on the Stream". Default should be "yes" (moved #68)

  1. Once a file is selected via the browse, the "Upload File" and "Clear" buttons should be activated, and colored. Thereafter pressing the "upload" button will upload the selected file as a New Item. Pressing "Clear" prior to uploading will deselect the selected file clearing the input box, uncolor "Upload File" and "Clear" buttons, and return to preselection state. done
  2. A File Item that was checked as "appear on the stream" should be treated like a Link Item... appears in the Stream, All, Front, etc, voteable, commentable, etc. (moved #68)

4a. Items deselected for "appearing on the stream" will only be findable via the Files Stream View in the Space to which they were uploaded. (moved #68)

  1. A New Stream View called "Files" should be created for Spaces if File Uploading is enabled on that Space. Clicking Files should present only the File Items on the Stream, in order of most recent first. done

future enhancements:

  1. A directory structure under files. At upload time, if the target Space has Directories defined, then user Uploading must select the subdirectory (from a menu) where they want their file to to be filed. The File Stream View will display the underlying directories. Note: this version assumes only one level deep directory structure, we may want to consider a deeper structure. (moved to #69)
  2. create View Sorts for Active, Hot, New, etc for the Files Sub Stream... this obviously requires an extension to the UI concept.... will need to think about that UI extension very carefully (moved #69)

    Requirements Added Later:

    • upload files to a common spaces url rather than a user based one done
    • better error message when file is too big to upload
    • busy wait icon rather than % display for IE7/8 where we can't get feedback on upload progress
    • add language to suggest videos should be uploaded to youtube or vimeo done
    • A tweak on the Link Title for accessing the underlying file (see comment below for screen shot with suggested copy)
    • the file view should not appear if file uploading is turned off for the space.
myers commented 11 years ago

Could we remove the "appearing on the stream" or "not appearing on the stream" feature for first cut?

memetic007 commented 11 years ago

yes

myers commented 11 years ago

One thought is that we currently have two Link (that's the name of the ORM class) subtypes, Link to other site and self posts. I think I will make files be a third type. In the long term we can write code to filter out the Link (uploaded file) subtype.

memetic007 commented 11 years ago

That was my assumption, that Files would be a third Link type... similar but niot identical in functionality to Link to Other Site Type. .

On Thu, Aug 22, 2013 at 11:55 AM, Myers Carpenter notifications@github.comwrote:

One thought is that we currently have two Link (that's the name of the ORM class) subtypes, Link to other site and self posts. I think I will make files be a third type. In the long term we can write code to filter out the Link (uploaded file) subtype.

— Reply to this email directly or view it on GitHubhttps://github.com/new-day-international/reddit/issues/22#issuecomment-23101378 .

Jim Rutt JPR Ventures

myers commented 11 years ago

I intend to put a sanity check upload limit. These are not to limit legitimate uses, but to halt attacks (spam, DoS). I also plan to monitor how these limits are working out. These will be in a config file to allow easy changes. I am planning to put these limits on at first: No more than 5GB cumulative total in uploads (stop unbound storage growth), not more than total 5,000 files (stop unbound iteration over filelist time).

Part of the delivery of this ticket will be a solution to monitor how these limits are effecting users so we can catch problems before they happen. Perhaps an emailed report or nagios check (that would mean setting up nagios too)

memetic007 commented 11 years ago

No more than 5GB cumulative total in uploads (stop unbound storage growth), not more than total 5,000 files (stop unbound iteration over filelist time).

Is that per user total quota? Or per user per month? What about file size limits? I'd be good with 10meg as a file size limit. We should strongly discourage people from loading us up with big videos... let 'em use vimeo or YouTube.

On Thu, Aug 22, 2013 at 1:45 PM, Myers Carpenter notifications@github.comwrote:

I intend to put a sanity check upload limit. These are not to limit legitimate uses, but to halt attacks (spam, DoS). I also plan to monitor how these limits are working out. These will be in a config file to allow easy changes. I am planning to put these limits on at first: No more than 5GB cumulative total in uploads (stop unbound storage growth), not more than total 5,000 files (stop unbound iteration over filelist time).

Part of the delivery of this ticket will be a solution to monitor how these limits are effecting users so we can catch problems before they happen. Perhaps an emailed report or nagios check (that would mean setting up nagios too)

— Reply to this email directly or view it on GitHubhttps://github.com/new-day-international/reddit/issues/22#issuecomment-23110995 .

Jim Rutt JPR Ventures

myers commented 11 years ago

Sorry, I messed that up.

5GB / 5,000 per person. Before we show the form (and create a signed doc that allows them to upload directly to S3) we check they are under the limit.

What about putting a note on the page about uploading video and suggest youtube or vimeo?

memetic007 commented 11 years ago

What about putting a note on the page about uploading video and suggest youtube or vimeo?

yes, do it.

If we have a limit of 10MB per file a limit of 500 files would seem to be a reasonable starting place. Be a LONG time I expect anyone other an abuser would reach 500.

On Thu, Aug 22, 2013 at 3:15 PM, Myers Carpenter notifications@github.comwrote:

Sorry, I messed that up.

5GB / 5,000 per person. Before we show the form (and create a signed doc that allows them to upload directly to S3) we check they are under the limit.

What about putting a note on the page about uploading video and suggest youtube or vimeo?

— Reply to this email directly or view it on GitHubhttps://github.com/new-day-international/reddit/issues/22#issuecomment-23117379 .

Jim Rutt JPR Ventures

memetic007 commented 11 years ago

status?

myers commented 11 years ago

Done

Working on

memetic007 commented 11 years ago

No File View yet.....

filesnoview

memetic007 commented 11 years ago

Also, noticed files are being stored in what looks like the User's Directory: http://files.test.lightnet.is/u/Curtis_Faith_007/game_b_logo.png

Probably makes more sense for them to be stored in a Spaces Directory where they were loaded. Thus the above URL might be:

http://test.lightnet.is/space/watercooler/files/game_b_logo.png

If you need the files subdomain for S3 purposes then:

http://files.test.lightnet.is/space/watercooler/game_b_logo.png

memetic007 commented 11 years ago

A tweak on the Link Title for accessing the underlying file

fileslinktitle

memetic007 commented 11 years ago

Tested uploading an oversized file. Correctly failed, but not very informative error message.

Also, is it possible to check the file size BEFORE uploading, and abort then?

filesuploadfailmessage

myers commented 11 years ago

Thanks for all the feedback and ideas. I'll get to hacking on these now, unless you want to peel off some of them to another lower priority ticket.

myers commented 11 years ago

Todo (summarizing the above comments)

myers commented 11 years ago

@memetic007 There's a problem with the flow if we do the files by user edit: I meant space. If you follow the "submit new file" from the front page you end up on a page with no space selected yet. We start uploading the file as soon as the file is picked, but in this case we wouldn't know what space you wanted to post to yet.

If you followed the link from a specific space, the space field is already populated but can be changed after the file has been uploaded to a the specific space's s3 directory.

One idea for a fix would be not have a file upload option on the front page, and once you pick to upload a file the space is locked to the one you where on when you chose to upload.

Another idea is to move the Space selection field to the top of the form. Once the file is uploaded it's disabled for editing.

Another idea is to start uploads when you click the "submit" button.

Any other ideas for solutions?

myers commented 11 years ago

After talking with @inflector about this we picked 'start uploads when you click the "submit" button'.

The user experience would be "you pick a file, the title is guessed, and then you fill out the rest of the form, then click submit. The form fields all become disabled, and the upload progress is printed next to the submit button, when upload completes you are redirected to the links comment page"

memetic007 commented 11 years ago

hmmm, interesting set of issues... yeah any "multiSpace" such as MySpaces, Mod, and pseudo-spaces like Friends will have that problem.

Eliminating the File Option makes the system more modal than it needs to be. (Though I assume you are checking in the case that the desired space doesn't accept File?)

So, my initial reaction would be to:

  1. Move the Space field above the upload filed.
  2. Pop an error if user tries to select file before selecting Space
  3. Lock Space field upon starting of Upload, but unlocking on a Cancel

Other possibility, I suppose is to upload to a temp space, and then rename the S3 key upon Submit. That also provides for a clean way to dispose of files that get uploaded but never submitted .. .they'll be in a temp dir which can be cleaned every day or two.

=jim

On Thu, Sep 5, 2013 at 4:42 PM, Myers Carpenter notifications@github.comwrote:

@memetic007 https://github.com/memetic007 There's a problem with the flow if we do the files by user. If you follow the "submit new file" from the front page you end up on a page with no space selected yet. We start uploading the file as soon as the file is picked, but in this case we wouldn't know what space you wanted to post to yet.

If you followed the link from a specific space, the space field is already populated but can be changed after the file has been uploaded to a specific location.

One idea for a fix would be not have a file upload option on the front page, and once you pick to upload a file the space is locked to the one you where on when you chose to upload.

Another idea is to move the Space selection field to the top of the form. Once the file is uploaded it's fixed.

Another idea is to start uploads when you click the "submit" button.

Any other ideas for solutions?

— Reply to this email directly or view it on GitHubhttps://github.com/new-day-international/reddit/issues/22#issuecomment-23900003 .

Jim Rutt JPR Ventures

memetic007 commented 11 years ago

I think I like my second alternative better assuming it isn't too difficult. The quick loading while pondering is a nice productivity feature: Other possibility, I suppose is to upload to a temp space, and then rename the S3 key upon Submit. That also provides for a clean way to dispose of files that get uploaded but never submitted .. .they'll be in a temp dir which can be cleaned every day or two.

On Thu, Sep 5, 2013 at 4:57 PM, Jim Rutt jimrutt@gmail.com wrote:

hmmm, interesting set of issues... yeah any "multiSpace" such as MySpaces, Mod, and pseudo-spaces like Friends will have that problem.

Eliminating the File Option makes the system more modal than it needs to be. (Though I assume you are checking in the case that the desired space doesn't accept File?)

So, my initial reaction would be to:

  1. Move the Space field above the upload filed.
  2. Pop an error if user tries to select file before selecting Space
  3. Lock Space field upon starting of Upload, but unlocking on a Cancel

Other possibility, I suppose is to upload to a temp space, and then rename the S3 key upon Submit. That also provides for a clean way to dispose of files that get uploaded but never submitted .. .they'll be in a temp dir which can be cleaned every day or two.

=jim

On Thu, Sep 5, 2013 at 4:42 PM, Myers Carpenter notifications@github.comwrote:

@memetic007 https://github.com/memetic007 There's a problem with the flow if we do the files by user. If you follow the "submit new file" from the front page you end up on a page with no space selected yet. We start uploading the file as soon as the file is picked, but in this case we wouldn't know what space you wanted to post to yet.

If you followed the link from a specific space, the space field is already populated but can be changed after the file has been uploaded to a specific location.

One idea for a fix would be not have a file upload option on the front page, and once you pick to upload a file the space is locked to the one you where on when you chose to upload.

Another idea is to move the Space selection field to the top of the form. Once the file is uploaded it's fixed.

Another idea is to start uploads when you click the "submit" button.

Any other ideas for solutions?

— Reply to this email directly or view it on GitHubhttps://github.com/new-day-international/reddit/issues/22#issuecomment-23900003 .

Jim Rutt JPR Ventures

Jim Rutt JPR Ventures

myers commented 11 years ago

I'm going to leave the upload code like it is now, so temporally the file will go to /u/Username_1/filename.txt, then I'll write new code so that when the link is submitted, the file will be copied to '/space/filename.txt' and deleted from '/u/Username_1/filename.txt' (there is no rename in s3, but copy is fast because all names are just pointers). The clean up task can then just list files in the /u/ directory. Bonus feature would be we could see what users are having problems uploading just by looking at that dir.

memetic007 commented 11 years ago

Inconsistency in "Edit Submit Button Label". Probably best to eliminate that option entirely.

filesoptionalsubmitbutton

(moved this to #61)

inflector commented 11 years ago

I removed the Custom Labels for links and text posts (actually left them in but "display: none")

myers commented 11 years ago

All open issues moved to #96