search

boncey / Flickr4Java

Java API For Flickr. Fork of FlickrJ
BSD 2-Clause "Simplified" License
176 stars 154 forks source link

Unable to upload large files #36

Open kipz opened 11 years ago

kipz commented 11 years ago

When uploading large files, a stream is passed to the API:

Flickr.getUploader().upload(InputStream, params)

However, REST.java buffers everything in to a ByteArrayOutputStream before passing it to the scribe OAuth library. Therefore, uploading large files can often cause out of memory errors (depends on the size of file and heap of course).

Scribe OAuth (as far as I know) doesn't support any streaming yet, so we could either create our own implementation, swap out for an implementation that does support this, or ask and wait for those guys to fix theirs.

I've opened an issue on Scribe related to this:

https://github.com/fernandezpablo85/scribe-java/issues/347

kipz commented 11 years ago

I forgot to mention, that this currently prevents me from uploading videos (MOVs) over 100MB with a default heap with Oracle JRE 7 on Mac.

boncey commented 11 years ago

I don't use Flickr4Java for video (yet) so I've not seen this.

I guess we can see what the Scribe folks say as that would be the best solution I think. If they request a patch is that something you'd consider doing?

kipz commented 11 years ago

I'll have a go, but to be honest, doing all that on top of HTTPConnection will be a bit of a bore. Ideally they would integrate Apache HTTP client (or similar) which would give much of it for free.

boncey commented 11 years ago

Good point. This pull request looks interesting, wonder if it will get accepted. https://github.com/fernandezpablo85/scribe-java/pull/250

kipz commented 11 years ago

Let's hope so, but it's looking pretty quiet over there at the moment...