ShokoAnime / ShokoServer

Repository for Shoko Server.
http://shokoanime.com/shoko-server/
MIT License
389 stars 75 forks source link

Cloud Support (Amazon Unlimited Cloud) #171

Closed maxpiva closed 7 years ago

maxpiva commented 8 years ago

I just got, a free one year, for Amazon Cloud Unlimited, which is... really unlimited (Datahoarders in reddit users confirms, 25 TB of movies and series mounting Amazon Cloud with NETDRIVE using plex server ;) . I just started uploading my anime collection. Seems nice to use JMM, and stream from the cloud directly. And we can kiss goodbye our Raids :P

Since JMM uses ED2K hashes to identify via ANIDB the correct anime, I did some research, and seems You can get the MD5 from the amazon cloud, without reading the whole file, as is, JMM uses ED2K since ANIDB uses ED2K, but, ANIDB also stores MD5 and we can store MD5, if enabled.

So the easy way.... Ask Anidb dewds to support FILE searching via MD5. If not we can do some kind of eD2K cross MD5....

I'm currently refactoring JMM Server You can check the progress in New Database Branch of my repo, some features are, blazing speed, and multi-user support (anidb). But, i'll happy to add an Amazon Cloud Unlimited Filesys to JMM to handle this.

CuddleBear92 commented 8 years ago

since you are using it, how is the experience for you and such ATM? does it has its own local pc client (cant seem to find anything about it) other than ODrive (which looks fine TBH). is pausing mid file upload a thing on their service? im just thinking about this as a user standpoint, ATM im using Jottacloud (which is also unlimited (so far at 2.8TB)) (i would also trust their security to be better tho because of the Norwegian laws of Jotta) is the download speeds (yes down from the cloud) any good from day to day? is it hitting you max speed no matter when it is? (sadly Jotta is a bit lacking there for me 24/7.... usually around 5MB/s when i can handle around 11 :/)

i love the idea of uploading everything to the cloud..... i keep running out of damn space!

sorry for the slightly offtopic talk, as a service it sounds interesting for sure and if it could be worked into JMM it would be great, also when talking about clouds (more like a seedbox) put.io does handle CRC32 hashes on downloads and does support streaming when codecs and fileformat is supported (mp4s (also supports converting it on the site for the stream aswell)) ofc a seedbox like thing isnt for everyone tho, also sad to see jolicloud does not support amazon just yet.... might have to shout at them about it XD

maxpiva commented 8 years ago

Amazon Cloud Desktop for uploading. Seems faster than netdrive. You can mount the whole cloud as a disk with NETDRIVE. And copy files, read and write like an HD. I tested directly playing with Windows Media Player, on some bluray rips, and it start with a second hiccup, then play perfectly. Seems people are already Doing AMAZON CLOUD <-> NETDRIVE <-> PLEX, with a very good perfomance. Up to 4 people playing at the same time. But ofc, it might be better or worst on some places. The main problem with this setup and JMM, is that JMM hashes the files (it could ask to download all the files), and downloading your entire library for hashing is nono. I'm using netdrive, since it mount the cloud, without syncing the whole thing like odrive.

Btw, you can test there is 3 month free unlimited in amazon.

CuddleBear92 commented 8 years ago

sounds interesting, might switch to Amazon at the end of the year then when my Jotta account run out. also, thanks for telling me about netdrive. looks useful in many ways. i will deff start using this fulltime if JMM has support for it for sure. great way to save local space on those 720p/1080p BD rips (and i can finally get the 1080 versions because of all the space we save locally.) you say that you have tested it with WMP, so does things like MPC/VLC work, i know MPC doesnt like streaming when used with programs like livestreamer (for twitch.tv) and other streaming services.

jmediamanager commented 8 years ago

Sounds pretty interesting, although would be pretty much useless to anyone in Australia (like me) since we have very limited bandwidth quotas. Although once I had uploaded my collection (which would take around a year) it could be feasible.

I guess what could be interesting about this is that you could possible share collections with other users. But all sorts of legal issues once you start uploading and sharing.

In regards to using MD5's, how unique are they compared to E2DK's?

CuddleBear92 commented 8 years ago

sharing would be alot easier for sure as alot of the cloud sites has the copy to your own cloud button, and with services like jolicloud you could move it to another cloud that you pay for. i can deff see that this will get more common, i even see some streaming sites uses googledrive as its host as thats the youtube stream aswell.

even with people that have slow bandwidth for uploads and just wants to stream to watch it would get alot simpler over year IMO. i for one would love to be able to share my collection as a hole for the JMM community.

also @maxpiva would netdrive work with things like plex aswell? have you tried the smoothness of that?

and again.... sorry for the offtopics..... i will just hide in a corner now XD

maxpiva commented 8 years ago

Well, i read the EULA, if you share Copyright work, you're in hot water.... But, if you don't you seems to be ok. btw. The main problem with this is deduplication. After all who dosn't have a ripped cd with mp3.

Amazon uses deduplication, donno if they use it only for your work, or for all files in the service (make sense). also i dont know which program uses that capability, but deduplication means, if you try to upload a file, that already exists in the cloud, you skip the upload part, the file just appear in your cloud, without uploading it.

I guess they use MD5 and Filesize to deduplicate. Like anidb uses ED2K and FileSize.

Leo, I think ed2k and MD5 has similar collision probablity, the combo MD5+FileSize is almost unclonable.

But, deduplication, may get our files deleted in the future... user A has the same file than user B user A shares the file. user A got a copyright infrigment. user A got the file deleted, but also user B, because of deduplication.

That already happens of dropbox. Never happens on Amazon, but who knows.

So the benefits are, it can actually take less time to upload your collection, depending the archives already in the cloud. But you could end with some files deleted.

There is a workaround Stablebit CloudDrive. Own encrypted/ornot filesystem inside the cloud, they just use a bigfile like sectors of the disk. You can format the drive with the filesystem you want...

Not tried plex yet. only read users in reddit with success stories.

jmediamanager commented 8 years ago

Yeah deduplication is what I assumed they would use. Most could services would use it. That is definitely a worry if all your files suddenly got DCMA take-downs, and your collection is gone :(

maxpiva commented 7 years ago

Amazon Cloud, Google Drive and OneDrive are done deal.