search

p0ody / ff2ebook

WIP.
http://www.ff2ebook.com
18 stars 2 forks source link

Archive #35

Closed dano1o closed 3 years ago

dano1o commented 3 years ago

Some stoires that used to be in the archive are gone. Is there anyway for them to be recovered?

dano1o commented 3 years ago

So no then?

Multipanda commented 3 years ago

I've noticed the same thing, not sure it's much of an archive if it doesn't actually keep archived deleted stories.

I've seen more than one reddit thread talking about a deleted fic with link til ff2ebook with the archived story no longer being available. And one author where I backed up multiple deleted stories no longer have them, weirdly enough it only shows one story under his name and it's one that is currently online, and it hasn't archived his 13 other stories.

So it seems like the site isn't actually actively archiving stuff at all, or at least not doing it consistently.

p0ody commented 3 years ago

The thing is that I am only allowed 100gb of storage with my web host, so once in a while, when I hit over 100gb I have to do a clean up.

I run a script that delete fics that have not been downloaded lately.

I know this is not exactly the best for an archive, but to begin with I kept the files mostly to use as a cache and not have to redownload everything everytime, the archive thing was a bonus.

Sorry for the inconvenience, but there is no way to get the fic back.

Multipanda commented 3 years ago

It's really terrible, honestly, is there no way you can increase the storage? Maybe work on aggressively compressing everything and changing formats to something that takes less room. I remember someone uploading a siterip of FFnet a few years ago where they fit the entire site in a little more than 100GB

Worst case you should save them offline, and then upload them elsewhere somehow, or just store them until the day you have a solution with more online storage.

I get that storage isn't free, but you can't call it an archive at all with that approach, quite frankly, because it means users really can't trust the service.

And when you make it a popularity contest, it means that those fics that needs it the most won't be saved longer term. Instead it will only be recent fics and the fics that are most likely to be reuploaded elsewhere anyway. There's already a bunch of reddit threads with dead links, which is kinda sad.

I think it's something you should state on your site, so people doesn't rely on it more than they should. If it is a monetary concern you could also state that, and maybe have as a donation goal an actual permanent archive, which I think is what's needed more than anything. There are plenty of ways to create ebooks from sites, whereas backup archives are in short supply.

I'm probably being a bit too harsh, but I feel like if you call something an archive, you have a bit of a responsibility to live up to that. :-)

bastien8060 commented 3 years ago

I remember someone fit 400GB worth of eBooks in about 120GB (the whole archive back in 2015) by storing it in a zip file. It may be a solution, but it would be very vulnerable to DDoS/dos attacks, if the server were to decompress the file from the huge archive every time. Also, they didn't store it as an eBook which is the purpose of FF2ebook. However, they only just stored in a .txt, without the necessary data structure to retrieve it as an eBook afterwards.

bastien8060 commented 3 years ago

Also guys, I am not dropping the original idea of making FanFiction.net work. My older solution worked before but stopped working since Cloudflare was updated once again.

Right now, I am reverse engineering FanFiction.net Android App. I found all the API's backends already which were encrypted in the app. I'm on a good lead so far.

The api communication to the server are encrypted in Aes128, with a special key hidden in the binary /data/data/com.fictionpress.fanfiction/.  (with the spaces at the end). The file is only stored for a fraction of a second on startup.

Right now, I just need to figure which lines deletes the file. Then I will prevent it from being deleted, retrieve the file with the 16bytes encryption key inside. This will get me to snoop on the traffic, the app makes with the server and get the Api key too.

When all that is done, I'll have the Encryption key, The Api key, and I can make FF2ebook communicate to FanFiction.net via the API, pretending to be the android app. This would be even faster than originally as they really care about their android users, i've heard.

They probably won't counter attack, because:

Edit: Api endpoint's are:

Visiting from a browser won't work because the http handshake/communication will fail as its not encrypted with the "Encryption key"

Multipanda commented 3 years ago

Well, literally any solution is better than deleting fics. What about on-the-fly conversion to ebook format, rather than storing in it? Though if they were just agressively enough compressed the ebooks format could still work, and as a stopgap measure, maybe having it in only epub or mobi, rather than both, whichever takes up less space.

StarWolf3000 commented 3 years ago

On-the-fly conversion would require to store fics in another form before conversion (e.q. as text in the database), which may even need more space than storing as epub. Also on-the-fly conversion from FF.net would not work for deleted works.

p0ody commented 3 years ago

I found an old backup from Nov 2019 on my PC. If you wanna look into finding a specific fic. You will need the fic ID though.

filename are as follow site_ficID_unixTimestamp.epub I created a torrent (55GB). Let me know if this doesn't work, i've never created a torrent before lol. Torrent link or copy into you address bar http://ff2ebook.com/torrent/ff2ebook-archive-2019-11-23-fix.torrent if the direct link doesn't work.

bastien8060 commented 3 years ago

Thats awesome, I checked it works! I would only recommand to split the archive, if one only wants one ebook or two.Also the link for some reasons doesn't want to download on chrome. If it helps, I used wget http://ff2ebook.com/torrent/ff2ebook-archive-2019-11-23.torrent in the terminal.

Edit: or just use another browser

bastien8060 commented 3 years ago

Btw guys, about FF.net, I've found bunch of passwords (api keys?) in the FanFiction.net app's code that I didn't see! They reside in the code that gets the story before the request(s) is/are encrypted.

You can see here by yourself (https://ibb.co/vkHffcz).

I've found now a 16bytes encryption key (correct length for AES128, which is what the app uses to communicate to the server and get the stories!). In Hex form, it gives (E357082D4774D2DDD62298988075D864). It leaves to figure which Api key is used, the IV (almost done), and the GCM tag.

Edit: few lines below, there are some error codes ("key length is wrong", "Encrypt Message is wrong", "cypher error"), which hints that the keys may be used for encryption/decryption, or some kind like that. Lower again, I can see the use of AES, so again, sure.

Also if you think I should talk in the other issue thread, tell me. I just thought of replying where people were already

PenguinPaws commented 3 years ago

I found an old backup from Nov 2019 on my PC. If you wanna look into finding a specific fic. You will need the fic ID though.

filename are as follow site_ficID_unixTimestamp.epub I created a torrent (55GB). Let me know if this doesn't work, i've never created a torrent before lol. Torrent link or copy into you address bar http://ff2ebook.com/torrent/ff2ebook-archive-2019-11-23.torrent if the direct link doesn't work.

Do you happen to remember how you created that zip file originally? No matter what I do, I can't extract it. WinRAR v5.61, 7-Zip v19.00 (2019-02-21), and 7-Zip v21.00 alpha (2021-01-19) all can list the contents, but throw an error of unsupported compression method for each of the files inside. :/

Undercover487 commented 3 years ago

Do you happen to remember how you created that zip file originally? No matter what I do, I can't extract it. WinRAR v5.61, 7-Zip v19.00 (2019-02-21), and 7-Zip v21.00 alpha (2021-01-19) all can list the contents, but throw an error of unsupported compression method for each of the files inside. :/

Hey mate, where you able to unzip the file?

p0ody commented 3 years ago

Oh. i realized that i used 7zip ZS instead of regular 7Zip (I dont remember why i had that installed). Im currently rearchiving it, So either you wait for the new version to redownload or you can get 7zip ZS here https://github.com/mcmilk/7-Zip-zstd/releases/tag/19.00-v1.4.8-R1

PenguinPaws commented 3 years ago

Thank you. I was able to extract it after installing 7zip ZS.

Just a FYI for everyone - make sure you have ~88GB free to extract the archive file.

p0ody commented 3 years ago

Sorry for the delay, but the new torrent is ready with a proper 7zip archive

http://ff2ebook.com/torrent/ff2ebook-archive-2019-11-23-fix.torrent

PhoenixFireOwl commented 3 years ago

Unable to download archive. Torrent shows no seeding? Please help. Thank you.

p0ody commented 3 years ago

Sorry, my torrent client was closed after a windows update

PhoenixFireOwl commented 3 years ago

Im trying to download it again. Would you please let me know if and when it's available, again, please? Thank you so much for saving these!

PenguinPaws commented 3 years ago

Im trying to download it again. Would you please let me know if and when it's available, again, please? Thank you so much for saving these!

There's 2 torent links. Which one are you downloading from?

PhoenixFireOwl commented 3 years ago

I was able to download it. However, the frustrating thing is that you have to have the story ID to look up the correct epub or mobi file. It's too bad that they weren't saved with the actual story name and author like it does when each is downloaded from the site. I'm also looking for stories no longer found on fanfiction.net but would hopefully be on the site (or now in this archive).

Still good that this archive is available. I appreciate it. Thank you!

p0ody commented 3 years ago

Check the archive on www.ff2ebook.com, I added a bunch of the old archive back on there