search

codiak / socialarchive-dapp

Dapp built on Swarm to back-up and interact with Twitter data.
GNU General Public License v3.0
2 stars 3 forks source link

Performance of upload degraded #56

Open ameer-clara opened 2 years ago

ameer-clara commented 2 years ago

compared to the version on 1/22 299c7d99

kirlen commented 2 years ago

cause of the "slowdown" is https://github.com/codiak/socialarchive-dapp/commit/cc2fb28bc8bc0cca14ec46cd5300c1e07688c3ee: if there was an archive cached in IDB, code was no longer actually processing the zip provided by the user. I.e. we were seeing (and believing) false instantaneousness!

Nevertheless, the current processing time feels quite long. The main culprit is the fact that we need to download any additional referenced media that's not already stored in the twitter archive in order to include it in our archive. @ameer-clara, I think you're under the impression that most media is already stored in the twitter archive -- that's not what i'm seeing. @codiak?

In any case, we probably should explore both additional optimizations and better progress display. I had a friend of mine try it with his ~120Mb twitter archive, and the additional media roughly doubled the size (not to mention taking far too long with supplemental feedback).

One idea for maybe a better user experience: defer download of the media URLs until upload time, so user can immediately browse their archive before uploading it. Yes, the upload will take longer, but user can just walk away at that point. Thoughts?