File expiry

[ultramancool]

Requested by niols from IRC.

Make files expire after a user-selectable (maybe with a low default to save on server space) period of time.

We could do this by just deleting expired files on access and/or via cron job. Will need to store the expiry date/time somehow though.

[andre-d]

I discussed this with @k3d3 the other day. At the worst if we want to avoid having a DB we could just store it in a folder symlinked with the expiry date. So the directory 2015-08-08 with a symlink to the blob inside it. The cron job just has to enter that folder and hard delete the file linked.

[ultramancool]

Yeah, that only works for daily expires and doesn't achieve much precision though. Will discuss later.

[k3d3]

Well, it doesn't have to be daily; it can be a week in the future - if today is 2015-08-01 then you make a directory called 2015-08-08, and make a symlink in that directory. Then, come 2015-08-08, just delete the actual files that are symlinked in that directory. It does, however, mean that you won't achieve any precision beyond one day.

In any case, we should have some solution where figuring out what files need to be deleted doesn't require scanning a directory, and instead can work by just directly accessing something (In this case, a 2015-08-08 directory)

[BetterToAutomateTheWorld]

Hi,

I am new here and I never used Up1 in the past, your project hightly interrest me but this file expiration feature is unfortunately in my point of view, a lack for Up1.

So, i agreed with this feature if somebody can implement it I will enjoy to use Up1 instead of Zerobin (& others) for my paste/file/img/video upload :)

[xeverse]

This is essential feature. First thing to implement ever. Of course with super secure deletion and such.

[ultramancool]

xeverse, care to explain your use case and threat model in which it would be substantially useful and which other more basic precautions would not protect from?

This is essential feature. First thing to implement ever.

— Reply to this email directly or view it on GitHub https://github.com/Upload/Up1/issues/19#issuecomment-184785123.

[xeverse]

care to explain your use case and threat model in which it would be substantially useful

some information should only exist for a limited specified amount of time in this universe. in the sad age of corp-gov mass surveilance and log it all forever data retention policy all those keys are harvested and kept by whoever controls the nodes or mail services. for security and privacy protection it is essential to securely destroy the information as soon as it is no longer needed or expired. information ecology. no ip logging too of course.

[andre-d]

some information should only exist for a limited specified amount of time in this universe. in the age of corp-gov mass surveilance and log it all forever data retention policy all those keys are harvested and kept by whoever controls the nodes or mail services.

I totally understand, but if anyone harvested your key they also equally could have harvested the data you uploaded and deleting the data from our server does nothing for this use case.

[xeverse]

but if anyone harvested your key they also equally could have harvested the data you uploaded

in case of these massive webmail providers they just keep the data, i.e the up1 image link in the mail. the link is not necessarily followed until they get subpoena or specific data request. but it should have been gone by then. this timeframe between actual logging and request for analysis should be a safety window.

[ultramancool]

Even if we were to delete it (which you can trigger manually, just not automatically after X days), you're still relying on the security of many of the same algorithms to prevent the decryption of data those same government agencies could have captured over the wire, if they can break those algorithms, SSL and Tor fall too.

So, if 3 letter agencies can already log your SSL or Tor traffic and perform later attacks on that - does offering deletion really get us or our users anything? On Feb 16, 2016 12:54 PM, "xeverse" notifications@github.com wrote:

care to explain your use case and threat model in which it would be substantially useful some information should only exist for a limited specified amount of time in this universe. in the age of corp-gov mass surveilance and log it all forever data retention policy all those keys are harvested and kept by whoever controls the nodes or mail services. for security and privacy protection it is essential to securely destroy the information as soon as it is no longer needed or expired. information ecology.

no ip logging too of course.

— Reply to this email directly or view it on GitHub https://github.com/Upload/Up1/issues/19#issuecomment-184799811.

[andre-d]

You referred to this feature request as the "First thing to implement ever". You believe our first priority should be ensuring that in case someone subpoenas your email order to retrieve a copy of your key, the data is already deleted? Why not only send your key over a medium which will not be stored and subpoenad in the first place?

[ultramancool]

Yeah if you're sending Up1 links over an unsecured medium like email you're already screwed, Up1 only provides any level of security when a secure channel has already been established. On Feb 16, 2016 1:06 PM, "Andre d" notifications@github.com wrote:

You referred to this feature request as the "First thing to implement ever". You believe our first priority should be ensuring that in case someone subpoenas your email order to retrieve a copy of your key, the data is already deleted? Why not only send your key over a medium which will not be stored and subpoenad in the first place?

— Reply to this email directly or view it on GitHub https://github.com/Upload/Up1/issues/19#issuecomment-184804015.

[xeverse]

@ultramancool

which you can trigger manually

this isn't cool or feasible sometimes.

you're still relying on the security of many of the same algorithms

This is true. in case of data channel security but not emails.

@andre-d

Why not only send your key over a medium which will not be stored and subpoenad in the first place?

Right. That would be best case scenario. Now always possible tho. Majority of people we have to communicate with are braindead about information security or even basic network privacy issues. They are gonna use just fgmail no matter what.

maybe it's just me. cypherpunkish. but client side encrypted hosting is cypherpunk. isn't it? protecting privacy in this scene is a must.

[andre-d]

protecting privacy in this scene is a must.

We can not protect the privacy of people who are sending keys over a non-secure medium, they have lost that privacy as soon as they did that and it is impossible to resolve that issue with hacks to shred things just in case someone only read the data after the fact.

[xeverse]

ok. at least i've communicated my point of view.

the ultimate occult reason is that some information affects the cause and effect in the universe (chaos theory thing) as long as it exists. so must be destroyed asap if no longer needed..

[andre-d]

some information affects the cause and effect in the universe (chaos theory thing) as long as it exists. so must be destroyed asap.

Emailing your key is as good as emailing the data itself in plaintext, if you are indeed doing that with such disastrous information then I suggest you rethink your delivery medium ASAP.

[xeverse]

The link and the key is metadata. Which they keep forever these days. I.e when they attacked me my isp confessed to me that kept all metadata and happily shared it with them when they requested it. The actual content it refers to is investigated if necessary.

[ultramancool]

xeverse - they also keep full data to and from encrypted services like Tor, SSL and of course, Up1 itself, not just metadata.

[xeverse]

Maybe, not always i hope, yet. That would be trully unfortunate state of affairs.

[ultramancool]

This is why we explicitly warn against sending Up1 links over insecure mediums. Do not do it. Anyone can grab it, ISPs anywhere along the line have access as well as anyone who hacks any of them, not just 3 letter agencies.

[k3d3]

The best use for Up1 is when you have a secure communications channel set up already (e.g. self-hosted XMPP, Signal, OTR, etc.) but it's not feasible to share files or images over that communications channel.

It does not secure the key when sent over an insecure channel such as email, and file expiry does not change this.

There are legitimate reasons why you'd want file expiry (for example, keeping the file count low on personal servers, preventing malware from grabbing keys in the future, or simply keeping a defense-in-depth approach and not keeping files around for longer than you need them), and for those reasons we'll still implement it in the future, however your concern isn't solved by file expiry.

[xeverse]

Good point. The clearnet is dead for human rights advocates or hacktivists. No one could have imagined back in the days that global control freaks and fascist states would turn our tech agains us as a tool of control and surveilance. Geeks were happy it just worked. Now we have to redesign everything form scratch all network infrastructure all isdn protocols with privacy in mind.

[andre-d]

Please refrain from off topic political discussions on the issue tracker

[xeverse]

@k3d3

It does not secure the key when sent over an insecure channel such as email, and file expiry does not change this.

Hmmm. Data retention laws and nasty gag orders i guess. https://www.eff.org/deeplinks/2014/04/warrant-canary-faq

[xeverse]

@andre-d .Ok. Thanks for the chat.

[tristan-k]

Any progress on this matter?

[gangelop]

One approach for easy expiry would be an optional redis backend. hastebin is a simple and similar node app which does that. You configure a global expiration time (TTL) of 30 days from last access and redis handles it. This is a good solution for an administrator who just wants to manage storage space.

[k3d3]

We actually had a predecessor to Up1 (though privately) that was based on hastebin. The big difference was that we added some client-side javascript to decrypt with a key passed along in the anchor, so we actually know hastebin pretty well (and it's great for what it does!).

With that being said, I'm fairly hesitant on adding server-side dependencies like redis to Up1, because it makes it more complicated to set up.

I've written up some ideas on how to solve this in issue #58, if you're interested in taking a look and commenting on it.