Lifecycle

[andreikop]

This PR adds lifecycle configuration support.

8 hours estimation in the TODO was quite precise!

$ cat lc.xml                                            
<LifeCycleConfiguration>
  <Rule>
    <ID>DeleteAfterBecomingNonCurrent</ID>
    <Filter>
       <Prefix>logs/</Prefix>
    </Filter>
    <Status>Enabled</Status>
    <NoncurrentVersionExpiration>
      <NoncurrentDays>1</NoncurrentDays>
    </NoncurrentVersionExpiration>
  </Rule>
</LifeCycleConfiguration>
$ build/bin/s3  setlifecycle andreikop-test filename=lc.xml
$ build/bin/s3  getlifecycle andreikop-test             
<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Rule><ID>DeleteAfterBecomingNonCurrent</ID><Filter><Prefix>logs/</Prefix></Filter><Status>Enabled</Status><NoncurrentVersionExpiration><NoncurrentDays>1</NoncurrentDays></NoncurrentVersionExpiration></Rule></LifecycleConfiguration>

[bji]

Awesome work. A couple of things:

• Is it possible to represent the lifecycle document structurally instead of just passing it back as an XML blob? One of the things I tried to do when I first wrote libs3 is to make as much of the S3 services programmatically a part of the API as I could. When you return an XML blob, you put the burden of understanding its contents and operating on it on the user of the API. If the API were instead to represent the lifecycle parameters programmatically, hiding the fact that an XML blob is used by the S3 REST API as the mechanism for representing the structural data associated with lifecycle, then you make the job of the user of libs3 much easier and make this feature of Amazon S3 something that can be easily programmed.

I realize that the above represents considerably more work than just returning the blob ... but think about it :)

• Would you be interested in taking ownership of libs3? As you can tell I haven't worked on it in years. My original motivation was to create a C library that I could use programmatically as part of a FUSE filesystem on Linux to allow me to "mount" an S3 bucket as a regular Linux filesystem. I was doing this as part of a product idea I had -- I was thinking of building what at the time were "plug computer" form factor devices, that were the size of an A/C adapter and that would plug into the wall and have an ethernet cable port, that would present a networked filesystem representation of an S3 bucket, giving you essentially a local networked file store that was backed by S3. I envisioned having a certain amount of on-board disk cache in place so that copies over the network to the device would be fast (up to tens of GB, until the cache was filled), and then slowly over time the device would "sync" those files up to S3. I also envisioned adding special handling to it like the ability to drop camera photos/videos into a directory and have them get turned by the device into static web pages representing a camera album, and upload that to S3 as a photo gallery (at the time that I was considering this, S3 didn't support "web hosting" features, so there were some missing pieces that have since been added by Amazon).

I gave up on it because I reconsidered whether or not anyone would really want an "infinite capacity" local network storage device that fetched files at internet-connection speeds (since the device would have to re-download files from S3 that were no longer in its local cache before providing them to the user), and also that was costly for storing data (since at the time it was $0.13 per GB-month for S3 storage, meaning that storing 100 GB would cost $13 per month, at a time when NAS devices that would store 100 GB for free were readily available).

However, I still finished libs3 just for completeness' sake, and that's as far as I ever got with it. I used it in some of my own little personal programs for managing my own uploads to S3, but even that functionality has since been eclipsed by the Amazon aws cli command.

I don't know why anyone would use libs3 anymore, except to have a "most efficient" C based programmatic API to S3, but since libs3 doesn't even support all of S3's features, and is getting further and further behind as Amazon adds new REST API that libs3 doesn't support, I just figured it would die a slow death.

But if you find it useful and want to keep developing with it, please feel free --- it might be easier if you're the owner though so you don't have to wait for me to accept your pull requests. Would you be interested in that?

One side note -- because I am the copyright holder of libs3, I was able to re-license it under a private non-open-source license a couple of times that a few companies who were interested in using it in their products wanted me to do so. That was back when I had it under GPL, not LGPL, and so they had to get a different license for me if they wanted to use it "commercially" at all. I made some money off of those licenses (three companies licensed it for a total of $6,000) which was a nice bonus for work I had done never expecting to get paid, but nobody has asked for such a license in many years (last time was probably 2012 or so), and furthermore, I re-licensed libs3 under LGPL at one point after having been asked to do so several times by different people and finally acquiescing just to not have to deal with the requests anymore (I kind of regret doing so now because I really doing like LGPL in cases like this, but what's done is done).

So -- if you'd like to become the owner of libs3, I can add you to the copyright notice and we can figure out how to get this repository under your control ...

[andreikop]

Is it possible to represent the lifecycle document structurally instead of just passing it back as an XML blob?

ACL is expressed as C structures, but ACL is much simpler. For lifecycle there are 21 XML tag now, tags could be added in the future, there are nested structures. API will be complicated and might be not up to date.

I don't know why anyone would use libs3 anymore

You did a grate job and libs3 is very useful. We process data with C++ application because we have to use C libraries, and we use libs3 to interact with the storage.

if you'd like to become the owner of libs3

I'm not sure now how often I'll contribute to libs3. Will see in the future!

[bji]

OK then, please let me know if you ever want to become co-owner of libs3 and not have to wait for my pull request approval, I'll be happy to make that happen.