search

ipfs / go-ds-s3

An s3 datastore implementation
MIT License
236 stars 66 forks source link

Reduce library size #81

Open ianopolous opened 4 years ago

ianopolous commented 4 years ago

The latest version of this has almost doubled the size to 60mb. This is way too much for a glorified rest client. For comparison the Java S3 client is ~3mb. Is there any way to reduce this by stripping out unused dependencies? Or just including the S3 sdk, and not all of amazon?

Stebalien commented 4 years ago

Is that compiled size or source code size? We're using amazon's s3 sdk.

ianopolous commented 4 years ago

Compiled size. I've also opened an issue on amazons go sdk about importing only S3 client and dependencies. I'm also referring to Amazon's S3 client. I think the problem here is it's including clients for all of amazon's ~50 services. Not sure why Go isn't stripping out unused components though during compilation.

ianopolous commented 4 years ago

Amazon have replied and they are already working it. Tracking issue here: https://github.com/aws/aws-sdk-go-v2/issues/444

So I think it's mostly just a matter of waiting for that to be released and importing that sdk instead.

ianopolous commented 4 years ago

@Stebalien Is there a size where would you consider including it in ipfs proper, like 3mb say? Or is even that too much to make every user pay. If only we could store the ipfs binary in ipld proper and selectively download the merkle linked modules on demand. :-)

Stebalien commented 4 years ago

I'll have to think about it. Honestly, go-ipfs is already pretty massive so 3MiB might be acceptable.

We may just want to ship multiple IPFS versions, one with additional supported datastores. Plugins were supposed to solve this issue, but they just don't work reliably.

ianopolous commented 4 years ago

A better solution to this is to just implement our own S3 request signer. It is quite simple (<300 lines of Java). My Java implementation is here: https://github.com/Peergos/Peergos/blob/master/src/peergos/server/storage/S3Request.java

Then this should be < 1 KiB not 60 MiB, and thus could be included in ipfs directly without requiring a plugin.

ianopolous commented 4 years ago

I've checked that I had implemented all the calls used by this library, and I'd missed bulk delete. So I've added that and the result is still ~ 500 lines of very simple code. Zero dependencies. This would make a good small dev grant target, @momack2.