search

cyclopsgroup / cym2

CyclopsGroup Maven plugins and extensions
Apache License 2.0
5 stars 1 forks source link

update new files only? #1

Closed carrot-garden closed 2 years ago

carrot-garden commented 12 years ago

Jiaqi:

I am curious if your plugin does update new files only?

in other words, if source & target file has same time stamp, will your wagon run upload anyway?

thanks

Andrei.

jiaqi commented 12 years ago

Reading code in https://github.com/jiaqi/cym2/blob/master/awss3/src/main/java/org/cyclopsgroup/cym2/awss3/S3Wagon.java, which I completely forgot, I don't think it checks timestamp. Though I am not 100% sure about the base class.

I often make a few file changes and call wagon to upload entire site to S3. According to logging, it does look like maven iterates through every file no matter it's changed or not and uploads all. There's noticeable delay during upload of each file. So I believe it always uploads all files without looking at timestamp.

Regards Jiaqi Guo

carrot-garden commented 12 years ago

ok; I guess I have to steal your code an play with it a bit to see if I can convince maven/wagon get smart :-)

jiaqi commented 12 years ago

Good luck on that. There are several gotchas I want to bring out, which you may already know.

  1. Don't easily trust timestamp. Time on each OS may drift or may not make sense at all. If you want it to be selective and checking last modified time, make it optional so that user can still choose to upload everything when they want. Checksum checking is much better for this matter comparing to last update timestamp.
  2. I think AWS Java SDK has a smart putObject method, which takes File as input and does some checking and preparation that it can't do for InputStream. I'm not sure how much optimization it does though.
  3. The current wagon class extends StreamWagon, which may not give you much room to improve. To be really smart, you probably have to throw away StreamWagon and do a lot of work by yourself.

Regards Jiaqi Guo

carrot-garden commented 12 years ago

great ideas, thank you for sharing;

I will keep you posted if I come up with a solution;