search

awslabs / amazon-sqs-java-extended-client-lib

An extension to the Amazon SQS client that enables sending and receiving messages up to 2GB via Amazon S3.
Apache License 2.0
213 stars 109 forks source link

No upgrade path from 1.0.x to 1.1.0 due to change in serialized pointer #55

Closed brettkail-wk closed 4 years ago

brettkail-wk commented 4 years ago

There is no way to incrementally upgrade from 1.0.2 to 1.1.0 because the "pointer" payload has changed from:

["com.amazon.sqs.javamessaging.MessageS3Pointer", {"s3BucketName": "...", "s3Key": "..."}]

...to:

["software.amazon.payloadoffloading.PayloadS3Pointer", {"s3BucketName": "...", "s3Key": "..."}]

The former no longer exists in 1.1.0, and the latter never existed in 1.0.x. That means I can't upgrade my sender to 1.1.0 because it will send PayloadS3Pointer, which the receiver will not recognize, and I can't upgrade my receiver to 1.1.0 because senders are still sending MessageS3Pointer, which will no longer be recognized. The only option is to upgrade all senders and receivers at the same time, which is a non-starter for our services. What is the recommended upgrade path for users of amazon-sqs-java-extended-client-lib?

There are various tricks with @JsonTypeInfo(use=Id.NAME, include=As.WRAPPER_ARRAY), @JsonTypeName, and ObjectMapper.registerSubtypes that could be used to control which names can be read/written using Jackson, but it might be easiest to just read/write raw JSON instead. At this point, you probably don't want to just use those mechanisms to revert to the old type since that would similarly break anyone now using PayloadS3Pointer, so perhaps it would be best to add ExtendedClientConfiguration.useLegacyPointerType that causes the client to write the old MessageS3Pointer type in the JSON, and then update JsonDataConverter to support either the old or new type. Projects using 1.0.x would still be broken when upgrading to 1.1.0, but they can at least remain compatible by opting into the compatibility boolean.

adam-aws commented 4 years ago

Hi @brettkail-wk, would simply adding a check in receiveMessage to change any "com.amazon.sqs.javamessaging.MessageS3Pointer" to software.amazon.payloadoffloading.PayloadS3Pointer allowing it to be processed fix this issue (PR #56)? You would first upgrade receivers, they would be able to handle calls from MessageS3Pointer and PayloadS3Pointer, afterwards you could upgrade the senders to 1.1.x

Tested it out with sender using 1.0.2 and receiver using 1.1.x and it's able to receive messages.

brettkail-wk commented 4 years ago

@adam-aws No, because we use request+reply queues, so all processes are simultaneously sender+receiver, so there's no good way to just upgrade the receivers. I believe that approach could work if a 1.0.3 release was made with something like that (i.e., a version that still sends MessageS3Pointer but can receive either). Then, we could upgrade all processes to 1.0.3 first and then upgrade all to 1.1.0. If you go forward with that, I would suggest adding a note to the README that 1.1.0 is backwards incompatible, and then list the upgrade steps.

adam-aws commented 4 years ago

Released v1.0.3 to be able to have a way to upgrade to 1.2.0