Ask for help about distributed transaction based on Hystrix fallback

[Mavlarn]

I have a question about how should I use hystrix fallback to implement distributed transaction. I asked a question in stackoverfow but got no answer, so I ask here. And this is my question:

I am using spring cloud to implement my micro services system, a ticket sale platform. The scenario is, there is a zuul proxy, a eureka registry, and 3 service: user service, order service and ticket service. Services use feign declarative REST Client to communicate with each other.

Now there is a function to buy tickets, the main process is as below:

1. order service accepts request to create order
2. order service creates Order entity with Pending status.
3. order service calls user service to process user pay.
4. order service calls ticket service to update user tickets.
5. order service updates the order entity as FINISHED.

And I want to use Hystrix Fallback to implement transaction. For example, if the payment process is finished, but some error happened during ticket movement. How to revet user payment, and order status. Because user payment is in other service.

The following is my current solution, I am not sure whether it is proper. Or is there any other better way to do that.

At first, the OrderResource:

@RestController
@RequestMapping("/api/order")
public class OrderResource {

  @HystrixCommand(fallbackMethod = "createFallback")
  @PostMapping(value = "/")
  public Order create(@RequestBody Order order) {
    return orderService.create(order);
  }

  private Order createFallback(Order order) {
    return orderService.createFallback(order);
  }
}

Then the OrderService:

@Service
public class OrderService {

    @Transactional
    public Order create(Order order) {
        order.setStatus("PENDING");
        order = orderRepository.save(order);

        UserPayDTO payDTO = new UserPayDTO();
        userCompositeService.payForOrder(payDTO);

        order.setStatus("PAID");
        order = orderRepository.save(order);

        ticketCompositeService.moveTickets(ticketIds, currentUserId);

        order.setStatus("FINISHED");
        order = orderRepository.save(order);
        return order;
    }

    @Transactional
    public Order createFallback(Order order) {
        // order is the object processed in create(), there is Transaction in create(), so saving order will be rollback,
        // but the order instance still exist.
        if (order.getId() == null) { // order not saved even.
            return null;
        }
        UserPayDTO payDTO = new UserPayDTO();
        try {
            if (order.getStatus() == "FINISHED") { // order finished, must be paid and ticket moved
                userCompositeService.payForOrderFallback(payDTO);
                ticketCompositeService.moveTicketsFallback(getTicketIdList(order.getTicketIds()), currentUserId);
            } else if (order.getStatus() == "PAID") { // is paid, but not sure whether has error during ticket movement.
                userCompositeService.payForOrderFallback(payDTO);
                ticketCompositeService.moveTicketsFallback(getTicketIdList(order.getTicketIds()), currentUserId);
            } else if (order.getStatus() == "PENDING") { // maybe have error during payment.
                userCompositeService.payForOrderFallback(payDTO);
            }
        } catch (Exception e) {
            LOG.error(e.getMessage(), e);
        }

        order.setStatus("FAILED");
        orderRepository.save(order); // order saving is rollbacked during create(), I save it here to trace the failed orders.
        return order;
    }
}

Some key points here are:

• Using @HystrixCommand in OrderResource.create(order) method, with fallback function.
• If there is some error in creation, the order instance used in OrderResource.create(order) will be used again in fallback function. Although the persistence of this order will be roll-backed. But the data in this instance still can be used to check the running.
• So I use a status: 'PENDING', 'PAID', 'FINISHED' to check whether some service call is made. ticketCompositeService and userCompositeService is a feign client. For feign client method payForOrder(), there is another method payForOrderFallback() for fallback.
• I need to make sure the fallback methods can be called multiple times, for re-check or timeout.
• I add try/catch for ticketCompositeService and userCompositeService call, to make sure the order will be save anyway with 'FAILED' status.

It seems that this solution can work at the most of the time. Except that, in fallback function, if there is some error in userCompositeService.payForOrderFallback(payDTO);, then the following composite service call will not be called.

And, another problem is, I think it is too complicated.

So, for this scenario, how should I implement dist transaction properly and effectively. Any suggestion or advice will help. Thanks.

[mattrjacobs]

I don't think I would use Hystrix and a transaction together. I have no experience doing so - if others do please chime in.

My reasoning is that there are many ways a fallback can get triggered. Handling a transaction semantic failure, timeout, short-circuit, etc with the same fallback logic sounds like adding extra complexity.

Personally, I would fine a way to separate the transaction aspect into one layer and then wrap that layer with Hystrix (if Hystrix is necessary for resilience purposes)

[Mavlarn]

yes I also find it is not proper to implement dist transaction with fallback.

So, based on this scenario:

1. accept request in order service, and create order.
2. call user service to charge
3. call ticket service to move tickets. (when this part has error, but user charge has been finished.)

How should I fix this?

I know in some cases, I can use MQ to drive the process. But I don't want to use MQ for now.

[bltb]

This seems like a general architectural question and not related to Hystrix.

From a system architecture perspective, if possible, IMHO and YMMV, I would avoid "distributed transactions" / "two phase commits" all together - and I mean, avoid like the plague... :running_man: .

I would ask myself...

• Do I really need distributed transactions?
• Do I really need Hystrix?
• If I really have a truly distributed system, why am I avoiding using messages?

P.S. If you want to experience some really funky system behaviour wrap Hystrix commands inside Hystrix commands and then throw one or two Spring @Transactional annotations into your code. :smile:

[bltb]

@Mavlarn , I notice that you are actually using JMS already...

https://github.com/Mavlarn/spring-cloud-dist-transaction-tutorial/tree/master/jms-jta-service/src/main

Why not just create a listener for a message called "reverse order / charge" or something?

[Mavlarn]

@bltb that repo is just a learning project. In my current product, we use spring cloud without mq, and not considering distributed transaction. I just avoid that by designing business process properly.

But now, I have a function as above: 'buying tickets', interacting with 3 other services. So I need to make sure if there is an error in the last service calling, I should rollback the previous service process.

At the beginning, I thought the Hystrix fallback can do the 'rollback' job. But as discussed above, it seems that it is not a good solution.