Non-shipped items adding costs

[clarknelson]

Description We sell two types of products on our site, physical products and classes attended in person. We noticed that when you add classes to the cart, it increases the UPS costs, even though it doesn't add any weight to the package. I'm not entirely sure how to debug something like this so I hope you can help somehow.

Steps to reproduce

1. Two product types: One without variants / weight, one with weight
2. Add both to cart, modifying the value of the "weightless" product should adjust the shipping costs

Additional info

• Plugin version: 2.4.15
• Craft version: 3.7.28
• Multi-site: No

[engram-design]

For the moment, I would select the Free Shipping option on every classes product, which will set the shippable value of that product to 0. Any other shippable products in the cart will be charged as normal, so you're essentially zeroing out any non-shippable items.

You'll also want to enabled the applyFreeShipping config setting - https://verbb.io/craft-plugins/postie/docs/get-started/configuration

We're looking into whether it's correct behaviour to exclude any products that don't have Show the Dimensions and Weight fields for products of this type checkbox checked for the product type. But it could potentially be a breaking change for some users, so we're a little hesitant to roll that out.

[clarknelson]

okay thank you so much, I will implement this fix and let you know!

[clarknelson]

thanks for looking into this and providing some solutions, but it looks like the issue still exists for me. I think I am going to try with a different provider, fedex vs ups, and see if it makes any difference.

[engram-design]

Hmm, It should exclude any product that has free shipping (and if you have applyFreeShipping => true in your config).

In your cart, do you have a mix of products that are and aren't shippable? You'll also need to update the order (add or remove items/quantity or change address details) to trigger a change.

[clarknelson]

Yes actually I needed to create a new cart, everything is looking good to me now!

[clarknelson]

I have been working and testing with this a bit and unfortunately i'm not convinced its fixed. It looks good adding the first few items, but after that it starts to increase shipping costs. Even stranger is when you remove the "free" item it will sometimes increase the shipping.

I wrote a script that set the free shipping toggle for all the relevant products in the system. And I've added a standard config file from the default on the website:

<?php

return [
    '*' => [
        'applyFreeShipping' => true,

        # default settings
        # https://verbb.io/craft-plugins/postie/docs/get-started/configuration
        'pluginName' => 'Postie',
        'hasCpSection' => true,
        'enableCaching' => false,
        'displayDebug' => false,
        'displayErrors' => true,
        'manualFetchRates' => false,
        'fetchRatesPostValue' => 'postie-fetch-rates',
        'providers' => [],
    ]
];

I believe you can test a bit on our staging website if you wish.

Physical Product (requires shipping): https://msc.betasite.cloud/products/floriani-heat-n-sta-tearaway Weight / height / length / width all set to 1 for testing.

Class Product (no shipping required): https://msc.betasite.cloud/classes/test-class-entry The free shipping toggle is enabled for this product

After adding to the cart you will need to go to the 2nd screen in the flow to add your shipping information. Then on the third screen you can change the shipping option, or return the cart screen to adjust quantities.

Maybe we could try a different provider? We only have UPS enabled right now and would prefer to stay with that. Maybe I could try 'manualFetchRates' or try to configure the settings. If you have any ideas this is the last issue before we can launch our site updates.

Thanks for all the help so far! Clark

[clarknelson]

I tried with FedEx and it looks like the shipping rate is more stable. But I am running into an issue when adjusting the quantity of items from the cart page:

2022-01-25 12:05:21 [-][181427][-][error][postie] FedEx: Rate Error: Package 1 - Invalid dimensions.

This happens for both the physical and event products, but it has me thinking there may be a problem with the dimensions and that is the source of our issue.

[engram-design]

So I've added 6 of the test classes, which eqautes to $600, and going through checkout (with a California address), doesn't seem to charge me extra?

In fact the 3rd screen is the payment screen, I don't get a shipping selection (which is correct if you're using the Commerce templates as a basis, as that's what they seem to do when no shipping methods are available).

So all providers will really treat packages the same, so switching to Fedex or UPS won't likely solve the issue.

[clarknelson]

If we just have the classes, the shipping information is hidden. You will need to have the physical product in your cart also to see the shipping options.

[engram-design]

I can definitely replicate that on your site.

Those prices certainly shouldn't change.

One thing to ask - does your product type for classes have Show the Dimensions and Weight fields for products of this type enabled? if that's disabled, you shouldn't even need to mark products as having "Free Shipping". It should exclude any product from calculations that have no dimensions/weights set, or if they're set to 0.

Are you able to confirm the details of your Test Class product?

[clarknelson]

I believe that is how we have it configured, I have both enabled as "overkill".

And the classes product type page, the product weight / dimensions should not be checked.

[clarknelson]

There is a fairly simple AJAX query on the site. When you modify the quantity in the cart, it will run ajax to update the cart. That will return the cart object, and I update the shipping options based on the contents of the order. I don't think it should be a big deal but may be worth mentioning.

I am dumping some variables from the UPS provider service, I noticed a few pieces that may be interesting or relevant.

The $modifyRatesEvent object has different rates that what is within the cart object. And I see a message "User Id and Shipper Number combination is not qualified to receive negotiated rates. Your invoice may vary from the displayed reference rates." Maybe this is something I should resolve? I am not entirely familiar with UPS's API or services quite frankly.

Another potential issue, the weight of the package originally looks correct. I have 2x items weighing 14.1lb, an item weighing 1lb, and a weightless class item. But when it becomes "billing rate" it is a slightly larger number.

I'll keep digging to try and see what may be happening. If there's anything you think I should look at let me know. We are pushing back the launch date until we get this resolved.

[engram-design]

I've just noticed your "Events" shipping category. Can you let me know what that's configured with? That's probably the only difference to my testing. If a purchasable thing has no defined weight/dimensions, it's simply not factored into the box-packaging algorithm, and hence not included in the rate request for provider estimates.

For the UPS error, that's just a warning. Basically means that your UPS account isn't setup with negotiated rates, which is totally fine for some customers. If there were negotiated rates available, it'd use those as they'd be cheaper.

As for the rate disparity (S_GROUND is 62.16 from the API, 68.38 on the front-end that's quite interesting. That's a 6.22, roughly 10% markup. Just wanted to check if you'd set any markup rates (amount or percentage) on rates?

As for the billing rate, I believe that's the weight reported back from UPS, I'm not sure why that's different, but I'd need to look into it. It could likely be their designated weight bands for pricing (33lbs being the top of that price band).

[clarknelson]

The markup rate was set to 10%, so that explains that bug.

[clarknelson]

I believe the problem is the "insurance", changing the class quantity while this option is enabled will change this shipping.

[engram-design]

Ah good to know about the markup rate, and yep Insurance will add more costs to items depending on the order total. I suppose that should be the shippable total for an order, as it'd be including the non-shippable items.

To get this fix early, change your verbb/postie requirement in composer.json to:

"require": {
  "verbb/postie": "dev-craft-3 as 2.4.15",
  "...": "..."
}

Then run composer update.

[clarknelson]

Thank you I have been testing without insurance and it seems to run much better, I will try with your patch.

One final bug I noticed, when I add items it looks good, but when I remove items sometimes the shipping goes up. I believe this may have to do with the AJAX query that will post to commerce/cart/update-cart. The returned cart object has an availableShippingMethods property that I update the front-end with. Maybe this information is cached somehow? Or there is a race condition between fetching the rates and when the commerce/cart/update-cart function returns?

[clarknelson]

I tried the patch but I don't think it works. Is there an event I can listen for when the shipping rates return? That may solve my cache issues

[clarknelson]

https://imgur.com/a/cuNkZzc

[clarknelson]

As far as I can tell, the rate displayed is what is returned from the query in the plugin, and is what is attached to the cart object on the AJAX return. If you refresh the page a few seconds later, the "correct" rate is displayed.

[clarknelson]

I did spend some time re-writing the AJAX on my cart page, which may have been responsible for some of the bugs. I do think there is some kind of race condition, where "update-cart" returns before postie can fetch the new rates. I "fixed" this by adding an additional AJAX query. The first query updates the cart quantities, and the second will fetch the shipping rates. By this time they seem to be correct.

If you have a chance, can you take a look at the "free shipping" products adding shipping costs when "insurance" is enabled for UPS? It may be the last item we need to resolve before rolling out the shipping feature

[engram-design]

Sorry to leave you hanging - I've been out of action for about a week.

That does seem odd though, as the fetching of rates should be done upon updating the cart, but I'll double check it. It's likely that Commerce's controller returns the cart (caches or memoized) incorrectly.

So as per my patch above, products with no dimensions/weight set or set to "Free Shipping" will be excluded from box packing, and insurance estimations, so this shouldn't be factoring into insurance costs anymore. Instead of looking at the entire cost for the order, it'll now look at the cost of all the products being shipped (box-packed), so the amounts should be correct. Still not that case for you?

[clarknelson]

I believe that is true, I have disabled insurance for UPS on the dev website and it no longer charges for classes. I can create you an account to adjust the settings in the dashboard if that is convenient for testing. You can email me the info email@clarknelson.com

[engram-design]

@clarknelson I believe both your issues should now be solved. The non-shippable products was something on my end incorrect.

To get this fix early, change your verbb/postie requirement in composer.json to:

"require": {
  "verbb/postie": "dev-craft-3 as 2.4.15",
  "...": "..."
}

Then run composer update.

The caching thing was sort of implemented right, but I've improved it. For some background, when updating an order, Commerce will recalculate rates a whopping 6 times. That's 6 requests off to the provider to fetch rates. As such Postie adds a layer of caching at two levels.

First, with enableCaching set to true this will store the results from the provider in the cache, against a unique key for the order. This key will be used to check whether or not the order has been changed, for every subsequent request to update the cart. Unless the customer has updated their cart contents or shipping address, the cached data will be used, and a request to the provider won't ever be hit, no matter how many times you update your cart.

Now, with enableCaching set to false, we had some in-memory (memoized) caching to prevent 6 calls off to the provider again. The different to the above approach is that there is always at least 1 call to the provider on every cart update. I've improved the mechanism here for that instead of ignoring any rates if any were set previously. What I was finding is that the first 2 runs of Commerce fetching available shipping methods, the order details were the old details. Then on the 3rd go, it was using the new amounts. But because Postie would ignore everything but the first call, they weren't accurate.

In both cases, in my testing, there should no longer be any lag. I would highly recommend switching on enableCaching => true for production.

[clarknelson]

Wow that's a really great answer thank you for taking the time to explain it to me. It seems to work on dev! I can't thank you enough for helping me through this.