InboundShipments issue (develop branch)

viavacavi commented 4 years ago

This is a continuation from issue #214

I'm currently trying to use the InboundShipments API, specifically create_inbound_shipment_plan() This is using the current Develop branch, running on Python 3.8 on Ubuntu 20.04

inboundShipments = mws.InboundShipments(
    access_key=settings.get['az_access_key'], secret_key=settings.get['az_secret_key'],
    account_id=settings.get['az_seller_id'],auth_token=settings.get['az_auth_token']
)
address = {
    "name": settings.get['companyName'],
    "address_1":settings.get['address1'],
    "address_2": settings.get['address2'],
    "city": settings.get['city'],
    "district_or_county": settings.get['district'],
    "state_or_province": settings.get['region'],
    "postal_code": settings.get['postal'],
    "country": settings.get['country']
}
inboundShipments.set_ship_from_address(address)

# Simple version of items works...
items=[ {'sku': '012-34-5678', 'quantity': 50},  {'sku': '123-45-6789', 'quantity': 100} ]

inbound.create_inbound_shipment_plan(items,label_preference='SELLER_LABEL')

The above code with 2 items works (with an XML response stating the SKUs don't exist obviously). However, I'm having issues with more complicated lists of items. The item generation below with 60 items fails:

items=[]
for i in range(1,60):
    items.append({'sku': 'sku-'+str(i), 'quantity': i})

With attempts to create shipping plans with a large number of items, I get a variety of errors, ranging from: HTTP Error 400 Error 405 which returns HTML (not XML)

We have over 300 SKUs on Amazon, so our shipping plans can be pretty large. Looking over the MWS developer API documentation, I don't see a stated limit on the number of items that can be added with create_inbound_shipment_plan(). I'm not sure if the issue is on Amazon's end, python-amazon-mws, or urllib.

Here is an example error from trying to add 30 items:

DEBUG:urllib3.connectionpool:https://mws.amazonservices.com:443 "POST /FulfillmentInboundShipment/2010-10-01?AWSAccessKeyId=XXXXXXXX&Action=CreateInboundShipmentPlan&InboundShipmentPlanRequestItems.member.1.Quantity=1&InboundShipmentPlanRequestItems.member.1.SellerSKU=sku-1&InboundShipmentPlanRequestItems.member.10.Quantity=10&InboundShipmentPlanRequestItems.member.10.SellerSKU=sku-10&InboundShipmentPlanRequestItems.member.11.Quantity=11&InboundShipmentPlanRequestItems.member.11.SellerSKU=sku-11&InboundShipmentPlanRequestItems.member.12.Quantity=12&InboundShipmentPlanRequestItems.member.12.SellerSKU=sku-12&InboundShipmentPlanRequestItems.member.13.Quantity=13&InboundShipmentPlanRequestItems.member.13.SellerSKU=sku-13&InboundShipmentPlanRequestItems.member.14.Quantity=14&InboundShipmentPlanRequestItems.member.14.SellerSKU=sku-14&InboundShipmentPlanRequestItems.member.15.Quantity=15&InboundShipmentPlanRequestItems.member.15.SellerSKU=sku-15&InboundShipmentPlanRequestItems.member.16.Quantity=16&InboundShipmentPlanRequestItems.member.16.SellerSKU=sku-16&InboundShipmentPlanRequestItems.member.17.Quantity=17&InboundShipmentPlanRequestItems.member.17.SellerSKU=sku-17&InboundShipmentPlanRequestItems.member.18.Quantity=18&InboundShipmentPlanRequestItems.member.18.SellerSKU=sku-18&InboundShipmentPlanRequestItems.member.19.Quantity=19&InboundShipmentPlanRequestItems.member.19.SellerSKU=sku-19&InboundShipmentPlanRequestItems.member.2.Quantity=2&InboundShipmentPlanRequestItems.member.2.SellerSKU=sku-2&InboundShipmentPlanRequestItems.member.20.Quantity=20&InboundShipmentPlanRequestItems.member.20.SellerSKU=sku-20&InboundShipmentPlanRequestItems.member.21.Quantity=21&InboundShipmentPlanRequestItems.member.21.SellerSKU=sku-21&InboundShipmentPlanRequestItems.member.22.Quantity=22&InboundShipmentPlanRequestItems.member.22.SellerSKU=sku-22&InboundShipmentPlanRequestItems.member.23.Quantity=23&InboundShipmentPlanRequestItems.member.23.SellerSKU=sku-23&InboundShipmentPlanRequestItems.member.24.Quantity=24&InboundShipmentPlanRequestItems.member.24.SellerSKU=sku-24&InboundShipmentPlanRequestItems.member.25.Quantity=25&InboundShipmentPlanRequestItems.member.25.SellerSKU=sku-25&InboundShipmentPlanRequestItems.member.26.Quantity=26&InboundShipmentPlanRequestItems.member.26.SellerSKU=sku-26&InboundShipmentPlanRequestItems.member.27.Quantity=27&InboundShipmentPlanRequestItems.member.27.SellerSKU=sku-27&InboundShipmentPlanRequestItems.member.28.Quantity=28&InboundShipmentPlanRequestItems.member.28.SellerSKU=sku-28&InboundShipmentPlanRequestItems.member.29.Quantity=29&InboundShipmentPlanRequestItems.member.29.SellerSKU=sku-29&InboundShipmentPlanRequestItems.member.3.Quantity=3&InboundShipmentPlanRequestItems.member.3.SellerSKU=sku-3&InboundShipmentPlanRequestItems.member.30.Quantity=30&InboundShipmentPlanRequestItems.member.30.SellerSKU=sku-30&InboundShipmentPlanRequestItems.member.31.Quantity=31&InboundShipmentPlanRequestItems.member.31.SellerSKU=sku-31&InboundShipmentPlanRequestItems.member.4.Quantity=4&InboundShipmentPlanRequestItems.member.4.SellerSKU=sku-4&InboundShipmentPlanRequestItems.member.5.Quantity=5&InboundShipmentPlanRequestItems.member.5.SellerSKU=sku-5&InboundShipmentPlanRequestItems.member.6.Quantity=6&InboundShipmentPlanRequestItems.member.6.SellerSKU=sku-6&InboundShipmentPlanRequestItems.member.7.Quantity=7&InboundShipmentPlanRequestItems.member.7.SellerSKU=sku-7&InboundShipmentPlanRequestItems.member.8.Quantity=8&InboundShipmentPlanRequestItems.member.8.SellerSKU=sku-8&InboundShipmentPlanRequestItems.member.9.Quantity=9&InboundShipmentPlanRequestItems.member.9.SellerSKU=sku-9&LabelPrepPreference=SELLER_LABEL&MWSAuthToken=XXXXXXXX&SellerId=XXXXXXXXX&ShipFromAddress.AddressLine1=1234%20US%20Rt.%206&ShipFromAddress.City=Cleveland&ShipFromAddress.CountryCode=US&ShipFromAddress.Name=MyBusiness&ShipFromAddress.PostalCode=11111&ShipFromAddress.StateOrProvinceCode=OH&ShipToCountryCode=US&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2020-09-25T11%3A04%3A24&Version=2010-10-01&Signature=lOrDp3aLVz0sh5abS04QDNmB3daGUl1ME7%2BM2FHU3uA%3D HTTP/1.1" 400 None
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/mws/mws.py", line 339, in make_request
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: XXXXXXXXXXXXX (masked out)
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/viavacavi/ViaVinyl/viavinyl.py", line 204, in <module>
    ComputeReplenishments(wProd,db)
  File "/home/viavacavi/ViaVinyl/update.py", line 238, in ComputeReplenishments
    inbound.create_inbound_shipment_plan(items,label_preference='SELLER_LABEL')
  File "/usr/local/lib/python3.8/dist-packages/mws/apis/inbound_shipments.py", line 240, in create_inbound_shipment_plan
    return self.make_request(data, method="POST")
  File "/usr/local/lib/python3.8/dist-packages/mws/mws.py", line 361, in make_request
    raise error
mws.mws.MWSError

viavacavi commented 4 years ago

Update: Found this mws documentation stating that up to 200 items can be added through CreateInboundShipmentPlan: https://docs.developer.amazonservices.com/en_UK/fba_guide/FBAGuide_CreateShipment200Items.html

viavacavi commented 4 years ago

Can anyone confirm my results? I can't seem to get any more than about 20 SKUs to work properly.

Bobspadger commented 4 years ago

From the top of my head, I'm pretty sure a lot of the calls are limited to batches of 20 - this is the case for a few of the endpoints.

viavacavi commented 4 years ago

@Bobspadger that was my initial thought too, but the above link clearly states a recommendation (not even limit) of 200 skus per request. In my own tests, the number of skus that I can successfully create a shipment plan with varies from 25-28 depending on the request. There appears to be an issue with python-amazon-mws or urllib.

Bobspadger commented 4 years ago

Looking at the trace though, this is a 400 error from Amazon - suggesting the issue is starting there.

Now, this could be because of a badly formatted input (our issue) or that there is too much data being sent, or, what I have seen before is the URL is too long.

Are you able to capture the full request, and the response (these are available on the return from the API)

If you need to XXX out various bits for security thats cool but I think it will help us find the source of the issue.

Amazon often return more precise errors in the xml they return

viavacavi commented 4 years ago

Ok, I have done a ton of debugging and research today. Here is what I've found:

Using the MWS scratchpad, I can definitely create a shipment plan with a large number of skus:
However, look at the 400 response:
This shows that a response(400) is a valid amazon response, and this is where the problem lies with the python-amazon-mws code. The response (400) means "This operation may be partially successful...", and is a flag for the programmer to check the XML response for errors that may have happened during the transaction. However, python-amazon-mws treats the response 400 as an exception and bombs out. Further complicating this, the response dictwrapper isn't even set, so the programmer can't even see what the issue may have been.
The correct way this should be handled is that response(400) should be allowed as perfectly valid, and the response dictwrapper should be set the .response key appropriately so that the programmer can decide on how to act on it instead of the program just bombing out on an MWSError exception
in my own code that has not been working, response(400) has been returned because one of my SKUs has a COVID-19 restock limitation. Again, this should be perfectly valid as the rest of the SKUs are added to the plan as they should (the operation was "partially successful) - the response XML is just notifying the programmer that one of the SKUs weren't added per the request.

Here is a section in the MWS docs regarding error handling: https://docs.developer.amazonservices.com/en_US/dev_guide/DG_Errors.html

cajohn2757 commented 4 years ago

Whenever I run my script it doesn't seem to POST to Amazon's System. I am not getting any errors; I just don't know why it's not uploading the shipping plan. When I run it through MWS scratchpad its response is 200, so I'm not sure where my this is breaking down for me.

viavacavi commented 4 years ago

@cajohn2757 Only plans created through the Seller Central interface appear online. You will have to process the response from create_inbound_shipment_plan to create the individual shipments

If you want to have a shipping plan show up in seller central, you can instead use the Feeds API with _POST_FLAT_FILE_FBA_CREATE_INBOUNDPLAN

Bobspadger commented 4 years ago

Ok, I have done a ton of debugging and research today. Here is what I've found:

Using the MWS scratchpad, I can definitely create a shipment plan with a large number of skus:

However, look at the 400 response:

This shows that a response(400) is a valid amazon response, and this is where the problem lies with the python-amazon-mws code. The response (400) means "This operation may be partially successful...", and is a flag for the programmer to check the XML response for errors that may have happened during the transaction. However, python-amazon-mws treats the response 400 as an exception and bombs out. Further complicating this, the response dictwrapper isn't even set, so the programmer can't even see what the issue may have been.

The correct way this should be handled is that response(400) should be allowed as perfectly valid, and the response dictwrapper should be set the .response key appropriately so that the programmer can decide on how to act on it instead of the program just bombing out on an MWSError exception

in my own code that has not been working, response(400) has been returned because one of my SKUs has a COVID-19 restock limitation. Again, this should be perfectly valid as the rest of the SKUs are added to the plan as they should (the operation was "partially successful) - the response XML is just notifying the programmer that one of the SKUs weren't added per the request.

Here is a section in the MWS docs regarding error handling: https://docs.developer.amazonservices.com/en_US/dev_guide/DG_Errors.html

Cool - thats fantastic work. Thanks 👍

I'll try to take a look at that part of the code ASAP

GriceTurrble commented 4 years ago

Deleted my prior response as I re-read the convo, and what I stated before didn't seem relevant.

While it's true PAM could be more graceful about handling these errors, in my experience the "partially successful" error response is dubiously named. You need the successful response from create_inbound_shipment_plan in order to make use of the plans it generates. If the only response is an error, then essentially the entire request has failed: you have to remove the offending SKUs from the payload and try create_inbound_shipment_plan again with valid SKUs only, or they won't generate a shipment plan for you.

I have not tested extensively with HTTP400 errors in the response payload in the new code we've been working with, but it's definitely something we need to check for. I do know that requests will to a raise_for_status call on the response and raise some exception for any 400 error; and PAM would not capture that exception properly to then parse the error.

Something I'll have to check for in the next sprint when I have some time over the next few weeks, though. Appreciate the work you've done debugging the issue, @viavacavi :)

viavacavi commented 4 years ago

@GriceTurrble I agree, it's a very strange choice Amazon has made with the HTTP400 error in these cases. In any case, PAM should definitely let the programmer handle this type of error handling, as without the XML response from Amazon there is no way to know what is causing the crash. For now I'm just going to eliminate the try/except block around HTTPError in make_request so that I can keep pushing forward in my code. Perhaps the exception should only be thrown on an HTTPError in which no XML response is recieved.

Thanks for your attention on this, PAM is an excellent project. I'm quite familiar with the MWS API, so hopefully I can help out when/where possible (I'm still a bit new to Python, but I'm advancing pretty well)

python-amazon-mws / python-amazon-mws

InboundShipments issue (develop branch) #215